fix the bug that evenhanlder ignores the update per sync-period

[oliviassss]

Issue

#3264
#2800
#2515

Description

Currently, there is a conflict between the controller runtime flag --sync-period, which is to force reconcile every fix interval, and the eventhandlers, which are to avoid unnecessary reconciles to optimize the AWS API usage. When the controller runtime sends an update event to the eventhandler per interval in --sync-period, the eventhandler will ignore the update event if there is no difference in the annotations/specs of the ingress or service. Therefore, if the end users made some manual modification to the resources managed by the controller as mentioned in the issues above, the controller could not revert since it does not reconcile under this situation.

We fix the bug by leveraging k8s resourceVersions -

• if the resourceVersions are the same between old and new objects, the controller will continue to reconcile;
• if the resourceVersions do change, then the eventhandler continues to check the annotation and specs to see if it is an update event with real change in resources.

However, as the best practice, we do not recommend manual modification, nor should the users depend on the controller auto-reconciliation to heal the resources they changed, or to mitigate any security risks. Since no matter how frequent the controller reconciles, there should always be a gap.

Test

• created an ingress with alb.ingress.kubernetes.io/wafv2-acl-arn, verified the waf acl got associated to the provisioned ALB. Then removed the waf acl from the ALB via console, verified the controller will re-associalte the waf acl to the ALB.
• created an ingress, then added a new inbound SG rule to the ALB via console, verified the controller will revert the change of the inbound SG rules
• created an ingress, and then deleted the provisioned ALB from console, verified the controller will re-create the ALB
• created a service type Load Balancer, then removed the NLB from console, verified the controller will re-create the NLB.
• Verified the controller can successfully delete all the resources created by it.

Checklist

• Added tests that cover your change (if possible)
• Added/modified documentation as required (such as the README.md, or the docs directory)
• Manually tested
• Made sure the title of the PR is a good description that can go into the release notes

BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯

• Backfilled missing tests for code in same general area 🎉
• Refactored something and made the world a better place 🌟

[M00nF1sh]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: M00nF1sh, oliviassss

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [M00nF1sh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment