Contour gateway api provisionner revert annotations

[laserpedro]

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

After the service envoy-xxx-gateway is created I patched it as indicated as the doc so that my cloud provider creates the LB. However, I notice that those annotations are overriden with the default ones after a certain amount of time.

What did you expect to happen:

I expect the annotations to remain in the envoy service (they are crucial as they enable the registration of new eks nodes)

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Contour version: 1.29
• Kubernetes version: (use kubectl version): 1.29
• Kubernetes installer & version: aws eks
• Cloud provider or hardware configuration: bottlerocket
• OS (e.g. from /etc/os-release): linux amazon

[laserpedro]

NOTE: the contourDeployment enables to auto define annotations and tolerations for envoy fleet so I guess that this is in a way solved. However I would be curious to know whether the behavior was intentional to revert the annotations ?

[skriss]

xref #4473

I would say the behavior for this scenario is underspecified; however, I think it would be reasonable for the provisioner to not remove annotations that were set on the service out-of-band, and to only ensure that the annotations the provisioner owns are present and set correctly.