helm: configurable annotations for agent and operator pods

[mvisonneau]

This change allows the enduser to specify arbitrary annotations for the agent and operator pods. This is not necessarily Cilium related but is for example a prerequisite to get Datadog autodiscovery configured. Thanks to this change, it gets as "simple" as:

---
agent:
  podsAnnotations:
    ad.datadoghq.com/cilium-agent.check_names: '["cilium"]'
    ad.datadoghq.com/cilium-agent.init_configs: '[{}]'
    ad.datadoghq.com/cilium-agent.instances: '[{"agent_endpoint": "http://%%host%%:9090/metrics"}]'

operator:
  podsAnnotations:
    ad.datadoghq.com/cilium-operator.check_names: '["cilium"]'
    ad.datadoghq.com/cilium-operator.init_configs: '[{}]'
    ad.datadoghq.com/cilium-operator.instances: '[{"operator_endpoint": "http://%%host%%:6942/metrics"}]'

[maintainer-s-little-helper]

Please set the appropriate release note label.

[coveralls]

Coverage decreased (-0.02%) to 37.125% when pulling 8e32f4a on mvisonneau:helm/podsAnnotations into 2f04afd on cilium:master.

[joestringer]

Thanks for the contribution, one minor comment below.

[nebril]

smoke tests passed, merging

[mvisonneau]

is there any plan to backport this PR in 1.8 or 1.9 at some point? 🤔

[dominik318]

@mvisonneau just wanted to let you know that the prevailing spelling for most helm charts out there seems to be podAnnotations, not podsAnnotations. Do we think it is too late to change?

[mvisonneau]

oh indeed @dominik318, I'll submit another PR to amend! I reckon it is still fine to change it given that it hasn't been backported yet 🤷

[joestringer]

Glad we caught the typo before pushing it into a release :-) We discussed on #13458 to make these changes available in a subsequent v1.8.x release, assuming @mvisonneau prepares the changes against the v1.8 branch.