Allow K8S node labels to be propagated as Pod annotations

[barkbay]

Fixes #3933

How to test (tested on GKE)

• Start the operator with the following flag: --exposed-node-labels="topology.kubernetes.io/.*,failure-domain.beta.kubernetes.io/.*"
• Annotate an Elasticsearch deployment: k annotate --overwrite es/elasticsearch-sample eck.k8s.elastic.co/downward-node-labels="topology.kubernetes.io/zone,topology.kubernetes.io/region"
• Wait for the Pods to be restarted
• Check the annotations: k get pod -l common.k8s.elastic.co/type=elasticsearch -o go-template='{{range .items}}{{.metadata.annotations}}{{printf "\n"}}{{end}}'

To get the logs of the init script you can run k logs elasticsearch-sample-es-default-0 -c elastic-internal-init-filesystem -f

Misc

• I eventually decided to reuse the elastic-internal-init-filesystem init container to wait for the annotations to be set.
• This PR should not restart the current RUNNING Pods. The elastic-internal-init-filesystem init-container should be left untouched as long as the feature is not used on an Elasticsearch resource.
• Pods are patched early in the reconciliation loop. I'm not sure if there's an ideal place (or a place we should avoid)
• A Pod is considered as scheduled when both its Spec.NodeName has a non-empty value and the relevant condition PodScheduled is set to True (see K8S binding code here)
• The primary intent for this PR is to propagate topology node labels, hence node labels are considered as immutable and K8S Nodes are not watched.
• When an Elasticsearch resource is submitted the admission controller only check that the node label is allowed. Node labels are not checked until the Pod is scheduled. This is mostly because node labels may be inconsistent.
• I'll try to think about some e2e tests in a subsequent PR.

To be addressed in follow up PRs

• Update documentation, including docs/operating-eck/operating-eck.asciidoc
• Add E2E Tests
• Enable telemetry Add downward node labels api to telemetry #5101

[pebrc]

Looks really good. 👍 I did a few simple tests and it worked as advertised. I have only a few small comments nothing major. Naming suggestions are totally optional of course.

[sebgl]

(applies to anything:) how do we decide whether a feature is specified as an annotation or as a field in the spec?
We went the annotation way here but I'm not sure about the reasoning. There is an "alpha" feel to the annotation, but I suspect we'd still have to commit to maintaining this working over time anyway.

From a user pov, I'm wondering if i'd rather want to do something like this, much higher-level:

spec:
  allocationAwareness:
    enable: true # defaults to false
    attribute: "zone" # default
    nodeLabel: "topology.kubernetes.io/zone" # default

and have the operator automatically handle the Elasticsearch configuration part + the downward API + the environment variable propagation?

[barkbay]

I suspect we'd still have to commit to maintaining this working over time anyway.

I think one of the reason is that I have the feeling to implement something which might be part of K8S at some point. I'm still hoping for an "official" solution. It feels then easier to deprecate and remove an annotation than a field in the spec (assuming we don't want to maintain a feature which is natively available in K8S).
It also allows us to get some feedback early, and adjust the feature if needed. This PR might then be a building block for a more advanced feature as the one outlined in your message.
Happy to reconsider if we think it deserves an update of the Elasticsearch schema.

[pebrc]

LGTM, I think this will be really useful 🚀