Allow init container modifications

[charith-elastic]

Currently, if a user wants to change a small aspect of an init container (e.g. setting custom resource limits), they have to provide the full definition of the init container in the podTemplate.

Presumably we do this to give users full control over how init containers are executed. However, there are a couple of UX issues worth considering:

• Most users are probably only interested in minor tweaks such as changing the resource limits of the container
• The way the operator treats initContainers of the podTemplate is confusing. (Other sections in the podTemplate get merged while initContainers simply get replaced if the container name matches a built-in container)

[ferozsalam]

The elastic-internal-init-filesystem init container is OOMKilled on my machine when I use the default ECK resource definitions (following the Quickstart guide for ECK). I found this issue, which led me to the workaround, however I do think being able to increase the default resources in a less verbose manner would be quite useful.

[sebgl]

@ferozsalam interesting. The current memory limit is set to 20Mi (#2186 and #2241). Which should normally be enough for that init container, afaik that's the first time we hear it's not.

What memory limit are you applying as a workaround? Can you give more details about your Kubernetes environment?

[ferozsalam]

@sebgl I've done some testing and it's fine with 30Mi (potentially, lower, I didn't bother going below 10Mi increments).

Regarding my k8s setup - it's a minikube instance on a desktop. Resources are ample though, and I'm typically able to run several dozen containers on it simultaneously. I have attached the output of kubectl describe po and kubectl logs for the relevant container, if this is of any use:
describe_po.txt
logs.txt

[sebgl]

@ferozsalam thanks, that's very useful.
It seems to crash while copying Elasticsearch bin/ content to the emptyDir:

Copying /usr/share/elasticsearch/bin/* to /mnt/elastic-internal/elasticsearch-bin-local/
'/usr/share/elasticsearch/bin/elasticsearch' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch'
'/usr/share/elasticsearch/bin/elasticsearch-certgen' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-certgen'
'/usr/share/elasticsearch/bin/elasticsearch-certutil' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-certutil'
'/usr/share/elasticsearch/bin/elasticsearch-cli' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-cli'
'/usr/share/elasticsearch/bin/elasticsearch-croneval' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-croneval'
'/usr/share/elasticsearch/bin/elasticsearch-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-env'
'/usr/share/elasticsearch/bin/elasticsearch-enve' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-enve'
'/usr/share/elasticsearch/bin/elasticsearch-keystore' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-keystore'
'/usr/share/elasticsearch/bin/elasticsearch-migrate' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-migrate'
'/usr/share/elasticsearch/bin/elasticsearch-node' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-node'
'/usr/share/elasticsearch/bin/elasticsearch-plugin' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-plugin'
'/usr/share/elasticsearch/bin/elasticsearch-saml-metadata' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-saml-metadata'
'/usr/share/elasticsearch/bin/elasticsearch-setup-passwords' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-setup-passwords'
'/usr/share/elasticsearch/bin/elasticsearch-shard' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-shard'
'/usr/share/elasticsearch/bin/elasticsearch-sql-cli' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-sql-cli'
'/usr/share/elasticsearch/bin/elasticsearch-sql-cli-7.5.2.jar' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-sql-cli-7.5.2.jar'
/mnt/elastic-internal/scripts/prepare-fs.sh: line 80:    22 Killed                  cp -av /usr/share/elasticsearch/bin/* /mnt/elastic-internal/elasticsearch-bin-local/

[jgoeres]

I am sometimes observing the exact same problem on my minikube instance. Today, deploying my workload that includes an ECK-powered ES worked several dozen times just fine, now all of a sudden, the elastic-internal-init-filesystem init container gets OOMKilled all the time:

'/usr/share/elasticsearch/bin/elasticsearch-sql-cli' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-sql-cli'
'/usr/share/elasticsearch/bin/elasticsearch-sql-cli-7.6.0.jar' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-sql-cli-7.6.0.jar'
/mnt/elastic-internal/scripts/prepare-fs.sh: line 80:    23 Killed                  cp -av /usr/share/elasticsearch/bin/* /mnt/elastic-internal/elasticsearch-bin-local/

Having the option to overwrite selected aspects of the init container instead of having to copy the entire default definition and modify it (and thus possibly miss any changes that might be done by newer versions of the operator) would be really useful.