Init containers should tolerate restarts

[barkbay]

It may happen that an init container is restarted even if the Pod is already running.

See the following issues:

• init-container restarted after docker prune image kubernetes/kubernetes#86531
• Init-containers are restarting with no reason after completion. kubernetes/kubernetes#67261 (comment)

The main issue comes from the init containers that initialize the keystores. For example the Elasticsearch keystore can only be created once, any additional run expect an input from stdin:

echo 'Initializing keystore.'
+ /usr/share/elasticsearch/bin/elasticsearch-keystore create
Initializing keystore.
Exception in thread "main" java.lang.IllegalStateException: unable to read from standard input; is standard input open and a tty attached?
        at org.elasticsearch.cli.Terminal$SystemTerminal.readText(Terminal.java:212)

Another issue is when we use the --force flag, as we do for the APM Server for example: we should not execute blindly --force since this is (re)creating a temporary empty keystore while the APM Server is running.

I think we can solve this problem by using a "flag" on the filesystem. If we want to do some "fine grain recovery" in order to recover from various, more complicated, situations then we need to dig into any keystore specific behaviour, maybe deal with some implementation details, and keep in mind that the keystore might be in used when the init container is running.