[FLINK-13865][docker] Support custom Flink config through env variable

[functicons]

If the env variable FLINK_PROPERTIES is not empty, the value will be
appended to conf/flink-conf.yaml. The variable should be a multi-line
string with each line in the format of ": ".

[patricklucas]

@functicons thanks for the suggestion. Usually I've been hesitant to add more ways to configure Flink that are only supported by the Docker image, but maybe this is a reasonable stopgap until Flink itself provides a better solution. (e.g. by allowing use of env vars to configure any option; e.g. a mapping from $FLINK_TASKMANAGER_HEAP_SIZE to taskmanager.heap.size)

@tillrohrmann can you weigh in if you think this is a good approach for the time being? While it's not my ideal solution, at least it gives a "way out" for users who don't want to produce their own config file, and who can accept running the image as root or flink (since the entrypoint script needs to write out a file).

[patricklucas]

Please only modify the top-level docker-entrypoint.sh script instead of the generated files.

[tillrohrmann]

I think that this solution works even though it is not nice.

Maybe another idea would be to extend the jobmanager.sh and taskmanager.sh script to forward arbitrary command line options to the flink-console.sh/flink-daemon.sh script. That way one could pass in some dynamic properties which are being parsed and then overwrite existing configuration values via -Dconfig.key=config.value.

[patricklucas]

Alright, I'd like to merge this as an "unsupported" feature—that is, the parameter may go away in a future minor release (e.g. it will definitely be kept for 1.9.z, but may be removed in 1.10 releases).

[tweise]

@patricklucas @tillrohrmann can we please make this an "official" feature as it is needed to use these images in a Kubernetes environment.

For background, this idea was borrowed from https://github.com/lyft/flinkk8soperator/blob/373cb21e6d5c7247dfc7c18c7a4d202ce2eedb2a/examples/wordcount/docker-entrypoint.sh#L17

It would be nice to be able to depend on this, and to be able to uses these images with FlinkK8sOperator.

@functicons thanks for contributing this here, and I think it would be good style to attribute work appropriately.