Add flink

[iemejia]

Hello,

I and other members of the flink community have been working on refining the docker image with the goal of making it an official image, and we expect to soon start a PR to get it out.
https://github.com/apache/flink/blob/master/flink-contrib/docker-flink/Dockerfile

Notice that we are still cleaning stuff, but I wanted to start the discussion here to get some early feedback.

[iemejia]

I have noticed that in other apache projects the image maintainers are particulars, who create a docker repo with all the variations for the Dockerfile versions, is there a way we can avoid this and have the image still live in the apache repo? This makes me think about an extra question about how the images are built, I am not sure we can pass this env variables or I don't understand why people create all this variations (e.g. while looking at the solr official image).

[iemejia]

@patricklucas (in case you have something to add here).

[yosifkit]

Sorry for the slow reply. 😞

You are welcome to host the Dockerfile and context in the git repo of your choice, so within the Apache repo would work. The requirements/guidelines we have from the Official-Images perspective are in the readme in official-images but I'll point out a couple to answer some of your questions:

• we build sets of tags from each given context (git repo, commit id, directory)
  • build process is roughly equivalent to git archive commit:directory/ -o tmp.tar, tar -xf tmp.tar -C tmp-location and then docker build tmp-location
  • we don't currently support things like build-arg on the docker build (haven't had a good reason yet to support it since build args have more complex docker cache behavior)
  • we also don't have a way to specify a build context where the Dockerfile is in a sub-directory (no reason has come up to support it)
• the Dockerfile needs to be "Repeatable", so that an update to the base image would not cause a rebuild of the image to contain a new version

Overall I think your Dockerfile and entrypoint script are looking pretty close for inclusion. The main change I would make is to download a specific version of the flink_dist with curl or wget in the Dockerfile, since there is no way to prepare the build context in the official images beyond what exists in the directory in git.

[patricklucas]

Hi, @yosifkit. A bit has actually changed since @iemejia originally posted this.

We decided instead to follow suit with a number of other official images and created a dedicated repo for the Dockerfiles and related scripts, docker-flink/docker-flink.

The Dockerfiles therein should be much more in line with expectations.

[yosifkit]

@patricklucas Those look really great! 😍 I would be worried about committing the KEYS file rather than just grabbing keys from a key server, but updating a list in the Dockerfile would be about the same since you are upstream and generate the KEYS file as part of you regular release process anyway so it is fine.

So, in conclusion, I would say it would be really easy to get this in the official-images. I think you just need a PR to https://github.com/docker-library/official-images and then one here to add some docs.

[iemejia]

Oh that's great news to know that the image is quite close to what you expect. We will work on this really soon. For technical reasons (an issue with flink's networking and docker) we will start probably supporting from Flink version 1.2.1, currently on release process. Thanks a lot.

[iemejia]

@yosifkit I just added a first version of the docs in our fork, can you check and tell me if there are things to fix, so we can PR this soon. Thanks.
https://github.com/docker-flink/docs/tree/master/flink

[patricklucas]

@yosifkit We actually started by using a keyserver, but the maintenance burden of inlining the signatures of every key we might use seemed pointless given that the KEYS file is guaranteed by upstream to remain up to date. Also, the keyserver pool proved very unreliabile—I actually noticed other projects adding retry/other complex logic to their Dockerfiles in the last few weeks just to deal with that—so this route seemed the best one, removing that external dependency.