[FLINK-4118] The docker-flink image is outdated (1.0.2) and can be slimmed down #2176

iemejia · 2016-06-28T13:05:40Z

Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the How To Contribute guide.
In addition to going through the list, please provide a meaningful description of your changes.

General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the JIRA id)
Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
Tests & Build
- Functionality added by the pull request is covered by tests
- mvn clean verify has been executed successfully locally or a Travis build has passed

Some of the changes include: - Remove unneeded dependencies (nano, wget) - Remove apt lists to reduce image size - Reduce number of layers on the docker image (best docker practice) - Remove useless variables and base the code in generic ones e.g. FLINK_HOME - Change the default JDK from oracle to openjdk-8-jre-headless, based on two reasons: 1. You cannot legally repackage the oracle jdk in docker images 2. The open-jdk headless is more appropriate for a server image (no GUI stuff) - Return port assignation to the standard FLINK one: Variable: docker-flink -> flink taskmanager.rpc.port: 6121 -> 6122 taskmanager.data.port: 6122 -> 6121 jobmanager.web.port: 8080 -> 8081

…cution

…ient too

iemejia · 2016-06-28T13:06:00Z

The docker images script was simplified and the image size was reduced.

Previous image:
flink latest 6475add651c7 24 minutes ago 711.6 MB

Image after FLINK-4118
flink latest 555e60f24c10 20 seconds ago 252.5 MB

iemejia · 2016-06-28T13:17:52Z

Sorry I had to rebase my previous PR but this is the definitive one.

We don't use the conf folder anymore for the docker image.

aljoscha · 2016-06-30T09:35:59Z

flink-contrib/docker-flink/docker-entrypoint.sh

-
-parallelization.degree.default: %parallelism%
+# general configuration
+sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: `grep -c ^processor /proc/cpuinfo`/g" $FLINK_HOME/conf/flink-conf.yaml


This should only be executed on the JobManager since it will print taskmanager_1 | sed: can't create temp file '/usr/local/flink/conf/flink-conf.yamlXXXXXX': Read-only file system which might worry some users.

aljoscha · 2016-06-30T09:37:13Z

Nice work! I didn't know how to use docker but I managed to set it up and use the new version on OS X without a problem. So it seems to work well, and the code is a lot simpler and the image is smaller.

LGTM minus the one comment I had about the config file.

…ager Done as suggested in the Pull Request

iemejia · 2016-06-30T13:52:38Z

Nice, I just fixed as you suggested. I have three questions:

This container is based on the Java JRE (to keep it small), Does Flink in any part do some magic that requires a full JDK (like live recompiles) ? If no, I think this is almost perfect now.
Are you aware of any other flink dependency that uses any native OS library (I ask this because if this is the case it must be added to the container, I did this for snappy because I found the issue while testing a Beam pipeline, but I don't know if there are others).
In the docker image I left supervisor because I didn't find an easy way to start flink in normal mode, the scripts to start both taskmanager and jobmanager go into daemon mode immediately, is there something that can be done to change this (this will reduce the image in 40 more MB), but well I can work on that for a different PR.

aljoscha · 2016-06-30T14:59:23Z

flink-contrib/docker-flink/README.md


-Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive when running containers.
+    docker build -t "flink" flink


On my machine this doesn't work, but this does: docker build -t "flink" . Is this maybe a leftover from the earlier version where there was a flink directory?

(Earlier I was using sh build.sh, that's why I didn't spot the problem.)

Nice catch, just fixed :)

… image.

aljoscha · 2016-06-30T15:07:51Z

Re 1. I think this should be fine. There is some dynamic code generation but this uses Janino as a library so that shouldn't be a problem.

Re 2. I'm not aware of any but it should be easy to add them if we find any in the future.

Re 3. This is not easily possible since the & is hardcoded in flink-daemon.sh:

$JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null &

I'd be very happy if you'd like to work on adding a flag or setting to make the daemons start non-daemonized. It should not be too hard to add that, IMHO.

iemejia · 2016-06-30T15:13:37Z

Awasome, thanks @aljoscha, let's merge !

aljoscha · 2016-06-30T16:52:55Z

One last thing I would like to try is running a job from an existing Flink installation using $FLINK_HOME/bin/flink run -m <jobmanager:port> <your_jar> as described in the README.

I suspect it has something to do with setting the right IP or network setting because Akka is very particular about the IP to which it is bound. Did you get this to work? I'm only managing to access the Web Dashboard.

iemejia · 2016-07-01T11:26:49Z

Hi, I tested it with the basic word count and with the beam pipeline example that @ecesena put for his flink/beam demo.

I don't know if it does not work because you are running on docker for mac, but check two things:

docker assigns an internal IP to each running container (the one corresponding to the jobmanager is the one to put in the -m argument).
The path you pass in the examples must be accesible from the container, e.g. you must copy the file in all the nodes (as I mention in the README), or mount them with a volume (-v on docker).
e.g.

$FLINK_HOME/bin/flink run -m 172.18.0.2:6123 examples/batch/WordCount.jar --input file:///tmp/kinglear.txt

The kinglear file must be in all the nodes. I copy those like this:

for i in $(docker ps --filter name=flink --format={{.ID}}); do                                
    docker cp ~/Desktop/flink-beam/kinglear.txt $i:/tmp/
done

I haven't seen that you can even execute that example without arguments:

bin/flink run -m 172.18.0.2:6123 examples/batch/WordCount.jar

aljoscha · 2016-07-01T13:22:43Z

Yes, this is exactly what I was trying on OS X. I'm quickly setting up a ubuntu VM to see if it works there.

aljoscha · 2016-07-01T14:06:57Z

You were right, I did exactly the same thing I did on OS X on a new Ubuntu 16.04 installation and it worked. 😃

aljoscha · 2016-07-01T14:08:10Z

flink-contrib/docker-flink/README.md


 - Upload a jar to the cluster

-`scp -P 220 <your_jar> root@localhost:/<your_path>`
+        for i in $(docker ps --filter name=flink --format={{.ID}}); do


The jar only needs to be uploaded to the JobManager container, so something like this should suffice:

docker cp <your_jar> $(docker ps --filter name=flink_jobmanager --format={{.ID}}):/<your_path>

Nice I didn't know that flink took care of this, fix in mins.

Yep, the TaskManagers pull it from the JobManager which keeps it in a component called BlobManager.

aljoscha · 2016-07-01T14:11:52Z

I had two more comments about the README but after that it should be good to merge.

…manager

iemejia · 2016-07-01T14:35:06Z

This should be ok now. In further PRs I expect to fix the daemon thing + maybe add a HA version using zookeeper of the docker-compose file.
One more question aljoscha, I intend to add the Beam Flink runner and contribute a similar version into Beam, however I don't know what is the best approach for this, I just tried naively to put the jars in $FLINK_HOME/lib but it didn't work, any ideas ?

aljoscha · 2016-07-01T14:39:01Z

That's great to hear! I'll write something on the Beam ML thread.

iemejia · 2016-07-01T14:49:18Z

Great, thanks for your review.

aljoscha · 2016-07-04T08:23:49Z

I merged it, thanks again for your work!

iemejia added 5 commits June 28, 2016 14:52

[FLINK-4118] Base the image on the official java alpine and remove ssh

04c18ed

[FLINK-4118] Remove unused configuration files and fix README

eff306e

[FLINK-4118] Update compose to v2 + add new entrypoint for direct exe…

e386b5e

…cution

[FLINK-4118] Change default entrypoint so users can run flink as a cl…

34f1884

…ient too

iemejia changed the title ~~Flink 4118~~ [FLINK-4118] The docker-flink image is outdated (1.0.2) and can be slimmed down Jun 28, 2016

[FLINK-4118] Removed reference from master pom file

7a645fc

We don't use the conf folder anymore for the docker image.

aljoscha reviewed Jun 30, 2016
View reviewed changes

[FLINK-4118] Prevent warning when editing read-only config of taskman…

e8e4c54

…ager Done as suggested in the Pull Request

aljoscha reviewed Jun 30, 2016
View reviewed changes

[FLINK-4118] Fix error in the README instructions to build the docker…

50dd78c

… image.

aljoscha reviewed Jul 1, 2016
View reviewed changes

[FLINK-4118] Fix README on how to upload jar + run job from the node …

34b61ea

…manager

asfgit closed this in ffaf10d Jul 4, 2016

iemejia deleted the FLINK-4118 branch July 4, 2016 11:35

rmetzger added the component=<none> label Mar 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-4118] The docker-flink image is outdated (1.0.2) and can be slimmed down #2176

[FLINK-4118] The docker-flink image is outdated (1.0.2) and can be slimmed down #2176

iemejia commented Jun 28, 2016

iemejia commented Jun 28, 2016

iemejia commented Jun 28, 2016

aljoscha Jun 30, 2016

aljoscha commented Jun 30, 2016

iemejia commented Jun 30, 2016 •

edited

aljoscha Jun 30, 2016

iemejia Jun 30, 2016

aljoscha commented Jun 30, 2016

iemejia commented Jun 30, 2016

aljoscha commented Jun 30, 2016

iemejia commented Jul 1, 2016 •

edited

aljoscha commented Jul 1, 2016

aljoscha commented Jul 1, 2016

aljoscha Jul 1, 2016

iemejia Jul 1, 2016

aljoscha Jul 1, 2016

aljoscha commented Jul 1, 2016

iemejia commented Jul 1, 2016 •

edited

aljoscha commented Jul 1, 2016

iemejia commented Jul 1, 2016

aljoscha commented Jul 4, 2016


		Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive when running containers.
		docker build -t "flink" flink

[FLINK-4118] The docker-flink image is outdated (1.0.2) and can be slimmed down #2176

[FLINK-4118] The docker-flink image is outdated (1.0.2) and can be slimmed down #2176

Conversation

iemejia commented Jun 28, 2016

iemejia commented Jun 28, 2016

iemejia commented Jun 28, 2016

aljoscha Jun 30, 2016

Choose a reason for hiding this comment

aljoscha commented Jun 30, 2016

iemejia commented Jun 30, 2016 • edited

aljoscha Jun 30, 2016

Choose a reason for hiding this comment

iemejia Jun 30, 2016

Choose a reason for hiding this comment

aljoscha commented Jun 30, 2016

iemejia commented Jun 30, 2016

aljoscha commented Jun 30, 2016

iemejia commented Jul 1, 2016 • edited

aljoscha commented Jul 1, 2016

aljoscha commented Jul 1, 2016

aljoscha Jul 1, 2016

Choose a reason for hiding this comment

iemejia Jul 1, 2016

Choose a reason for hiding this comment

aljoscha Jul 1, 2016

Choose a reason for hiding this comment

aljoscha commented Jul 1, 2016

iemejia commented Jul 1, 2016 • edited

aljoscha commented Jul 1, 2016

iemejia commented Jul 1, 2016

aljoscha commented Jul 4, 2016

iemejia commented Jun 30, 2016 •

edited

iemejia commented Jul 1, 2016 •

edited

iemejia commented Jul 1, 2016 •

edited