Mesos fail to create docker container of ipython-spark-dockerworker

[luckyfengyong]

The issue actually has been resolved in https://issues.apache.org/jira/browse/MESOS-4587 on Feb 16. Need pick up new Mesos package and rebuild image.

[kfoss]

Thanks for highlighting this. It looks like the current release (0.27.0) doesn't fix the issue yet. Will push update once 0.27.1-rc1 releases

[luckyfengyong]

Thank your team for the great work. I try to develop the same image but found what you did. It is even beyond my expectation :-). The only issue is the size of image, however I could customize the Dockerfile to only keep what I need.

[kfoss]

Appreciate the compliment...glad this helps in your work.

Agree the size of the image is an issue. This project started as a way to interact with the Hadoop ecosystem AND Spark AND the various Python libs we wanted, so it encapsulates a lot that could be cut. We've given thought to adding a way to optionally switch on/off certain packages, but that's difficult unless it's simply removing a "pip install..." line.

Regardless, here's an output of the layer sizes, which show clear step increases that might help target which image to cut:
lab41/spark-mesos-dockerworker-ipython 7.219 GB
lab41/spark-mesos-mesosworker-ipython 7.219 GB
lab41/spark-mesos-base 7.219 GB
lab41/spark-client-ipython 4.329 GB
lab41/spark-worker 4.329 GB
lab41/spark-master 4.329 GB
lab41/spark-base 4.329 GB
lab41/python-llvm 3.982 GB
lab41/python-datatools 3.595 GB
lab41/cdh5-hadoop 2.327 GB
lab41/oracle-jdk7 586.3 MB

[kfoss]

@luckyfengyong latest version includes mesos v0.27.2, which should fix the container creation issue