Mesos Containerizer Volumes and Port Mapping

[tkurmann]

So this is more of a question than an issue, but I hope this might help some other people.

My goal is to make a gpu-cluster using mesos/marathon for a small university lab environment, mainly for image processing deep-learning related tasks. it gets messy to have to coordinate who gets which GPU etc. I have the example of https://github.com/klueska-mesosphere/mesos-gpu-docker running, which is very nice.

Now, besides running the container we want to do something with it. Which means we need access to volumes (NFS/CIFS etc.) and need to access the container using SSH/web. From your docs I saw that the feature set of the Mesos Containerizer is limited.

How could I proceed to get port mapping like docker running and mount volumes?

[unterstein]

Hi @tkurmann,

you can use volumes and ports with universal containerizer the same way you would use with docker containerizer. The only limitation currently is, that in universal containerizer you could only use host networking. We are currently working on enabling bridge networking as well.

Volume doc: https://mesosphere.github.io/marathon/docs/persistent-volumes.html
Port doc: https://mesosphere.github.io/marathon/docs/ports.html

Does this answer your question?

Kind regards
Johannes

[tkurmann]

Hi @unterstein

Thank you for your answer. I actually saw those docs after writing the question but decided on leaving the question up as most things are not running as I would expect.

Regarding ports:
At the moment I have it running, but it is a bit hacky. Here is the json
{ "id": "/gpu-test", "cmd": null, "cpus": 10, "mem": 12800, "gpus": 1, "disk": 0, "instances": 1, "container": { "type": "MESOS", "volumes": [], "docker": { "image": "tkurmann/otl-mesos-keras", "credential": null, "forcePullImage": true } }, "portDefinitions": [ { "port": 22, "protocol": "tcp", "labels": {} } ] }

This gives an ENV variable named PORT_22, which I map to the sshd on startup of the docker.
if [ -z ${PORT_22+x} ]; then echo "Port 22"; else sed -i -- 's/Port 22/'"Port $PORT_22"'/g' /etc/ssh/sshd_config ; fi ; service ssh restart ;

All other attempts were not successful. Would you mind maybe making an example?

Regarding the volumes, I managed to mount local volumes. External volumes are failing due to the "name" not being able to contain any . : \. An example:

"volumes": [
      {
        "containerPath": "nfs-volume",
        "external": {
          "name": "192.168.1.100/test",
          "provider": "dvdi",
          "options": { "dvdi/driver": "nfs" }
          },
        "mode": "RW"
      }
]

Thank you very much,

Tom

[Queuecumber]

@unterstein Do you mind elaborating on the plan w.r.t. bridged networking for the mesos containerizer and what the progress has been?

[nfnt]

@Queuecumber The Mesos containerizer gained (experimental) support for port mapping (aka bridged networking) in the recently released Mesos 1.1.0. Marathon 1.4 doesn't support that yet, but it will land in 1.5, see #4548.

[Queuecumber]

@nfnt Thanks a lot for pointing me to that. Do you know what the timeline is for 1.5 or if there is a development branch I can build from to have it now? This feature is critical for my organizations project

[jdef]

@tkurmann I haven't tried getting an nfs volume mounted via the external volume (dvdi) implementation. I'd probably try a volume name like "nfstest" and then specifying additional volume options using option names like dvdi/some-nfs-option-name.

xref moby/moby#25202

[jdef]

@Queuecumber approximate timeline for 1.5 release Q2'17.

i've been hacking on an overhaul of the network API here: https://phabricator.mesosphere.com/D201. it's a bit out of sync w/ master for the moment since i've been distracted by other things. hoping to land these changes in the coming weeks.

[meichstedt]

Note: This issue has been migrated to https://jira.mesosphere.com/browse/MARATHON-1980. For more information see https://groups.google.com/forum/#!topic/marathon-framework/khtvf-ifnp8.