New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add volumes for docker driver #62

Closed
cbednarski opened this Issue Sep 19, 2015 · 18 comments

Comments

Projects
None yet
@cbednarski
Contributor

cbednarski commented Sep 19, 2015

Similar in theme to #60 and #61, we want to share data between tasks in a group. We will need to mount the task group allocation directory as a docker volume so we can share data between these processes.

@cbednarski cbednarski assigned cbednarski and unassigned cbednarski Sep 24, 2015

@zrml

This comment has been minimized.

Show comment
Hide comment
@zrml

zrml Sep 30, 2015

+1
Fundamental!
Thanks

zrml commented Sep 30, 2015

+1
Fundamental!
Thanks

@crhuber

This comment has been minimized.

Show comment
Hide comment

crhuber commented Oct 12, 2015

+1

@sshi100

This comment has been minimized.

Show comment
Hide comment

sshi100 commented Oct 22, 2015

+1

@dadgar

This comment has been minimized.

Show comment
Hide comment
@dadgar

dadgar Oct 22, 2015

Contributor

This PR closes this #290

Contributor

dadgar commented Oct 22, 2015

This PR closes this #290

@dadgar dadgar closed this Oct 22, 2015

@calvn

This comment has been minimized.

Show comment
Hide comment
@calvn

calvn Dec 22, 2015

Member

Is the doc updated with this? I don't see anything on https://nomadproject.io/docs/drivers/docker.html regarding volumes.

Member

calvn commented Dec 22, 2015

Is the doc updated with this? I don't see anything on https://nomadproject.io/docs/drivers/docker.html regarding volumes.

@diptanu

This comment has been minimized.

Show comment
Hide comment
@diptanu

diptanu Dec 22, 2015

Collaborator

@cleung2010 So this issue was about tasks in a group sharing an ephemeral volume. If you're looking for mounting host volumes in docker containers, that would land in the future.

Collaborator

diptanu commented Dec 22, 2015

@cleung2010 So this issue was about tasks in a group sharing an ephemeral volume. If you're looking for mounting host volumes in docker containers, that would land in the future.

@c4milo

This comment has been minimized.

Show comment
Hide comment
@c4milo

c4milo Dec 22, 2015

Contributor

@diptanu do you mean we cannot mount parts of the host's filesystem into Docker containers right now?

Contributor

c4milo commented Dec 22, 2015

@diptanu do you mean we cannot mount parts of the host's filesystem into Docker containers right now?

@calvn

This comment has been minimized.

Show comment
Hide comment
@calvn

calvn Dec 22, 2015

Member

@c4milo I think that's the case, don't see that in the docs or source. I believe @diptanu's response confirms it.

Member

calvn commented Dec 22, 2015

@c4milo I think that's the case, don't see that in the docs or source. I believe @diptanu's response confirms it.

@c4milo

This comment has been minimized.

Show comment
Hide comment
@c4milo

c4milo Dec 22, 2015

Contributor

@cleung2010 I believe you can pass -v arguments to the docker command, but I haven't tested it to be really sure. That's why I'm asking, I was under the impression this was possible today.

Contributor

c4milo commented Dec 22, 2015

@cleung2010 I believe you can pass -v arguments to the docker command, but I haven't tested it to be really sure. That's why I'm asking, I was under the impression this was possible today.

@c4milo

This comment has been minimized.

Show comment
Hide comment
@c4milo

c4milo Dec 22, 2015

Contributor

ah nevermind, it is intended for the actual binary run with the container. So, it is not possible.

Contributor

c4milo commented Dec 22, 2015

ah nevermind, it is intended for the actual binary run with the container. So, it is not possible.

@diptanu

This comment has been minimized.

Show comment
Hide comment
@diptanu

diptanu Dec 23, 2015

Collaborator

Ok so we currently create an allocation directory on the host and mount it on every container, so every container gets a host volume for writing data. But it is currently not possible to mount an arbitrary path on the host on a container.

This involves allowing operators to configure which paths on the host is allowed for mounting and which tasks can mount which paths on the host, etc. We want the design to be right so we haven't done anything on this yet. But this is something we understand is very important, so I think it would be nice to just down a bunch of use cases that would need us to mount host volumes.

Collaborator

diptanu commented Dec 23, 2015

Ok so we currently create an allocation directory on the host and mount it on every container, so every container gets a host volume for writing data. But it is currently not possible to mount an arbitrary path on the host on a container.

This involves allowing operators to configure which paths on the host is allowed for mounting and which tasks can mount which paths on the host, etc. We want the design to be right so we haven't done anything on this yet. But this is something we understand is very important, so I think it would be nice to just down a bunch of use cases that would need us to mount host volumes.

@c4milo

This comment has been minimized.

Show comment
Hide comment
@c4milo

c4milo Dec 23, 2015

Contributor

@diptanu, so my current use case is being able to access the host's Consul agent from containers. I was thinking in sharing the directory where unix sockets are being created so containers can use them. Unfortunately, the current allocation directory implementation does not seem to help with this specific use case as it purges the allocation directory shared among the containers.

Contributor

c4milo commented Dec 23, 2015

@diptanu, so my current use case is being able to access the host's Consul agent from containers. I was thinking in sharing the directory where unix sockets are being created so containers can use them. Unfortunately, the current allocation directory implementation does not seem to help with this specific use case as it purges the allocation directory shared among the containers.

@diptanu

This comment has been minimized.

Show comment
Hide comment
@diptanu

diptanu Dec 24, 2015

Collaborator

@c4milo Yeah we have been wondering how Tasks which have network isolation could talk to Consul, I guess that would be a good way to implement this.

Collaborator

diptanu commented Dec 24, 2015

@c4milo Yeah we have been wondering how Tasks which have network isolation could talk to Consul, I guess that would be a good way to implement this.

@erich-comsIO

This comment has been minimized.

Show comment
Hide comment
@erich-comsIO

erich-comsIO Dec 25, 2015

I've been using cephfs to persist and share volumes across containers using host volume mounts when starting docker. For example, I bring up containers and mount users' home directories, and that's very important for my application.

Should I be mounting cephfs volumes within the containers themselves? That would (I think) require the containers run in privileged mode which would seem otherwise unnecessary.

I've been using cephfs to persist and share volumes across containers using host volume mounts when starting docker. For example, I bring up containers and mount users' home directories, and that's very important for my application.

Should I be mounting cephfs volumes within the containers themselves? That would (I think) require the containers run in privileged mode which would seem otherwise unnecessary.

@ketzacoatl

This comment has been minimized.

Show comment
Hide comment
@ketzacoatl

ketzacoatl Dec 25, 2015

Contributor

I have a simple use case I'm sure lots of other people do: I'd like to run a docker task which mounts a volume as a path from the host. For example, in my case, a configuration management or logging agent would need access to the underlying host filesystem.

I have a lot of respect for Hashicorp and the projects you've collectively put forth, and I am often very grateful for the patience and care put into the abstractions, interfaces, etc, that are put to use.. in many respects, that is what sets Hashicorp apart from the others.. but this situation here seems counter-intuitive.

Please confirm, is our workaround for the moment to enable to raw_exec driver so we can run docker run .... -v ... ....? Seems silly to do that instead of using the docker driver directly, and only because there's no easy way to tell a task "add this filesystem path as a volume".

Contributor

ketzacoatl commented Dec 25, 2015

I have a simple use case I'm sure lots of other people do: I'd like to run a docker task which mounts a volume as a path from the host. For example, in my case, a configuration management or logging agent would need access to the underlying host filesystem.

I have a lot of respect for Hashicorp and the projects you've collectively put forth, and I am often very grateful for the patience and care put into the abstractions, interfaces, etc, that are put to use.. in many respects, that is what sets Hashicorp apart from the others.. but this situation here seems counter-intuitive.

Please confirm, is our workaround for the moment to enable to raw_exec driver so we can run docker run .... -v ... ....? Seems silly to do that instead of using the docker driver directly, and only because there's no easy way to tell a task "add this filesystem path as a volume".

@zapman449

This comment has been minimized.

Show comment
Hide comment
@zapman449

zapman449 Apr 18, 2016

Another use case: Volumes are frequently how secure details like Certificates are injected into web servers. Building the cert into the docker image is terrible. And a plethora of little loader scrips as the entry point into said images is an anti-pattern and should be avoided where possible. (loader script to go retrieve a cert from )

Another use case: Volumes are frequently how secure details like Certificates are injected into web servers. Building the cert into the docker image is terrible. And a plethora of little loader scrips as the entry point into said images is an anti-pattern and should be avoided where possible. (loader script to go retrieve a cert from )

@dvusboy

This comment has been minimized.

Show comment
Hide comment
@dvusboy

dvusboy Apr 18, 2016

Contributor

Yet one more: There are some system services that I'd like to use that would preferably be launched as containers. Some of them require access to the engine socket. The typical way to handle this with --volume=/var/run/docker.sock:/var/run/docker.sock. There is currently no way to do this with nomad. Neither exec nor raw_exec is acceptable alternative: Do you run the container attached, for example?

Contributor

dvusboy commented Apr 18, 2016

Yet one more: There are some system services that I'd like to use that would preferably be launched as containers. Some of them require access to the engine socket. The typical way to handle this with --volume=/var/run/docker.sock:/var/run/docker.sock. There is currently no way to do this with nomad. Neither exec nor raw_exec is acceptable alternative: Do you run the container attached, for example?

@ketzacoatl

This comment has been minimized.

Show comment
Hide comment
@ketzacoatl

ketzacoatl Apr 19, 2016

Contributor

In most cases, it's hard to say raw_exec would not be sufficient, though with docker it means you need to create a wrapper script to do it correctly. The general flow in the wrapper is to:
a) remove the named container if one exists (the script should || true or similar to avoid exiting if the named container does not exist)
b) create a new named container
c) start that named container in daemon mode, then
d) detached, run docker logs with "follow" (similar to tail -f), the script should exist if that docker logs exits (which means the container failed/stopped)

This follows the recommendations from docker upstream, see moby/moby#6791 for more details on that.

Contributor

ketzacoatl commented Apr 19, 2016

In most cases, it's hard to say raw_exec would not be sufficient, though with docker it means you need to create a wrapper script to do it correctly. The general flow in the wrapper is to:
a) remove the named container if one exists (the script should || true or similar to avoid exiting if the named container does not exist)
b) create a new named container
c) start that named container in daemon mode, then
d) detached, run docker logs with "follow" (similar to tail -f), the script should exist if that docker logs exits (which means the container failed/stopped)

This follows the recommendations from docker upstream, see moby/moby#6791 for more details on that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment