Proposal: Remote Shared Volumes #7249

Closed
vieux opened this Issue Jul 25, 2014 · 56 comments

Projects

None yet
@vieux
Member
vieux commented Jul 25, 2014

Remote Shared Volumes

Currently we use volumes with -v
You have either regular volumes -v /container/some/path to put some folder in a volume, stored in /var/lib/docker and shared volumes -v /host/some/path:/container/some/path that use bind-mount to share some folder from your host to your container.

You can prepend :ro or :rw to set the mode.

Obviously bind mount can't work on a remote machine, so if your client is on OSX and your daemon on boot2docker, -v /host/some/path:/container/some/path will look for /host/some/path on the boot2docker host, not you mac.

Proposal

Add a driver to the volumes to specify how to handle the shared volumes.
We would have 2 drivers for now:

  • mount as it is today:

    -v mount:/host/some/path:/container/some/path

  • fuse a new drive that uses fuse to share your volumes remotely:

    -v fuse:/host/some/path:/container/some/path

Note: if you do don't specify any driver, docker will try mount and if it cannot find the host folder, it'll default to fuse

To minimise the changes, for fuse we will create the fuse connection in /var/lib/docker/fuse/{fuseid} and then bind mount /var/lib/docker/fuse/{fuseid} inside the container as we do today.

Implementation

To not have any issue with firewall or complicated network setup the server has to instantiate the connection.

In /container/{name:.*}/start the daemon will create the fuse connection, associate this connection with a fuseId and return this fuseId
Then the client will call /fuse/attach/{fuseId}, the daemon will hijack the connection connect both ends together.

Dependencies

Regarding the code, github.com/hanwen/go-fuse, code.google.com/p/goprotobuf and pkg/vfuse made by @bradfitz will be added.

Regarding the runtime there is no new dependency.

POC

You can look here for a POC bradfitz@3b91411

@vieux
Member
vieux commented Jul 25, 2014

ping @cpuguy83

@tianon
Member
tianon commented Jul 25, 2014

Why temporary mounts in /tmp and not just inside /var/lib/docker somewhere specific so it's all managed together with the rest of our daemon files?

@vieux
Member
vieux commented Jul 25, 2014

@tianon 👍

@thaJeztah
Member

Like this idea!

Would it make sense to use file:// for mounting local paths, and other protocols (fuse://, maybe even smb://) ?

Or am I completely misinterpreting the description?

@tianon
Member
tianon commented Jul 25, 2014

❤️

So, this means I could technically mix-and-match, right? ie, from a mac connected to b2d, I could docker run -v mount:/some/b2d/stuff:... -v fuse:/Users/tianon/mycrappyproject:... ? 😄

@vieux
Member
vieux commented Jul 25, 2014

@thaJeztah I'm open to suggestions! you read right.

@tianon yes ❤️

@thaJeztah
Member

@vieux Love it!

@vishh
Contributor
vishh commented Jul 26, 2014

+1

@vieux is Dockerd acting as the fuse daemon for all the containers?

@cpuguy83
Contributor

Ok, so:

  1. FUSE gets installed on the docker host
  2. docker run -v fuse:/foo/bar:/foo
  3. daemon mounts /foo/bar from the client into /var/lib/docker/(some place for mounts)/
  4. daemon bind-mounts /var/lib/docker/(some place for mounts)/ into the container at /foo

Is this correct?

@vieux
Member
vieux commented Jul 28, 2014

@vishh only for the one with -v fuse:

@cpuguy83 what do you mean by FUSE gets installed, the kernel module ? Otherwise yes

@nathanleclaire
Contributor

👍

@vishh
Contributor
vishh commented Jul 28, 2014

@vieux: To avoid Dockerd becoming a potential bottleneck, would spawning a per-container FUSE daemon inside the user's container make sense?

@vieux
Member
vieux commented Jul 30, 2014

@vishh yes you are right, at least in another process

@SvenDowideit
Collaborator

we were bashing this around a bit in the dev meeting

and would like to consider abstracting things a little.
For example:

#on the daemon start
docker -d --volumes-base name:cifs://host/User/sven/docker-dir:username:sven,password:banana,crypt:ntlmsse,uid:100 ...

#on the client side, anywhere
docker run -v name:subdir:/containerdir ...

At this point, I argue strongly to only allow a user to automatically mount a subdir of their user dir - so we don't have user's private gpg keys etc open to the b2d vm by default.

So what I was going to propose is that boot2docker-cli 1.3.0 will auto create ~/docker-share and then auto create a samba share for it on both OS X and Windows, and then on bootup, boot2docker would start its daemon with the necessary -volumes-base. This obviously depends more on the timeline of the fuse implementation..

I am assuming that by requiring the mapping to be configured on the daemon side, we can mitigate some security issues - ( @vieux the --volumes-base mapping could become some optional config for this proposal :) )

@vieux please tell me if I should make a separate proposal, or if you'd like to merge some parts together already.

@SvenDowideit
Collaborator

I must point out one awesome feature of the fuse proposal - it will also work from inside a container.

so if you an image that uses a docker.sock and client, the container can -v mount its own dir into another container. This is horribly complicated to do using a separate NFS. Though probably better and faster if the daemon could detect it and use a local bind-mount.

nice.

@thaJeztah
Member

@SvenDowideit I agree that including username/password etc in the URLs is bad.

However, I think the regular way to specify them is;

cifs://username:password@host/User/sven/docker-dir

Just like, for example, a ftp connection;

ftp://username:password@ftp.example.com
@SvenDowideit
Collaborator

@thaJeztah I don't think I can say anything positive about leaving my password in my process list like that - and that hurts with specifying it on the daemon too.

the Example I've given for --volumes-base uses the same -o options list that are used when calling mount - I'm being very unimaginative :) (and alot more than username and password are needed to get linux mount to talk to OSX's samba)

@thaJeztah
Member

@SvenDowideit ah, I see. Missed the huge list of options following password (had to scroll the example to the right).

I thought the format could match the 'regular' format as is used, for example, when using "connect to server" (CMD+K) on OS X. Having to specify that amount of options is very unfortunate 😿

@vieux
Member
vieux commented Jul 30, 2014

@SvenDowideit I think it should be in a separate proposal, it could apply to all volumes, not only remote right ?

@SvenDowideit
Collaborator

my comment is mostly about defining a syntax, the implementation could still work the same way - we just abstract out the protocol, user&pass, and other mount options (and on OSX there are very many), simplifying the user's commandline, and allowing the options to change without the user experience changing.

for eg

docker run -v user:/src/docker:/src docker-dev will work, and the user alias could be

  • /home/sven (ie, local to the Docker daemon)
  • fuse://host/some/path - and the client and server have to talk :/ (ok, this is a pain point)
  • cifs://host/share -o rw,relatime,vers=1.0,sec=ntlmssp,cache=strict,username=sven,domain=SVENS-MAC-MINI,uid=1000,forceuid,gid=100,forcegid,addr=10.10.10.14,file_mode=0755,dir_mode=0755,nounix,noperm,rsize=61440,wsize=65536,actimeo=1 - ok, sorry, its in a different format, and not all apply
  • nfs://....

On Boot2Docker, the manage tool will create a share on init, and then pass that info to the vm at boot time - so adding an abstracted user:/path which defaults to client $HOME would help the Docs.

Yes, I will move this to a separate proposal later :)

@sherter
sherter commented Jul 31, 2014

I don't think the suggested mount:/ and fuse:/ syntax is good to differentiate between mounting from docker client and mounting from docker host. It doesn't say anything about where the data is coming from. Instead, it just says what system is used (what I probably don't care about at all).

When using the nfs:// smb:// cifs:// syntax I also can not differentiate between mounting this folders directly in docker host -> mount or from docker client -> fuse -> mount

So I would suggest using a syntax like

  • -v dockerclient:/some/path:/container/some/path
  • -v dockerclient:nfs://some/path:/container/some/path (which uses fuse)
  • -v dockerhost:nfs://some/path:/container/some/path (nfs connection happens within docker host)

Of course dockerclient could be simplified (to only client or something else) and dockerhost could be default value. It is just more explicit about where the data comes from.

@mrmaxon
mrmaxon commented Jul 31, 2014

I agree with fuse:// not beeing very precise but i don't think that you'll need to prefix host or client.
As far as I understand.. this feature will only be used in development environments and thus the network address being used should identify whether you are accessing host or client (both on the same network).

Correct me if I am wrong.

@sherter
sherter commented Jul 31, 2014

How would you tell docker (the command) which path to mount in a container, if it exists on both machines, the client machine (where you run the command docker) and the machine, where the docker daemon runs?

The example with paths from external machines is maybe a little confusing.
Consider the path /, which is very likely to exist on both systems, the client machine (running docker command) and host machine (running docker daemon).

I would suggest to differentiate between them like so:

  • -v client:/:/container/mnt/
  • -v daemon:/:/container/mnt/ or with defaulting to daemon: -v /:/container/mnt/

If I understand you right, you would use IP addresses/DNS names? Something like -v 127.0.0.1:/:/container/mnt/? Which localhost would that be, the machine running the daemon or the machine running docker command?

@mrmaxon
mrmaxon commented Jul 31, 2014

Well with nfs it's usually somethig like nfs://192.168.1.10/path/to/folder and the ip would identify the target.

@sherter
sherter commented Jul 31, 2014

So you would need to expose everything you want to mount in a container as a network share. That's exactly what we want to avoid with the fuse system.

@mrmaxon
mrmaxon commented Jul 31, 2014

Ok. I don't know much about fuse. As far as I understand it's a tool/sdk to create virtual file systems (by mounting directories) using various protocols (e.g. sshfs).

So what you want to do is:

  • use -v client:/absolute/path:/container/mnt
  • let the docker-cli share that directory for you
  • mount the shared directory to the host
  • mount that directory from the host to the container

Is that right?

@sherter
sherter commented Jul 31, 2014

right. see issue #4023 where this is discussed in detail

@cpuguy83
Contributor

I like "fuse" here as it is explicit about what is happening, and we assume since no address was put it we want the address of the calling client.

This also leaves it open for supporting more later on.

@jpetazzo
Contributor
jpetazzo commented Aug 6, 2014

I've mentioned it a couple of times on IRC, so I'd like to make sure that it gets documented somewhere: did you consider using 9p? (see https://gist.github.com/jpetazzo/5668338 for an example)

I got it working very easily on Linux, and within a matter of hours on OS X (to compile the diod server).

@sherter
sherter commented Aug 6, 2014

As far as I understand this proposal, the whole idea is to have no additional requirements on the client side. If I got it right, your approach requires an implementation of the 9p protocol on the client side. Is there one, for example, for windows?

So you transform server side dependencies to client side dependencies, don't you? Correct me, if I am wrong!

@freebroccolo

@jpetazzo the diod/9p approach sounds interesting but I'm not sure it's a particularly usable solution on OS X at the moment. I tried following your instructions but wasn't able to get it working on OS X, even though diod did compile and seemingly run after splicing the appropriate branches. I don't remember exactly what the problem was anymore but IIRC I was never able to properly connect to the server (I tried --allsquash and --squashuser nobody). Maybe you could give more detail on how you managed to get it working.

@c3s4r
c3s4r commented Aug 12, 2014

Sorry to bother you guys... This is a much needed feature. Any ideas when this could be available?

@xeor
xeor commented Aug 12, 2014

Agree.. This is something most of us are waiting for.

I know you want this to be implemented the correct way. But this is a huge show stopper for many projects.

I've introduced Docker to a handfull of people, and all of them gets the volume mounting wrong (since they are using OSX and boot2docker).

@jpetazzo
Contributor

@sherter: I'm assuming that it will be pretty straightforward to port diod on Windows (since it was pretty easy to get it working on a Mac). I could be wrong, though.

@darinmorrison: I'll try to give more details when I have some time (+a mac :-)). My main goal was to make sure that this solution was not overlooked.

@hunter
hunter commented Aug 14, 2014

Exciting. I assume this would mean that Ceph or Gluster disks could be mounted/unmounted as part of the container lifecycle

@pr3d4t0r

My team relies on Docker for production deployments to non-Linux machines via boot2docker. This proposal doesn't appear to address the elephant in the room: how would it mount the host's file system, not boot2docker's, with ease and without kludges? The Fuse protocol might work but a simpler solution for this would be to include the complete VirtualGuest additions and a well-documented path for "external" mounts, so that these can be mapped/shared with the container.

We've figured out ways of auto-mounting via -v, sshfs, CIFS, and VirtualGuest additions, and they all have quirks because Docker just doesn't support it and solutions have to be engineered to work around Docker's lack of features.

Fuse sounds like a great idea, perhaps included in some base image, and somehow mapped to a command line option (e.g. boot2docker --hostVolumes="/Users/foo,/Volumes/blah") that handles all the mapping when docker run -v is executed.

We're happy with local volume handling in Docker/Linux. Pushing the non-Linux environment setup to boot2docker options might be a great way to avoid adding complexity to Docker itself, and at the same time demarcate the code that deals with this situation.

Thoughts?

@SvenDowideit
Collaborator

What happens if there are network issues between the Docker daemon and the system from which the fuse mount is made? does the container stop, freeze, or is there a policy to deal with it?

@philippmuller

I've also run into the same issue on OSX. Any progress regarding this would be greatly appreciated.

@tianon tianon referenced this issue in boot2docker/boot2docker Sep 11, 2014
Merged

VirtualBox Guest Additions #534

@gaastonsr

Can we know how progress is going on? I'm excited for this.

@cpuguy83
Contributor

@gaastonsr the vbox shared folder modules have been temporarily included into boot2docker until this is ready.

Last I heard there was still work on the fuse implementation.
bradfitz@3b91411

This was referenced Oct 30, 2014
@kshlm
kshlm commented Nov 13, 2014

@SvenDowideit Could you start the new proposal for the syntax? If we can arrive on a syntax earlier, different drivers like the proposed Gluster #8881 and Ceph #8854 drivers could be implemented.

The discussion here is mainly around how a local docker client directory can be shared with a remote docker daemon, which isn't necessarily concerned with the syntax for the achieving the same.

@SvenDowideit
Collaborator

@kshlm yes, sorry, I will write something up this week if I can - I'm in SF, and was talking to Solomon about it, and he's given me some really cool ideas to combine with the docker volumes stuff.

@mk-qi
mk-qi commented Jan 23, 2015

+1

@thaJeztah
Member

(@vieux, @SvenDowideit, @icecrime) I noticed there are (afaik) now four separate proposals to add support for (named and/or remote) volumes; #7249 (this one), #9250, #9803, and #8424.

Three of them are created by docker maintainers; I think one should be picked to proceed with, to prevent the discussion to be spread over three separate (but related/competing) proposals?

Also related; #7127 (comment) (using different drivers for volumes, e.g. ceph, gluster, raw block devices)

@nathanleclaire
Contributor

I think Docker volumes (esp. -v) are too overloaded as it is. The direction which I envision for the future is separating "local volume" (which is what you always get when you do a bind mount, keeping that abstraction in its current state) from "way to connect my local computer's files with that remote volume" (and then that could be managed with NFS et al. or Docker-specific tools).

@thaJeztah
Member

I agree, better not add more functionality to -v and use volume management functions. #9803 and #9250 are more in that direction.

@erSitzt
erSitzt commented Feb 23, 2015

👍

@DanTheColoradan

👍

@resouer
Contributor
resouer commented Mar 26, 2015

Is this still moving forward? As we want to implement glusterfs driver which need support of this PR ...

@cybertk
cybertk commented Apr 10, 2015

+1, it's useful to create a DFS based data volume container, all persistent data are stored in remote DFS

@SerCeMan

👍

@noetix
noetix commented May 26, 2015

+1

@cpuguy83
Contributor

Hi all. Docker 1.7 will include a re-architected volume subsystem which now uses a driver-based approach, and the experimental channel includes support for out-of-process volume plugins.

Here is a demo of a docker plugin for flocker: https://www.youtube.com/watch?v=wOMRV1JI5Lo

@h0tbird
h0tbird commented Jul 30, 2015

+1 nice demo. I can't wait to add --volume-driver=ceph to my containers.

@dvc94ch dvc94ch referenced this issue in teamartemis/foreverboards Aug 12, 2015
Closed

Bug upload service #77

@geovanisouza92

👍

@jessfraz jessfraz removed the kind/proposal label Sep 8, 2015
@calavera
Contributor

Closing this as done. We already shipped volume plugins in 1.8 and there is a dedicated subcommand coming in 1.9 and there are already several out there. I believe there are even two different plugins for ceph.

We can build from that if we need additional features.

@calavera calavera closed this Sep 11, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment