New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Remote Shared Volumes #7249

Closed
vieux opened this Issue Jul 25, 2014 · 56 comments

Comments

Projects
None yet
@vieux
Copy link
Collaborator

vieux commented Jul 25, 2014

Remote Shared Volumes

Currently we use volumes with -v
You have either regular volumes -v /container/some/path to put some folder in a volume, stored in /var/lib/docker and shared volumes -v /host/some/path:/container/some/path that use bind-mount to share some folder from your host to your container.

You can prepend :ro or :rw to set the mode.

Obviously bind mount can't work on a remote machine, so if your client is on OSX and your daemon on boot2docker, -v /host/some/path:/container/some/path will look for /host/some/path on the boot2docker host, not you mac.

Proposal

Add a driver to the volumes to specify how to handle the shared volumes.
We would have 2 drivers for now:

  • mount as it is today:

    -v mount:/host/some/path:/container/some/path

  • fuse a new drive that uses fuse to share your volumes remotely:

    -v fuse:/host/some/path:/container/some/path

Note: if you do don't specify any driver, docker will try mount and if it cannot find the host folder, it'll default to fuse

To minimise the changes, for fuse we will create the fuse connection in /var/lib/docker/fuse/{fuseid} and then bind mount /var/lib/docker/fuse/{fuseid} inside the container as we do today.

Implementation

To not have any issue with firewall or complicated network setup the server has to instantiate the connection.

In /container/{name:.*}/start the daemon will create the fuse connection, associate this connection with a fuseId and return this fuseId
Then the client will call /fuse/attach/{fuseId}, the daemon will hijack the connection connect both ends together.

Dependencies

Regarding the code, github.com/hanwen/go-fuse, code.google.com/p/goprotobuf and pkg/vfuse made by @bradfitz will be added.

Regarding the runtime there is no new dependency.

POC

You can look here for a POC bradfitz@3b91411

@vieux

This comment has been minimized.

Copy link
Collaborator

vieux commented Jul 25, 2014

ping @cpuguy83

@tianon

This comment has been minimized.

Copy link
Member

tianon commented Jul 25, 2014

Why temporary mounts in /tmp and not just inside /var/lib/docker somewhere specific so it's all managed together with the rest of our daemon files?

@vieux

This comment has been minimized.

Copy link
Collaborator

vieux commented Jul 25, 2014

@tianon 👍

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jul 25, 2014

Like this idea!

Would it make sense to use file:// for mounting local paths, and other protocols (fuse://, maybe even smb://) ?

Or am I completely misinterpreting the description?

@tianon

This comment has been minimized.

Copy link
Member

tianon commented Jul 25, 2014

❤️

So, this means I could technically mix-and-match, right? ie, from a mac connected to b2d, I could docker run -v mount:/some/b2d/stuff:... -v fuse:/Users/tianon/mycrappyproject:... ? 😄

@vieux

This comment has been minimized.

Copy link
Collaborator

vieux commented Jul 25, 2014

@thaJeztah I'm open to suggestions! you read right.

@tianon yes ❤️

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jul 25, 2014

@vieux Love it!

@vishh

This comment has been minimized.

Copy link
Contributor

vishh commented Jul 26, 2014

+1

@vieux is Dockerd acting as the fuse daemon for all the containers?

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Jul 26, 2014

Ok, so:

  1. FUSE gets installed on the docker host
  2. docker run -v fuse:/foo/bar:/foo
  3. daemon mounts /foo/bar from the client into /var/lib/docker/(some place for mounts)/
  4. daemon bind-mounts /var/lib/docker/(some place for mounts)/ into the container at /foo

Is this correct?

@vieux

This comment has been minimized.

Copy link
Collaborator

vieux commented Jul 28, 2014

@vishh only for the one with -v fuse:

@cpuguy83 what do you mean by FUSE gets installed, the kernel module ? Otherwise yes

@nathanleclaire

This comment has been minimized.

Copy link
Contributor

nathanleclaire commented Jul 28, 2014

👍

@vishh

This comment has been minimized.

Copy link
Contributor

vishh commented Jul 28, 2014

@vieux: To avoid Dockerd becoming a potential bottleneck, would spawning a per-container FUSE daemon inside the user's container make sense?

@vieux

This comment has been minimized.

Copy link
Collaborator

vieux commented Jul 30, 2014

@vishh yes you are right, at least in another process

@SvenDowideit

This comment has been minimized.

Copy link
Contributor

SvenDowideit commented Jul 30, 2014

we were bashing this around a bit in the dev meeting

and would like to consider abstracting things a little.
For example:

#on the daemon start
docker -d --volumes-base name:cifs://host/User/sven/docker-dir:username:sven,password:banana,crypt:ntlmsse,uid:100 ...

#on the client side, anywhere
docker run -v name:subdir:/containerdir ...

At this point, I argue strongly to only allow a user to automatically mount a subdir of their user dir - so we don't have user's private gpg keys etc open to the b2d vm by default.

So what I was going to propose is that boot2docker-cli 1.3.0 will auto create ~/docker-share and then auto create a samba share for it on both OS X and Windows, and then on bootup, boot2docker would start its daemon with the necessary -volumes-base. This obviously depends more on the timeline of the fuse implementation..

I am assuming that by requiring the mapping to be configured on the daemon side, we can mitigate some security issues - ( @vieux the --volumes-base mapping could become some optional config for this proposal :) )

@vieux please tell me if I should make a separate proposal, or if you'd like to merge some parts together already.

@SvenDowideit

This comment has been minimized.

Copy link
Contributor

SvenDowideit commented Jul 30, 2014

I must point out one awesome feature of the fuse proposal - it will also work from inside a container.

so if you an image that uses a docker.sock and client, the container can -v mount its own dir into another container. This is horribly complicated to do using a separate NFS. Though probably better and faster if the daemon could detect it and use a local bind-mount.

nice.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jul 30, 2014

@SvenDowideit I agree that including username/password etc in the URLs is bad.

However, I think the regular way to specify them is;

cifs://username:password@host/User/sven/docker-dir

Just like, for example, a ftp connection;

ftp://username:password@ftp.example.com
@SvenDowideit

This comment has been minimized.

Copy link
Contributor

SvenDowideit commented Jul 30, 2014

@thaJeztah I don't think I can say anything positive about leaving my password in my process list like that - and that hurts with specifying it on the daemon too.

the Example I've given for --volumes-base uses the same -o options list that are used when calling mount - I'm being very unimaginative :) (and alot more than username and password are needed to get linux mount to talk to OSX's samba)

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jul 30, 2014

@SvenDowideit ah, I see. Missed the huge list of options following password (had to scroll the example to the right).

I thought the format could match the 'regular' format as is used, for example, when using "connect to server" (CMD+K) on OS X. Having to specify that amount of options is very unfortunate 😿

@vieux

This comment has been minimized.

Copy link
Collaborator

vieux commented Jul 30, 2014

@SvenDowideit I think it should be in a separate proposal, it could apply to all volumes, not only remote right ?

@SvenDowideit

This comment has been minimized.

Copy link
Contributor

SvenDowideit commented Jul 31, 2014

my comment is mostly about defining a syntax, the implementation could still work the same way - we just abstract out the protocol, user&pass, and other mount options (and on OSX there are very many), simplifying the user's commandline, and allowing the options to change without the user experience changing.

for eg

docker run -v user:/src/docker:/src docker-dev will work, and the user alias could be

  • /home/sven (ie, local to the Docker daemon)
  • fuse://host/some/path - and the client and server have to talk :/ (ok, this is a pain point)
  • cifs://host/share -o rw,relatime,vers=1.0,sec=ntlmssp,cache=strict,username=sven,domain=SVENS-MAC-MINI,uid=1000,forceuid,gid=100,forcegid,addr=10.10.10.14,file_mode=0755,dir_mode=0755,nounix,noperm,rsize=61440,wsize=65536,actimeo=1 - ok, sorry, its in a different format, and not all apply
  • nfs://....

On Boot2Docker, the manage tool will create a share on init, and then pass that info to the vm at boot time - so adding an abstracted user:/path which defaults to client $HOME would help the Docs.

Yes, I will move this to a separate proposal later :)

@sherter

This comment has been minimized.

Copy link

sherter commented Jul 31, 2014

I don't think the suggested mount:/ and fuse:/ syntax is good to differentiate between mounting from docker client and mounting from docker host. It doesn't say anything about where the data is coming from. Instead, it just says what system is used (what I probably don't care about at all).

When using the nfs:// smb:// cifs:// syntax I also can not differentiate between mounting this folders directly in docker host -> mount or from docker client -> fuse -> mount

So I would suggest using a syntax like

  • -v dockerclient:/some/path:/container/some/path
  • -v dockerclient:nfs://some/path:/container/some/path (which uses fuse)
  • -v dockerhost:nfs://some/path:/container/some/path (nfs connection happens within docker host)

Of course dockerclient could be simplified (to only client or something else) and dockerhost could be default value. It is just more explicit about where the data comes from.

@ghost

This comment has been minimized.

Copy link

ghost commented Jul 31, 2014

I agree with fuse:// not beeing very precise but i don't think that you'll need to prefix host or client.
As far as I understand.. this feature will only be used in development environments and thus the network address being used should identify whether you are accessing host or client (both on the same network).

Correct me if I am wrong.

@sherter

This comment has been minimized.

Copy link

sherter commented Jul 31, 2014

How would you tell docker (the command) which path to mount in a container, if it exists on both machines, the client machine (where you run the command docker) and the machine, where the docker daemon runs?

The example with paths from external machines is maybe a little confusing.
Consider the path /, which is very likely to exist on both systems, the client machine (running docker command) and host machine (running docker daemon).

I would suggest to differentiate between them like so:

  • -v client:/:/container/mnt/
  • -v daemon:/:/container/mnt/ or with defaulting to daemon: -v /:/container/mnt/

If I understand you right, you would use IP addresses/DNS names? Something like -v 127.0.0.1:/:/container/mnt/? Which localhost would that be, the machine running the daemon or the machine running docker command?

@ghost

This comment has been minimized.

Copy link

ghost commented Jul 31, 2014

Well with nfs it's usually somethig like nfs://192.168.1.10/path/to/folder and the ip would identify the target.

@sherter

This comment has been minimized.

Copy link

sherter commented Jul 31, 2014

So you would need to expose everything you want to mount in a container as a network share. That's exactly what we want to avoid with the fuse system.

@ghost

This comment has been minimized.

Copy link

ghost commented Jul 31, 2014

Ok. I don't know much about fuse. As far as I understand it's a tool/sdk to create virtual file systems (by mounting directories) using various protocols (e.g. sshfs).

So what you want to do is:

  • use -v client:/absolute/path:/container/mnt
  • let the docker-cli share that directory for you
  • mount the shared directory to the host
  • mount that directory from the host to the container

Is that right?

@sherter

This comment has been minimized.

Copy link

sherter commented Jul 31, 2014

right. see issue #4023 where this is discussed in detail

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Jul 31, 2014

I like "fuse" here as it is explicit about what is happening, and we assume since no address was put it we want the address of the calling client.

This also leaves it open for supporting more later on.

@mk-qi

This comment has been minimized.

Copy link

mk-qi commented Jan 23, 2015

+1

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jan 23, 2015

(@vieux, @SvenDowideit, @icecrime) I noticed there are (afaik) now four separate proposals to add support for (named and/or remote) volumes; #7249 (this one), #9250, #9803, and #8424.

Three of them are created by docker maintainers; I think one should be picked to proceed with, to prevent the discussion to be spread over three separate (but related/competing) proposals?

Also related; #7127 (comment) (using different drivers for volumes, e.g. ceph, gluster, raw block devices)

@nathanleclaire

This comment has been minimized.

Copy link
Contributor

nathanleclaire commented Jan 23, 2015

I think Docker volumes (esp. -v) are too overloaded as it is. The direction which I envision for the future is separating "local volume" (which is what you always get when you do a bind mount, keeping that abstraction in its current state) from "way to connect my local computer's files with that remote volume" (and then that could be managed with NFS et al. or Docker-specific tools).

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jan 23, 2015

I agree, better not add more functionality to -v and use volume management functions. #9803 and #9250 are more in that direction.

@erSitzt

This comment has been minimized.

Copy link

erSitzt commented Feb 23, 2015

👍

1 similar comment
@DanTheColoradan

This comment has been minimized.

Copy link

DanTheColoradan commented Feb 25, 2015

👍

@resouer

This comment has been minimized.

Copy link
Contributor

resouer commented Mar 26, 2015

Is this still moving forward? As we want to implement glusterfs driver which need support of this PR ...

@cybertk

This comment has been minimized.

Copy link

cybertk commented Apr 10, 2015

+1, it's useful to create a DFS based data volume container, all persistent data are stored in remote DFS

@SerCeMan

This comment has been minimized.

Copy link

SerCeMan commented Apr 28, 2015

👍

@noetix

This comment has been minimized.

Copy link

noetix commented May 26, 2015

+1

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented May 28, 2015

Hi all. Docker 1.7 will include a re-architected volume subsystem which now uses a driver-based approach, and the experimental channel includes support for out-of-process volume plugins.

Here is a demo of a docker plugin for flocker: https://www.youtube.com/watch?v=wOMRV1JI5Lo

@h0tbird

This comment has been minimized.

Copy link

h0tbird commented Jul 30, 2015

+1 nice demo. I can't wait to add --volume-driver=ceph to my containers.

@geovanisouza92

This comment has been minimized.

Copy link

geovanisouza92 commented Aug 13, 2015

👍

@jessfraz jessfraz removed the kind/proposal label Sep 8, 2015

@calavera

This comment has been minimized.

Copy link
Contributor

calavera commented Sep 11, 2015

Closing this as done. We already shipped volume plugins in 1.8 and there is a dedicated subcommand coming in 1.9 and there are already several out there. I believe there are even two different plugins for ceph.

We can build from that if we need additional features.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment