docker cp should support VOLUMEs #1992

darklajid · 2013-09-24T17:14:48Z

docker cp seems to be limited to the root/default volume, as confirmed on IRC:

$ cat Dockerfile
FROM ubuntu:precise
MAINTAINER Benjamin Podszun "dar@darklajid.de"

VOLUME /some/volume
RUN touch /some/volume/I_Want_To_Copy_This

CMD /bin/bash

$ docker build .
Uploading context 10240 bytes
Step 1 : FROM ubuntu:precise
Pulling repository ubuntu
---> 8dbd9e392a96layersfrom ubuntu, endpoint: https://cdn-registry-1.docker.io/v1/
Step 2 : MAINTAINER Benjamin Podszun "dar@darklajid.de"
---> Using cache
---> c950832bf997
Step 3 : VOLUME /some/volume
---> Running in f9b8fc1901f2
---> 56106cf843d5
Step 4 : RUN touch /some/volume/I_Want_To_Copy_This
---> Running in 31eca80e094d
---> 2cb2858fedee
Step 5 : CMD /bin/bash
---> Running in af26bafd07ba
---> 4e0f6538a826
Successfully built 4e0f6538a826

$ docker run -i -t 4e0f6538a826
root@d787a64f78f9:/#

$ docker cp d787a64f78f9:/some/volume/I_Want_To_Copy_This .
2013/09/24 17:12:11 Error: stat /var/lib/docker/containers/d787a64f78f94e88abcdda2c8224c3ecfface23d783c4308f08ef1a3367d2ae1/rootfs/some/volume/I_Want_To_Copy_This: no such file or directory

jpetazzo · 2013-09-24T18:34:52Z

IMHO, volumes should behave this way: when declaring a volume, if the volume is empty on the host, the content of the directory (in the container) will be copied to the volume (on the host) before binding the volume to the container.

This means that you could make any directory a volume without breaking the container behavior.

shykes · 2013-09-24T19:40:04Z

+1 we really need to implement this. It's a blocker for anyone with an initial db content in the image for example.

—
@solomonstre
@getdocker

On Tue, Sep 24, 2013 at 11:34 AM, Jérôme Petazzoni
notifications@github.com wrote:

IMHO, volumes should behave this way: when declaring a volume, if the volume is empty on the host, the content of the directory (in the container) will be copied to the volume (on the host) before binding the volume to the container.

This means that you could make any directory a volume without breaking the container behavior.

Reply to this email directly or view it on GitHub:
#1992 (comment)

crosbymichael · 2013-09-24T19:57:22Z

I'm working on this right now. should be done soon

jpetazzo · 2013-09-24T20:00:40Z

(Someone on IRC asked more details; so I'll try to explain it better.)

Suppose you have a MySQL image. In /var/lib/mysql, it has the initial content of the database (if you're using default paths).
Currently, if you do -v /var/lib/mysql, the directory will be mounted as a volume, and the volume will initially be empty.
As a result, MySQL won't start.

My proposal is the following: when doing -v /var/lib/mysql, initialize the content of the volume with the content of /var/lib/mysql in the image. That way, it will be a volume, but MySQL will start.

Now, what happens when you do something like -v /mnt/db:/var/lib/mysql? You're asking Docker to mount /mnt/db (in the host) to /var/lib/mysql (in the container). If you copy the content of /var/lilb/mysql (from the image) to the volume each time you start the container, you will reset the database each time... Which is probably not what you want.

Hence the "if empty" condition. If /mnt/db is empty, it will be initialized with the content of /var/lib/mysql. If it contains anything, it won't be touched, and be used as is.

solomonstre · 2013-09-24T20:11:51Z

For historical value I dug around for the first implementation of that logic in docker's great-granpa: dotcloud circa 2009! https://bitbucket.org/Foi3GraS/dotcloud-fork/src/c0db261cbbef198f0ded934a42dcabe1294a814f/volume/directory.py?at=default#cl-85

(looks like this fork by Foi3GraS is the only remaining trace of the original open-source dotcloud :)

—
@solomonstre
@getdocker

On Tue, Sep 24, 2013 at 13:01, Jérôme Petazzoni <notifications@github.com="mailto:notifications@github.com">> wrote:

(Someone on IRC asked more details; so I'll try to explain it better.)

Suppose you have a MySQL image. In /var/lib/mysql, it has the initial content of the database (if you're using default paths).

Currently, if you do -v /var/lib/mysql, the directory will be mounted as a volume, and the volume will initially be empty.

As a result, MySQL won't start.

My proposal is the following: when doing -v /var/lib/mysql, initialize the content of the volume with the content of /var/lib/mysql in the image. That way, it will be a volume, but MySQL will start.

Now, what happens when you do something like -v /mnt/db:/var/lib/mysql? You're asking Docker to mount /mnt/db (in the host) to /var/lib/mysql (in the container). If you copy the content of /var/lilb/mysql (from the image) to the volume each time you start the container, you will reset the database each time... Which is probably not what you want.

Hence the "if empty" condition. If /mnt/db is empty, it will be initialized with the content of /var/lib/mysql. If it contains anything, it won't be touched, and be used as is.

—
Reply to this email directly or view it on GitHub.

jpetazzo · 2013-09-24T20:18:47Z

Oh, it looks like I hadn't understood the initial question.

I thought the issue was "RUN cp some/file /to/some/volume doesn't work because the volume is initially empty", but in fact, it's "docker cp /from/some/volume doesn't work because since the volume is mounted in a private mnt namespace, Docker needs to manually resolve the actual path of the file".

My bad.

jpetazzo · 2013-09-24T20:18:56Z

Should I open a separate issue for volume initialization?

crosbymichael · 2013-09-24T20:57:42Z

I jumped into this issue thinking it would be a quick fix but it is not. The way volumes are implemented are a little challenging for copying the files. Say you have a volume /data with files in it and you want to cp the directory.

docker cp 45444:/data .

Because the container actually has the /data directory and the volume's layer directory has the files you will get a directory /layer copied to your system with the data instead of /data with the data. This is because we tar the directory and contents and the root directory in the volume's path is /layer not /data.

@jpetazzo @shykes Any ideas on how to solve this?

This was my initial thought

func (container *Container) Copy(resource string) (Archive, error) {
        if err := container.EnsureMounted(); err != nil {
                return nil, err
        }
        rootPath := container.RootfsPath()
        // Check if a container has any volumes and if it does, see if the resource
        // is inside a volume
        if container.Volumes != nil && len(container.Volumes) > 0 {
                for volume, volumePath := range container.Volumes {
                        if filepath.HasPrefix(resource, volume) {
                                rootPath = volumePath
                                // Because files in a volume are stored in the layer directory without
                                // the folder name we have to stripe the volume name from the resource
                                resource = strings.TrimPrefix(resource, volume)
                                break
                        }
                }
        }
        var filter []string
        basePath := path.Join(rootPath, resource)
        stat, err := os.Stat(basePath)
        if err != nil {
                return nil, err
        }
        if !stat.IsDir() {
                d, f := path.Split(basePath)
                basePath = d
                filter = []string{f}
        } else {
                filter = []string{path.Base(basePath)}
                basePath = path.Dir(basePath)
        }
        return TarFilter(basePath, Uncompressed, filter)
}

jpetazzo · 2013-09-24T21:25:18Z

I think it's the way to go; with a couple of caveats:

we might have to resolve recursively (in case there are nested volumes);
we probably want to forbid "/../" in the resource path.

solomonstre · 2013-09-25T00:42:45Z

Wouldn't it make more sense to copy the contents of the volume once, at
mount-time (if the volume is empty) and then after that simply apply the
copy orthogonally from the existence of volumes?

On Tue, Sep 24, 2013 at 2:25 PM, Jérôme Petazzoni
notifications@github.comwrote:

I think it's the way to go; with a couple of caveats:

we might have to resolve recursively (in case there are nested volumes);

we probably want to forbid "/../" in the resource path.

—

Reply to this email directly or view it on GitHubhttps://github.com//issues/1992#issuecomment-25043880
.

jpetazzo · 2013-09-25T00:55:31Z

Well, if I understand correctly, you can't do the copy orthogonally,
because the volume is mounted only in the container's namespace. It's not
mounted in Docker's namespace.

solomonstre · 2013-09-25T01:12:50Z

Ah, right. What if we also made sure they were mounted in docker's
namespace? Shouldn't be too hard. Then we go back to having copy being
orthogonal :)

On Tue, Sep 24, 2013 at 5:56 PM, Jérôme Petazzoni
notifications@github.comwrote:

Well, if I understand correctly, you can't do the copy orthogonally,
because the volume is mounted only in the container's namespace. It's not
mounted in Docker's namespace.

—

Reply to this email directly or view it on GitHubhttps://github.com//issues/1992#issuecomment-25055062
.

crosbymichael · 2013-11-21T18:30:05Z

Assigned myself, this will be implemented after the graph driver changes

crosbymichael · 2013-11-30T01:40:55Z

This first part of this is to have docker handled mounting volumes and not rely on lxc.

cpuguy83 · 2013-12-31T14:00:24Z

What's the status of this? Is this something I can get my hands into?

crosbymichael · 2013-12-31T15:28:46Z

This can be easily accomplished now that we are mounting volumes outside of LXC.

runvnc · 2014-01-06T10:15:29Z

This is the #1 thing I am hoping to see added to docker. Is someone already working on this? If not it might be a reason for me to learn Go.

cpuguy83 · 2014-01-06T18:10:50Z

I would like to, not sure where to look at the moment.

GrahamDumpleton · 2014-01-21T04:34:31Z

I am not sure if the issue I am encountering is a variant of this issue or not and whether the same fix will address it or not, so adding my case for triggering it as an additional test case.

Docker file:

FROM ubuntu:precise
ADD . /data
VOLUME /data

Build it:

$ docker build -t bug-report .
Uploading context 10.24 kB
Uploading context
Step 1 : FROM ubuntu:precise
 ---> 8dbd9e392a96
Step 2 : ADD . /data
 ---> 9e4c948bcf2f
Step 3 : VOLUME /data
 ---> Running in 65e79f2d5569
 ---> ee8d8ebb2b2a
Successfully built ee8d8ebb2b2a

Run it and create a new file in the volume, or even update an existing one. Leave the shell open initially.

$ docker run -t -i -name bug-report-container bug-report bash
root@7bc571994ccf:/# cd /data
root@7bc571994ccf:/data# ls -las
total 16
4 drwxr-xr-x  3  502 dialout 4096 Jan 21 04:08 .
4 drwxr-xr-x 33 root root    4096 Jan 21 04:31 ..
4 -rw-r--r--  1  502 dialout   45 Jan 21 04:07 Dockerfile
0 -rw-r--r--  1  502 dialout    0 Jan 21 04:08 ROOT-README
4 drwxr-xr-x  2  502 dialout 4096 Jan 21 04:07 subdir
root@7bc571994ccf:/data# touch bug.txt
root@7bc571994ccf:/data#

From separate window copy out updated file and it will work.

$ docker cp bug-report-container:/data/bug.txt /tmp/

Now exit container so it is stopped.

Run the copy again against the stopped container.

$ docker cp bug-report-container:/data/bug.txt /tmp/
2014/01/21 15:34:06 Error: stat /var/lib/docker/aufs/mnt/7bc571994ccf9baafc295913cfd2e69c58afe6cec8ad16520263e5d1c9fa509f/data/bug.txt: no such file or directory

crosbymichael · 2014-01-21T06:50:23Z

@GrahamDumpleton Your issue is resolved handled.

Closes moby#1992 Docker-DCO-1.0-Signed-off-by: Michael Crosby <crosby.michael@gmail.com> (github: crosbymichael)

cyphar · 2014-05-14T12:44:45Z

This should be closed, as #3472 fixes this.

scj7t4 · 2014-05-14T19:18:05Z

afaik that was not ever merged into docker and this is still an issue.

On Wed, May 14, 2014 at 7:45 AM, Aleksa Sarai notifications@github.comwrote:

This should be closed, as #3472 https://github.com/dotcloud/docker/pull/3472fixes this.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1992#issuecomment-43075201
.

brianclements · 2014-05-21T17:43:21Z

confirmed still as an issue for me. I have a volume /log that is shared by 3 containers simultaneously, not just one, if thats any help.

○ sudo docker cp mopidy:/m_config .

○ sudo docker cp mopidy:/log/m_config .
2014/05/21 10:33:46 Error: Could not find the file /log/m_config in container mopidy

○ sudo docker version
Client version: 0.11.1
Client API version: 1.11
Go version (client): go1.2.1
Git commit (client): fb99f99
Server version: 0.11.1
Server API version: 1.11
Git commit (server): fb99f99
Go version (server): go1.2.1
Last stable version: 0.11.1

jbeda · 2014-06-17T02:56:31Z

Confirmed that this still exists in 1.0.0. I got hit by this.

brianclements · 2014-06-17T04:13:16Z

Confirmed here again too also using my same exact example as before.

± sudo docker version
Client version: 1.0.0
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 63fe64c
Server version: 1.0.0
Server API version: 1.12
Go version (server): go1.2.1
Git commit (server): 63fe64c

goldmann · 2014-07-15T13:07:02Z

I was hit by this today.

$ docker version
Client version: 1.0.0
Client API version: 1.12
Go version (client): go1.2.2
Git commit (client): 63fe64c/1.0.0
Server version: 1.0.0
Server API version: 1.12
Go version (server): go1.2.2
Git commit (server): 63fe64c/1.0.0

$ rpm -q docker-io
docker-io-1.0.0-6.fc20.x86_64

gesellix · 2014-09-15T15:14:10Z

same issue here, what about #3472?

$ docker version
Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.3.1
Git commit (client): fa7b24f
OS/Arch (client): linux/amd64
Server version: 1.2.0
Server API version: 1.14
Go version (server): go1.3.1
Git commit (server): fa7b24f

Fixes moby#1992 Right now when you `docker cp` a path which is in a volume, the cp itself works, however you end up getting files that are in the container's fs rather than the files in the volume (which is not in the container's fs). This makes it so when you `docker cp` a path that is in a volume it follows the volume to the real path on the host. archive.go has been modified so that when you do `docker cp mydata:/foo .`, and /foo is the volume, the outputed folder is called "foo" instead of the volume ID (because we are telling it to tar up `/var/lib/docker/vfs/dir/<some id>` and not "foo", but the user would be expecting "foo", not the ID Signed-off-by: Brian Goff <cpuguy83@gmail.com>

ghost assigned crosbymichael Nov 21, 2013

crosbymichael mentioned this issue Jan 6, 2014

Allow docker cp from within volumes #3472

Closed

crosbymichael added a commit to crosbymichael/docker that referenced this issue Jan 22, 2014

Allow docker cp from within volumes

e5dad8e

Closes moby#1992 Docker-DCO-1.0-Signed-off-by: Michael Crosby <crosby.michael@gmail.com> (github: crosbymichael)

crosbymichael removed their assignment Mar 18, 2014

crosbymichael mentioned this issue Apr 8, 2014

docker cp for volumes-from gets wrong version #5098

Closed

cpuguy83 mentioned this issue Jun 18, 2014

Collected issues with Volumes #6496

Closed

12 tasks

cpuguy83 mentioned this issue Oct 11, 2014

Make container.Copy support volumes #8509

Merged

LK4D4 closed this as completed in #8509 Oct 21, 2014

jlhawn mentioned this issue May 20, 2015

docker cp to and from containers #13171

Merged

thaJeztah mentioned this issue Nov 9, 2017

docker cp does not work with mounted volumes on Windows containers #34576

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker cp should support VOLUMEs #1992

docker cp should support VOLUMEs #1992

darklajid commented Sep 24, 2013

jpetazzo commented Sep 24, 2013

shykes commented Sep 24, 2013

This means that you could make any directory a volume without breaking the container behavior.

crosbymichael commented Sep 24, 2013

jpetazzo commented Sep 24, 2013

solomonstre commented Sep 24, 2013

jpetazzo commented Sep 24, 2013

jpetazzo commented Sep 24, 2013

crosbymichael commented Sep 24, 2013

jpetazzo commented Sep 24, 2013

solomonstre commented Sep 25, 2013

jpetazzo commented Sep 25, 2013

solomonstre commented Sep 25, 2013

crosbymichael commented Nov 21, 2013

crosbymichael commented Nov 30, 2013

cpuguy83 commented Dec 31, 2013

crosbymichael commented Dec 31, 2013

runvnc commented Jan 6, 2014

cpuguy83 commented Jan 6, 2014

GrahamDumpleton commented Jan 21, 2014

crosbymichael commented Jan 21, 2014

cyphar commented May 14, 2014

scj7t4 commented May 14, 2014

brianclements commented May 21, 2014

jbeda commented Jun 17, 2014

brianclements commented Jun 17, 2014

goldmann commented Jul 15, 2014

gesellix commented Sep 15, 2014

docker cp should support VOLUMEs #1992

docker cp should support VOLUMEs #1992

Comments

darklajid commented Sep 24, 2013

jpetazzo commented Sep 24, 2013

shykes commented Sep 24, 2013

This means that you could make any directory a volume without breaking the container behavior.

crosbymichael commented Sep 24, 2013

jpetazzo commented Sep 24, 2013

solomonstre commented Sep 24, 2013

jpetazzo commented Sep 24, 2013

jpetazzo commented Sep 24, 2013

crosbymichael commented Sep 24, 2013

jpetazzo commented Sep 24, 2013

solomonstre commented Sep 25, 2013

jpetazzo commented Sep 25, 2013

solomonstre commented Sep 25, 2013

crosbymichael commented Nov 21, 2013

crosbymichael commented Nov 30, 2013

cpuguy83 commented Dec 31, 2013

crosbymichael commented Dec 31, 2013

runvnc commented Jan 6, 2014

cpuguy83 commented Jan 6, 2014

GrahamDumpleton commented Jan 21, 2014

crosbymichael commented Jan 21, 2014

cyphar commented May 14, 2014

scj7t4 commented May 14, 2014

brianclements commented May 21, 2014

jbeda commented Jun 17, 2014

brianclements commented Jun 17, 2014

goldmann commented Jul 15, 2014

gesellix commented Sep 15, 2014