New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker do not copy data to volume when use -v command to mount a volume #17470

Closed
xiaoshengaimm opened this Issue Oct 29, 2015 · 24 comments

Comments

Projects
None yet
@xiaoshengaimm
Copy link

xiaoshengaimm commented Oct 29, 2015

hi ~~

I found docker do not copy the mount point data to volume when use -v command to mount a volume.

however, it copy the data to the volume when use VOLUME instrument in the Dockerfile.

here is my Docker file:

# Pull base image.
FROM ubuntu

EXPOSE 9000
RUN mkdir -p /usr/apps
ADD ./package.tgz /usr/apps/

# Define default command.
CMD ["bash"]

when i start the image use -v command, ls show the directory is empty;

docker run --name sandbox -v /home/apps:/usr/apps sandbox ls /usr/apps

how ever when i use VOLUME instrument,it works; ls show the directory has files.

why? is the two way some different? i need your help.

thanks.

@GordonTheTurtle

This comment has been minimized.

Copy link

GordonTheTurtle commented Oct 29, 2015

Hi!

Please read this important information about creating issues.

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

This is an automated, informational response.

Thank you.

For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues


BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:
uname -a:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

@xiaoshengaimm

This comment has been minimized.

Copy link
Author

xiaoshengaimm commented Oct 29, 2015

docker version

Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

docker info

Containers: 1
Images: 56
Storage Driver: devicemapper
 Pool Name: docker-8:7-73946-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.327 GB
 Data Space Total: 107.4 GB
 Data Space Available: 4.54 GB
 Metadata Space Used: 3.256 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.90-1.el6.elrepo.x86_64
Operating System: <unknown>
CPUs: 4
Total Memory: 7.815 GiB
Name: cnode333-vm03
ID: IOQS:35DD:EUO2:65JH:ENPY:JLJO:MHKG:TLEF:YE4I:T4ON:F7SE:Z4V7
Username: xiaoshengaimm
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

uname -a

Linux cnode333-vm03 3.10.90-1.el6.elrepo.x86_64 #1 SMP Thu Oct 1 15:15:53 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

other infomation:
my host is a virtual machine that i creat it by myself. system is

CentOS release 6.6 (Final)
Kernel \r on an \m

result i received

docker run --name sandbox -v /home/apps:/usr/apps sandbox ls /usr/apps

when i run the container i can not find the extra file in package.tgz

result i expected
when i run the container i find the extra file in package.tgz.

Dockerfile is

# Pull base image.
FROM ubuntu

EXPOSE 9000
RUN mkdir -p /usr/apps
ADD ./package.tgz /usr/apps/

# Define default command.
CMD ["bash"]

thanks

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Oct 29, 2015

-v /host/path:/container/path does not copy data.
-v /container/path will copy data.
The 2nd form is equivalent of VOLUME /container/path in a Dockerfile.

@cpuguy83 cpuguy83 closed this Oct 29, 2015

@xiaoshengaimm

This comment has been minimized.

Copy link
Author

xiaoshengaimm commented Nov 2, 2015

ok~~ thanks for your answer.

is there any way to copy container's file to the VOLUME path when i start the container and using -v /host/path:/container/path? and Why -v /host/path:/container/path dose not copy data?

and the documents do not specify the difference.

thanks.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Nov 2, 2015

The reason is that when using a "bind mounted" directory from the host, you're telling docker that you want to take a file or directory from your host and use it in your container. Docker should not modify those files/directories, unless you explicitly do so. For example, you don't want -v /home/user/:/var/lib/mysql to result in your home-directory being replaced with a MySQL database.

@mariusGundersen

This comment has been minimized.

Copy link
Contributor

mariusGundersen commented Dec 14, 2015

Unfortunately (with docker 1.9) this is also true when you specify the name of a volume, not a path.

-v /host/path:/container/path does not copy data
-v /container/path will copy data and creates new volume with a random name
-v name:/container/path does not copy data but creates a new volume with the name name

It would be useful if it could copy the contents from the container if the volume doesn't exist or if the volume is empty

@ghost

This comment has been minimized.

Copy link

ghost commented Dec 29, 2015

+1 to @mariusGundersen last comment. Files refusing to copy and mount is preventing me from using Docker at all.

@vitalyisaev2

This comment has been minimized.

Copy link

vitalyisaev2 commented Dec 30, 2015

Hello!
I haven't checked it yet on my own, but people on Stackoverflow are reporting the similair issues with docker 1.9.1. So please clarify, did you break volume mounting in the recent releases? This is a fundamental feature.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Dec 30, 2015

@vitalyisaev2 there's no breaking change in docker 1.9.1, the only discussion is still on point "3" mentioned by @mariusGundersen, basically;

  1. bind mounting a host-directory in a container (docker run -v /some/host/dir/:/container/path/) uses the files that are present on the host. If the host-directory doesn't exist, a new, empty directory is created on the host and mounted in the container (this will change in future, and an error is shown in stead)
  2. using a "nameless" volume (docker run -v /container/path) will create a new volume, and copy the contents of /container/path into that volume
  3. using a "named" volume (docker run -v somename:/container/path) will create a new volume, named "somename", or use the existing "somename" volume, and use the files that are present inside that volume. If the volume is newly created, it will be empty. This behavior is still under discussion in #18670

I'm not sure what the cause is for the issue on stackoverflow, because the behavior of "1." has not changed, however, if boot2docker/VirtualBox is used, only files and directories inside /Users directory (OS X) on the host or C:\Users directory (Windows) are shared with the virtual machine by default. Directories other than that are not shared with the VM, so do not "exist" in the VM, therefore resulting in a new (empty) directory being created inside the VM, and mounted in the container the first time that command is run. Given that this tricked many people, we deprecated that behavior and will be producing an error instead ("Error: the mounted path does not exist" or similar).

@vitalyisaev2

This comment has been minimized.

Copy link

vitalyisaev2 commented Dec 30, 2015

@thaJeztah thank you, now it's clear

@digitaldavenyc

This comment has been minimized.

Copy link

digitaldavenyc commented Dec 30, 2015

@vitalyisaev2 I was the one who reported the issue on stack overflow. Nice to see the docker team is watching. @thaJeztah I was binding my directory using method #1 with my files in my Users directory. I downgraded docker and the issue was fixed.

I'd be happy to supply another information that isn't in the stackoverflow article that could be helpful.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Dec 30, 2015

@digitaldavenyc perhaps you can open a separate issue to prevent mixing up topics here? Happy to give it a look though (or see if I can get other people to look at it)

@jubins

This comment has been minimized.

Copy link

jubins commented Nov 24, 2016

@vitalyisaev2 , @thaJeztah So is this correct command to copy data?
$ docker run -it -v $Home/tf_files/ gcr.io/tensorflow/tensorflow:latest-devel

I want to virtually link whatever is in 'tf_files' folder to 'gcr.io/tensorflow/tensorflow:latest-devel' and I ran this command: $ docker run -it -v $Home/tf_files/ gcr.io/tensorflow/tensorflow:latest-devel
then did ls, it does not copy anything?

image

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Nov 24, 2016

@jubins it depends what you are trying to achieve, there are quite some things that play a role here though;

  • $Home is resolved in your local shell (note that $Home is case-sensitive on Linux, so you likely want $HOME (all uppercase). So if your $HOME is /home/jubin, the shell replaces what you wrote with -v /home/jubin/tf_files/
  • You specify a single path after -v, which means that docker creates an "anonymous" volume, copies all the data that was present inside the container at /home/jubin/tf_files/ to the volume, and mounts the volume at /home/jubin/tf_files/ inside the container. The tensorflow image does not have this directory (there's no "jubin" user inside the container), so it will create an empty directory, and mount an empty volume there.
  • If you want to "bind-mount" your local directory inside a container, you need to specify both the path on your host, and the path inside the container where you want that directory to be mounted, for example, -v /home/jubin/tf_files/:/path/in/container
  • Be aware that (I see you're running on Windows), MINGW is known to "mangle" Windows paths, http://www.mingw.org/wiki/Posix_path_conversion, so you may have to manually specify your home directory, and use a double slash at the start; for example -v //c/Users/jubin/tf_files/:/path/in/container/
  • Without knowing what your setup is (Docker for Windows, Docker Toolbox, Windows client and a remote host?), bind-mounting files from your host into a container may not be possible (in case of a remote host), or require additional configuration https://docs.docker.com/docker-for-windows/#/shared-drives, https://docs.docker.com/docker-for-windows/troubleshoot/#/volume-mounting-requires-shared-drives-for-linux-containers-and-for-any-project-directories-outside-of-cusers)
@activenode

This comment has been minimized.

Copy link

activenode commented Jan 15, 2017

Hello there. I want to add my 2 cents on this after having read all of your comments carefully.

I do not agree that it would be dangerous to copy files from the container to the host.
That is why I want do discuss it again.

Let's take the assumption of @thaJeztah :

For example, you don't want -v /home/user/:/var/lib/mysql to result in your home-directory being replaced with a MySQL database.

I do not agree with that.
You are mentioning a specific case (MySQL database). And yes probably (!) you did not intend to replace your home directory with a database. But also probably (!) when I enter 'sudo rm -r /var/www' I might have not intended that but instead I wanted to do 'sudo rm -r /var/www/project' but pressed enter to soon.
What I am trying to clarify is: What you are mentioning is a UX issue. This UX issue is really important but it could be solved on other ways not being in conflict with automatically copying files from the container to a given host path.

Example:
docker run -v /path/to/host/dir:/path/to/guest/dir should copy all files from container to host. IF and only IF the host dir is not empty I see 2 possibilites:

  1. Command is rejected with error message "Host dir must be empty"
  2. Command is not rejected but info message is given: "Files from container will not be copied as host volume is not empty"
@pandiloko

This comment has been minimized.

Copy link

pandiloko commented Mar 23, 2017

@activenode +1
It would be a nice feature. I already made some hacky workarounds synchronizing the files on container start but that would be defintively cleaner if on docker run a synchronization between container and host took place. If host is empty, copy from container. If container is empty, copy from host.

@tony-olech

This comment has been minimized.

Copy link

tony-olech commented Sep 15, 2017

This issue is important. I want the docker build command to initialize a volume that will hold persistent data and the docker run command to use it, i.e. copy to host first time.
I then want to be able to re-build and either use the previous volume data or to re-initialize it on the host.

docker run -v /path/to/host/dir:/path/to/guest/dir should copy all files from container to host if and only if host dir is empty solves my requirements and seems sensible and SAFE

@activenode

This comment has been minimized.

Copy link

activenode commented Oct 3, 2017

Hey @cpuguy83 . This discussion has taken some more turns. Do you still think closed status is the correct status for this issue?

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Oct 3, 2017

@activenode

  • We can't change the default.
  • I'm pretty 👎 on adding new syntax to the -v format.
  • We had similar discussions 3-4 years ago (I even had a PR to enable this) and it was decided that Docker should not be messing with the host path, and that since the user has control over it, they user can copy stuff into it.

I am not particularly averse to having an option on --mount type=bind to enable copying image content to an empty host path, but I'm not the only one to convince.
I would also recommend opening a separate issue

@r3lik

This comment has been minimized.

Copy link

r3lik commented Dec 10, 2017

This become even more problematic when you're running Docker inside Vagrant. I want to be able to have a shared/synced folder with my host and the VM guest, and in turn the Docker container and the VM host.
However, with the current implementation, I have to hack my way around all of this. Unfortunate.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Dec 10, 2017

@r3lik what has become problematic? This is a closed issue with multiple discussions.

@r3lik

This comment has been minimized.

Copy link

r3lik commented Dec 10, 2017

-v /host/path:/container/path does not copy data between the Docker container and the host (in my case a VM).
/host/path is my synced folder between my VM and physical host

Therefore my data isn't persistent.
I understand that my use case is a bit more specific, but surely there are people running Docker on top of Vagrant.

Anyway, my workaround is to do the following within the Vagrantfile:
config.vm.synced_folder "storage/", "/var/lib/docker/volumes/", create: true

This allows me to sync the data between my_host->vagrant->docker container

I've read a few discussions about this, and what I'm seeing is that it was decided not to implement this functionality. Is that still the case?

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Dec 11, 2017

@r3lik Your data is "persistent" because it's in the image already. Are you saying you want to copy data from the image onto the vagrant host?

it was decided not to implement this functionality

That is (sort of) correct. The path is a host path that the user has full control over. This copy can be facilitated without adding extra functionality to docker.

One thing you could do if you really really want Docker to handle this is to setup a volume as a bind mount... docker volume create --opt type=none --opt device=/host/path --opt o=bind my_bind_volume

@r3lik

This comment has been minimized.

Copy link

r3lik commented Dec 11, 2017

Thank you for clarifying :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment