Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Best way to move docker directory #3127

Closed
Soulou opened this Issue Dec 8, 2013 · 59 comments

Comments

Projects
None yet
@Soulou
Copy link
Contributor

Soulou commented Dec 8, 2013

Good evening,

I'm looking for the best way to move docker directory (default /var/lib/docker) to another directory. (Technically other disk partition)

I'm using devicemapper, and I think a lot of things are mounted, so it's not as easy as a mv

I suppose the first step is to kill all the containers, then is it necessary to remove them ? Does remove unmount everything ? Is it possible to move everything including the containers ? (because deleting them is a bit heavy)

Thank you !

@unclejack

This comment has been minimized.

Copy link
Contributor

unclejack commented Dec 8, 2013

@Soulou The simplest way to do this is to stop and kill all containers, reboot the box, stop the docker daemon after the fresh boot and then move the /var/lib/docker folder to another partition. I'm recommending the reboot to make sure all aufs mounts are unmounted.

@Soulou

This comment has been minimized.

Copy link
Contributor Author

Soulou commented Dec 8, 2013

Thanks unclejack for your fast answer.

Is it working the same for dm ? After killing all containers, the mounpoints are still there, I can't move the directory. (I'd like to avoid rebooting if possible)

@unclejack

This comment has been minimized.

Copy link
Contributor

unclejack commented Dec 8, 2013

@Soulou You have to stop the docker daemon completely and try to move the folder. That should make it work, but I haven't tried that. You can also use the steps above to move a devicemapper docker folder to another partition.

Please report back if you have any troubles. We can close this issue after you report back to let us know if you had any troubles when stopping docker (trying to move it without a reboot).

@Soulou

This comment has been minimized.

Copy link
Contributor Author

Soulou commented Dec 8, 2013

Ok it's done, and it worked quite well:

docker ps -q | xargs docker kill
stop docker
cd /var/lib/docker/devicemapper/mnt
umount ./*
mv /var/lib/docker $dest
ln -s $dest /var/lib/docker
start docker

And it seems to be working !

@Soulou Soulou closed this Dec 8, 2013

@thomasleveil

This comment has been minimized.

Copy link
Contributor

thomasleveil commented Feb 20, 2014

ln -s $dest /var/lib/docker

This was the cause of issues with the docker rm command (error about being Unable to remove filesystem)
The solution to avoid having a symbolic link is to start docker with the -g parameter

On Ubuntu that option is to be set in the /etc/default/docker file

DOCKER_OPTS="-g /somewhere/else/docker/"
@tianon

This comment has been minimized.

Copy link
Member

tianon commented Feb 20, 2014

and if you want to get really fancy, you can use something like:

DOCKER_OPTS="-g $(readlink -f /var/lib/docker)"
@Mulkave

This comment has been minimized.

Copy link

Mulkave commented Aug 30, 2014

Just dropping a note for CentOS users - the docker config lives at /etc/sysconfig/docker and the equivalent of the above is:

other_args="-g /data/docker"
@thefallentree

This comment has been minimized.

Copy link

thefallentree commented Mar 6, 2015

I found that if there are already some previous containers in the docker, moving the directory (and adding -g as above) still won't make them run again, complaining about mount failures.

It seems there are absolute paths somewhere written in container state, could this be solved?

@Willyfrog

This comment has been minimized.

Copy link

Willyfrog commented Apr 6, 2015

I've run into the same problem, that is "solved" by symlinking as the @Soulou script suggests.
It would be great to be able to change that mount failure, but at least by symlinking I don't have to rebuild everything again.

@conradob

This comment has been minimized.

Copy link

conradob commented Aug 24, 2015

I believe that in this guide you'll have a good explanation.

You can change Docker's storage base directory (where container and images go) using the -g option when starting the Docker daemon.

Ubuntu/Debian: edit your /etc/default/docker file with the -g option: DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4 -g /mnt"

Fedora/Centos: edit /etc/sysconfig/docker, and add the -g option in the other_args variable: ex. other_args="-g /var/lib/testdir". If there's more than one option, make sure you enclose them in " ". After a restart, (service docker restart) Docker should use the new directory.

Using a symlink is another method to change image storage.

Caution - These steps depend on your current /var/lib/docker being an actual directory (not a symlink to another location).

  1. Stop docker: service docker stop. Verify no docker process is running ps faux
  2. Double check docker really isn't running. Take a look at the current docker directory: ls /var/lib/docker/
    2b) Make a backup - tar -zcC /var/lib docker > /mnt/pd0/var_lib_docker-backup-$(date +%s).tar.gz
  3. Move the /var/lib/docker directory to your new partition: mv /var/lib/docker /mnt/pd0/docker
  4. Make a symlink: ln -s /mnt/pd0/docker /var/lib/docker
  5. Take a peek at the directory structure to make sure it looks like it did before the mv: ls /var/lib/docker/ (note the trailing slash to resolve the symlink)
  6. Start docker back up service docker start
  7. restart your containers
@tiangolo

This comment has been minimized.

Copy link

tiangolo commented Sep 29, 2015

I was having some issues related to this, I was having permissions problems inside the containers after changing the directory. Namely, a mysql container couldn't run because it didn't have write permissions to the /tmp/ directory.

To test it, and to check if permissions were right or not, I ran a test container:

docker run -it ubuntu bash

And I checked the permissions for the /tmp/ directory:

ls -l /

Before moving the /var/lib/docker files, the permissions for /tmp/ (inside the container) were rwxrwxrwx, after moving the files and creating a test container like before, the permissions were rwxr--r--.


This is what worked for me in a VirtualBox test machine (this may help other people that arrive here).

I mounted an external disk following this http://dev-random.net/creating-mounting-partition-lager-2-tb-linux/ and this https://help.ubuntu.com/community/Fstab

The new directory ended up in: /media/disk2/var/lib/docker/

I assume you already have a directory with a similar location for your partition.

  • First, I stopped Docker
service docker stop

I wanted to make sure everything would work before moving or removing my Docker /var/lib/docker files, so I wanted to "copy" the files instead of "moving" them. And that was my main problem, the files were copied without the original permissions.

  • To copy the files preserving permissions I used rsync with the "archive" option
rsync -a /var/lib/docker/* /media/disk2/var/lib/docker/
  • Then I edited the /etc/default/docker file
nano /etc/default/docker
  • I edited and uncommented the line with
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
  • That line ended up like
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /media/disk2/var/lib/docker"
  • Then I started Docker again
service docker start

And I was able to re-start my old containers without loosing any data (I created a dummy mysql and wordpress, added changes and checked them after moving the files).

@ghostsquad

This comment has been minimized.

Copy link

ghostsquad commented Oct 15, 2015

I'm trying your method here using rsync and I've found that the destination is 4-6x larger than the source. Any ideas why that might be?

@tiangolo

This comment has been minimized.

Copy link

tiangolo commented Oct 16, 2015

Hmm, that's weird @ghostsquad, how are you measuring the sizes? What kind of file system you have in each disk?

@stefanleh

This comment has been minimized.

Copy link

stefanleh commented Oct 17, 2015

Using current CentOS7 (3.10.0-229.14.1.el7.x86_64) it was necessary to modify /usr/lib/systemd/system/docker.service to get the -d /docker/dir/ parameter handed over to the executable:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/docker daemon -g /docker -H fd://
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity

[Install]
WantedBy=multi-user.target
@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Oct 17, 2015

@stefanleh it's really discouraged to edit the docker.service file directly. To override default settings, use drop-in files, as explained here; https://docs.docker.com/articles/systemd/#custom-docker-daemon-options

@stefanleh

This comment has been minimized.

Copy link

stefanleh commented Oct 17, 2015

@thaJeztah thank you for the hint.

Should'nt the EnvironmentFile=-/etc/sysconfig/docker and ExecStart=...... $other_args lines be in the original /usr/lib/systemd/system/docker.service then by default?

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Oct 17, 2015

@stefanleh no, various distros came back from using environment variables/files for configuration, that's why it isn't in there. However you are free to use them if you want using the method described.

@stefanleh

This comment has been minimized.

Copy link

stefanleh commented Oct 17, 2015

Ok, but then the use of /etc/sysconfig/docker in /etc/init.d/docker should be dropped as well.
This was the biggest cause of confusion for me. I started debugging there first and wondered some minutes why my changes are'nt reflected. (I'm Dev and usually relying on Ops for such tasks in my job)

@skairali

This comment has been minimized.

Copy link

skairali commented Jan 16, 2016

+1

@skairali

This comment has been minimized.

Copy link

skairali commented Jan 16, 2016

tiangolo commented on 30 Sep 2015 -> Solved issues for me

@zedtux

This comment has been minimized.

Copy link

zedtux commented May 31, 2016

@tiangolo I tried the instructions you gave, but when I try to run my container I have the following:

DEBU[0057] sandbox set key processing took 121.496115ms for container 4459631a33df967d3a525688f6638529a59eb2f69b53847368f4737f48ef4853
ERRO[0057] containerd: start container                   error=oci runtime error: permission denied id=4459631a33df967d3a525688f6638529a59eb2f69b53847368f4737f48ef4853
...
ERRO[0058] Handler for POST /v1.23/containers/rtorrent-rutorrent/start returned error: Container command 'supervisord' could not be invoked.
$ docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

So it seems to not work.

@asokani

This comment has been minimized.

Copy link

asokani commented Jun 1, 2016

I would recommend considering moving to ZFS! See https://docs.docker.com/engine/userguide/storagedriver/zfs-driver/

@tiangolo

This comment has been minimized.

Copy link

tiangolo commented Jun 1, 2016

@zedtux it seems you're having permissions issues. Make sure you are using rsync -a to copy the files. That will ensure that all the permissions remain the same.

If you use rsync with different options or for example cp you need to (somehow) make sure you keep the original permissions.

@mondain

This comment has been minimized.

Copy link

mondain commented Jun 9, 2016

I've tried the other solutions listed here and they fail in one way or another, so I've modified the one from @tiangolo and it works for me on
Ubuntu 16.04.

First stop Docker

service docker stop

Copy everything from /var/lib/docker to my new destination, which is /mnt/extra in my case

cd /var/lib
sudo rsync -a docker /mnt/extra/

Remove the docker directory

sudo rm -rf docker

Create a static link

sudo ln -s docker /mnt/extra/docker

Edit the /etc/default/docker file

sudo vi /etc/default/docker

Modify this line

#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"

Changing it to this

DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /mnt/extra/docker"

Restart Docker

service docker start

To test perms etc use this command

docker run -it ubuntu bash

If it all works, you're good to go!

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jun 9, 2016

It's also possible to use a daemon.json configuration file instead of /etc/default/docker. The /etc/default/docker is only used for systems running upstart, and not for systems running systemd, so is more portable. Also it allows reloading some configuration settings without restarting the daemon;

https://docs.docker.com/engine/reference/commandline/daemon/#daemon-configuration-file

@zedtux

This comment has been minimized.

Copy link

zedtux commented Jun 10, 2016

@tiangolo I followed using copy/past your instructions.

@eromoe

This comment has been minimized.

Copy link

eromoe commented Aug 22, 2016

On centos 6.5 docker 1.7 (I'm not allowed to update system or docker)

rsync -a /var/lib/docker/* /mnt/docker/data/ command never finish.

  • /var/lib/docker/ only 3.7G
  • /mnt/docker/data/ became larger and larger to 72G, and still got larger

I have to interrtpt the rsync and clean the folder. What should I do?

@t-oster

This comment has been minimized.

Copy link

t-oster commented Aug 22, 2016

use rsync -a --sparse ... as suggested here https://groups.google.com/forum/#!topic/docker-user/3mUVu4dNJBU

@eromoe

This comment has been minimized.

Copy link

eromoe commented Aug 23, 2016

@t-oster --sparse didn't work

[root@server1 docker]# cd /var/lib/docker
[root@server1 docker]# du -csh *
2.7M    containers
3.6G    devicemapper
1.6M    graph
5.5M    init
28K     linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
92M     volumes
3.7G    total
[root@server1 docker]# rsync -a -sparse /var/lib/docker/* /mnt/docker/data/

Still got larger and larger:

[root@server1 data]# du -csh *
26G     devicemapper
436K    graph
4.0K    init
24K     linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
324K    volumes
26G     total
[root@server1 data]# du -csh *
33G     devicemapper
436K    graph
4.0K    init
24K     linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
324K    volumes
33G     total
@t-oster

This comment has been minimized.

Copy link

t-oster commented Aug 23, 2016

does the target filesystem support sparse files? Also you need to delete the target file first, so that it gets created as sparse file.

@eromoe

This comment has been minimized.

Copy link

eromoe commented Aug 23, 2016

@t-oster

I have checked my filesystem.

df -T
/dev/vda1      ext4   20510332  16767576   2694232  87% /
tmpfs          tmpfs  16439744        60  16439684   1% /dev/shm
/dev/vdb1      ext3  309636900 206882212  87026060  71% /mnt

I search for ext3 sparse file support and filesystem support sparse file, not an article list all the filesystem which support sparse. The only related seems to be http://serverfault.com/questions/558761/best-linux-filesystem-for-sparse-files ,but it says All files systems that support inodes (ext2/3/4, xfs, btfs, etc) support files, use inodes instead of sparse , does that mean ext3 support sparse file?

Or it is the problem copy from ext4 to ext3 ?


And what do you mean target file ?

/mnt/docker/data/ is clean, no file in this dir
/var/lib/docker/ is docker images folder, I can't delete it

@t-oster

This comment has been minimized.

Copy link

t-oster commented Aug 23, 2016

should work. but in your line you wrote -sparse instead of --sparse

@eromoe

This comment has been minimized.

Copy link

eromoe commented Aug 24, 2016

@t-oster
sorry for late reply, I have to switch to server2 which env just like server1.

You are right, rsync doesn't eat up disk any more with --sparse
[root@server2 docker]# rsync -a --sparse /var/lib/docker/* /mnt/docker/data/

But rsync still stumed,
/var/lib/docker is 2.3G, /mnt/docker/data/ just stop at 2.0G...

[root@server2 data]# cd /var/lib/docker
[root@server2 docker]# du -csh *
3.3M    containers
2.3G    devicemapper
1.5M    graph
5.5M    init
8.0K    linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
1004K   volumes
2.3G    total

[root@server2 data]# cd /mnt/docker/data/
[root@server2 data]# du -csh *
3.3M    containers
2.0G    devicemapper
508K    graph
4.0K    init
8.0K    linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
4.0K    volumes
2.0G    total
@t-oster

This comment has been minimized.

Copy link

t-oster commented Aug 24, 2016

it may still be complete, did you try starting docker after bind-mounting /mnt/docker over /var/lib/docker?

@eromoe

This comment has been minimized.

Copy link

eromoe commented Aug 25, 2016

No, I interrupted rsync and used other_args="-g /mnt/docker/data/".
But docker images show nothing.

I think the key problem is rsync never finish.

@eromoe

This comment has been minimized.

Copy link

eromoe commented Aug 26, 2016

My bad, I don't know can use -v to see rsync progress. rsync just too slow, and take serveral hours, some times my ssh connection timed out also disturb rsync.

@btorresgil

This comment has been minimized.

Copy link
Contributor

btorresgil commented Nov 23, 2016

@thaJeztah Your last suggestion worked best. I'm on Ubuntu 16.04.1, but it shouldn't matter because this is a cross-distro solution.

Just put this json into /etc/docker/daemon.json:

{
  "graph": "/path/to/docker"
}

Worked for me and I didn't have to mess with upstart or systemd.

@danielhklein

This comment has been minimized.

Copy link

danielhklein commented Jan 18, 2017

Any ideas on how to fix this issue on CentOS 7.3? Using Docker v1.13. I was able to start the docker service with no issues after installing with

curl -fsSL https://get.docker.com/ | sh

but after stopping the service and modifying the docker.service file to show

ExecStart=/usr/bin/dockerd daemon -g /mnt1/docker-data

then

systemctl daemon-reload

it hangs when I try systemctl start docker

I've also tried ExecStart=/usr/bin/dockerd --graph="/mnt/docker-data" --storage-driver=overlay

And now, even if I revert docker.service back to its previous state, it still hangs at launch...

edit: got it working! steps:

  1. ExecStart=/usr/bin/dockerd --graph="/mnt/docker-data" --storage-driver=overlay in /usr/lib/systemd/system/docker.service

  2. systemctl daemon-reload

  3. reboot

  4. systemctl start docker

@sourcekick

This comment has been minimized.

Copy link

sourcekick commented Feb 21, 2017

@zedtux

This comment has been minimized.

Copy link

zedtux commented Feb 26, 2017

@sourcekick I followed the steps from your article and it seems to work! 👍

@ensemblebd

This comment has been minimized.

Copy link

ensemblebd commented Jun 1, 2017

If you've already moved your folder using one of the numerous alternative topics on the internet, without preserving permissions, you likely will still receive:
"no space left on device, permission denied" error within the container (docker logs ).

Still haven't found a resolution... Even ran chmod and umask to give everyone full access all the time. So it can't be permissions right?
Am starting to wonder if it's the fact that the new data directory is on a separate mounted drive.

@andyneff

This comment has been minimized.

Copy link

andyneff commented Jun 26, 2017

Update: -g and --graph are deprecated in 17.05, and you should be using --data-root instead now.

Also, as @thaJeztah pointed out, customize systemd by using /etc/docker/daemon.json (which appears to still use graph?)

{
  "graph": "/some/other/dir"
}

Works on 17.05.0-ce

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jun 26, 2017

which appears to still use graph?

Both "graph" and "data-root" should work. Even though deprecated, the old ("graph") will still work (for backward compatibility), but is hidden to encourage people to use the (less confusingly named) "data-root" name (see #28696).

The docs at docs.docker.com usually reflect the latest "stable" release, which don't have this change yet

@tmarplatt

This comment has been minimized.

Copy link

tmarplatt commented Jan 7, 2018

Is a symlink from /var/lib/docker necessary (any longer) when you've already set the data-root/graph option? Doesn't docker use this option alone to look up contents in a different folder?

@swarupkm

This comment has been minimized.

Copy link

swarupkm commented Jun 8, 2018

I was trying the above steps, but now i cant see any container/images
docker ps , docker ps -a , docker images are all empty.

However i have made sure both var/lib/docker and /new/path are having the data just in case.

Can anyone explain why I cannot see the containers?

My docker version is 1.31 , Fedora 28

@mkorycinski

This comment has been minimized.

Copy link

mkorycinski commented Aug 29, 2018

Hi,

I've noticed that when I specified 'graph' in daemon.json file, as stated above, newly specified folder grew enormously, to about 100gb, which is more than 5 times the /var/lib/docker directory. Any clues why that might be? Of course it grew just after I did sudo service docker restart, not because of newly build containers.

@andyneff

This comment has been minimized.

Copy link

andyneff commented Aug 30, 2018

@mkorycinski These file systems use an over lapping snapshot features: like overlay, overlay2, aufs, etc... So the directories in your graph dir have "snapshots" that appear in multiple places. Some basic linux commands like du command may count this same file many many times, for images, containers, etc... I believe relying on docker system du will give you a more reliable answer. df also gives a reliable answer for how much space is available.

@hktalent

This comment has been minimized.

Copy link

hktalent commented Sep 2, 2018

@Soulou @thomasleveil @tianon @Mulkave @conradob @tiangolo @thaJeztah @mborho @jerbia @btorresgil

cp -r  /Users/`whoami`/Library/Containers/com.docker.docker/Data /Volumes/kali/Data
rm -rf /Users/`whoami`/Library/Containers/com.docker.docker/Data
ln -s /Volumes/kali/Data /Users/`whoami`/Library/Containers/com.docker.docker/Data

now can not start docker

@bigspawn

This comment has been minimized.

Copy link

bigspawn commented Sep 14, 2018

Steps work for me:

  1. Stop all containers docker stop $(docker ps -a -q)

  2. Stop docker sudo service docker stop

  3. Copy docker directory cp -r /var/lib/docker/* /home/docker/

  4. Unmount all old docker overlays umount -f /var/lib/docker/overlay/*/* and umount -f /var/lib/docker/containers/*/mounts/shm

  5. Change in/etc/docker/daemon.json graph to new path

    {
      "graph": "/home/docker/"
    }
  6. Start docker sudo service docker start

  7. Remove old /var/lib/docker/

@fsevenm

This comment has been minimized.

Copy link

fsevenm commented Jan 1, 2019

Steps work for me:

  1. Stop all containers docker stop $(docker ps -a -q)
  2. Stop docker sudo service docker stop
  3. Copy docker directory cp -r /var/lib/docker/* /home/docker/
  4. Unmount all old docker overlays umount -f /var/lib/docker/overlay/*/* and umount -f /var/lib/docker/containers/*/mounts/shm
  5. Change in/etc/docker/daemon.json graph to new path
    {
      "graph": "/home/docker/"
    }
  6. Start docker sudo service docker start
  7. Remove old /var/lib/docker/

I didn't find daemon.json inside /etc/docker/, there was only key.json. Should I add it myself?

UPDATE:
And I got the solution here https://forums.docker.com/t/daemon-json-file-not-found-in-etc-docker/61657

Just create that file ourselves!

@andyneff

This comment has been minimized.

Copy link

andyneff commented Jan 2, 2019

@fsevenm Yes, if you have no daemon.json file, add one yourself. By default, there isn't one, but some plugins/packages add other fields for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.