Best way to move docker directory #3127

Closed
Soulou opened this Issue Dec 8, 2013 · 53 comments

Comments

Projects
None yet
@Soulou
Contributor

Soulou commented Dec 8, 2013

Good evening,

I'm looking for the best way to move docker directory (default /var/lib/docker) to another directory. (Technically other disk partition)

I'm using devicemapper, and I think a lot of things are mounted, so it's not as easy as a mv

I suppose the first step is to kill all the containers, then is it necessary to remove them ? Does remove unmount everything ? Is it possible to move everything including the containers ? (because deleting them is a bit heavy)

Thank you !

@unclejack

This comment has been minimized.

Show comment
Hide comment
@unclejack

unclejack Dec 8, 2013

Contributor

@Soulou The simplest way to do this is to stop and kill all containers, reboot the box, stop the docker daemon after the fresh boot and then move the /var/lib/docker folder to another partition. I'm recommending the reboot to make sure all aufs mounts are unmounted.

Contributor

unclejack commented Dec 8, 2013

@Soulou The simplest way to do this is to stop and kill all containers, reboot the box, stop the docker daemon after the fresh boot and then move the /var/lib/docker folder to another partition. I'm recommending the reboot to make sure all aufs mounts are unmounted.

@Soulou

This comment has been minimized.

Show comment
Hide comment
@Soulou

Soulou Dec 8, 2013

Contributor

Thanks unclejack for your fast answer.

Is it working the same for dm ? After killing all containers, the mounpoints are still there, I can't move the directory. (I'd like to avoid rebooting if possible)

Contributor

Soulou commented Dec 8, 2013

Thanks unclejack for your fast answer.

Is it working the same for dm ? After killing all containers, the mounpoints are still there, I can't move the directory. (I'd like to avoid rebooting if possible)

@unclejack

This comment has been minimized.

Show comment
Hide comment
@unclejack

unclejack Dec 8, 2013

Contributor

@Soulou You have to stop the docker daemon completely and try to move the folder. That should make it work, but I haven't tried that. You can also use the steps above to move a devicemapper docker folder to another partition.

Please report back if you have any troubles. We can close this issue after you report back to let us know if you had any troubles when stopping docker (trying to move it without a reboot).

Contributor

unclejack commented Dec 8, 2013

@Soulou You have to stop the docker daemon completely and try to move the folder. That should make it work, but I haven't tried that. You can also use the steps above to move a devicemapper docker folder to another partition.

Please report back if you have any troubles. We can close this issue after you report back to let us know if you had any troubles when stopping docker (trying to move it without a reboot).

@Soulou

This comment has been minimized.

Show comment
Hide comment
@Soulou

Soulou Dec 8, 2013

Contributor

Ok it's done, and it worked quite well:

docker ps -q | xargs docker kill
stop docker
cd /var/lib/docker/devicemapper/mnt
umount ./*
mv /var/lib/docker $dest
ln -s $dest /var/lib/docker
start docker

And it seems to be working !

Contributor

Soulou commented Dec 8, 2013

Ok it's done, and it worked quite well:

docker ps -q | xargs docker kill
stop docker
cd /var/lib/docker/devicemapper/mnt
umount ./*
mv /var/lib/docker $dest
ln -s $dest /var/lib/docker
start docker

And it seems to be working !

@Soulou Soulou closed this Dec 8, 2013

@thomasleveil

This comment has been minimized.

Show comment
Hide comment
@thomasleveil

thomasleveil Feb 20, 2014

Contributor

ln -s $dest /var/lib/docker

This was the cause of issues with the docker rm command (error about being Unable to remove filesystem)
The solution to avoid having a symbolic link is to start docker with the -g parameter

On Ubuntu that option is to be set in the /etc/default/docker file

DOCKER_OPTS="-g /somewhere/else/docker/"
Contributor

thomasleveil commented Feb 20, 2014

ln -s $dest /var/lib/docker

This was the cause of issues with the docker rm command (error about being Unable to remove filesystem)
The solution to avoid having a symbolic link is to start docker with the -g parameter

On Ubuntu that option is to be set in the /etc/default/docker file

DOCKER_OPTS="-g /somewhere/else/docker/"
@tianon

This comment has been minimized.

Show comment
Hide comment
@tianon

tianon Feb 20, 2014

Member

and if you want to get really fancy, you can use something like:

DOCKER_OPTS="-g $(readlink -f /var/lib/docker)"
Member

tianon commented Feb 20, 2014

and if you want to get really fancy, you can use something like:

DOCKER_OPTS="-g $(readlink -f /var/lib/docker)"
@Mulkave

This comment has been minimized.

Show comment
Hide comment
@Mulkave

Mulkave Aug 30, 2014

Just dropping a note for CentOS users - the docker config lives at /etc/sysconfig/docker and the equivalent of the above is:

other_args="-g /data/docker"

Mulkave commented Aug 30, 2014

Just dropping a note for CentOS users - the docker config lives at /etc/sysconfig/docker and the equivalent of the above is:

other_args="-g /data/docker"
@thefallentree

This comment has been minimized.

Show comment
Hide comment
@thefallentree

thefallentree Mar 6, 2015

I found that if there are already some previous containers in the docker, moving the directory (and adding -g as above) still won't make them run again, complaining about mount failures.

It seems there are absolute paths somewhere written in container state, could this be solved?

I found that if there are already some previous containers in the docker, moving the directory (and adding -g as above) still won't make them run again, complaining about mount failures.

It seems there are absolute paths somewhere written in container state, could this be solved?

@Willyfrog

This comment has been minimized.

Show comment
Hide comment
@Willyfrog

Willyfrog Apr 6, 2015

I've run into the same problem, that is "solved" by symlinking as the @Soulou script suggests.
It would be great to be able to change that mount failure, but at least by symlinking I don't have to rebuild everything again.

I've run into the same problem, that is "solved" by symlinking as the @Soulou script suggests.
It would be great to be able to change that mount failure, but at least by symlinking I don't have to rebuild everything again.

@conradob

This comment has been minimized.

Show comment
Hide comment
@conradob

conradob Aug 24, 2015

I believe that in this guide you'll have a good explanation.

You can change Docker's storage base directory (where container and images go) using the -g option when starting the Docker daemon.

Ubuntu/Debian: edit your /etc/default/docker file with the -g option: DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4 -g /mnt"

Fedora/Centos: edit /etc/sysconfig/docker, and add the -g option in the other_args variable: ex. other_args="-g /var/lib/testdir". If there's more than one option, make sure you enclose them in " ". After a restart, (service docker restart) Docker should use the new directory.

Using a symlink is another method to change image storage.

Caution - These steps depend on your current /var/lib/docker being an actual directory (not a symlink to another location).

  1. Stop docker: service docker stop. Verify no docker process is running ps faux
  2. Double check docker really isn't running. Take a look at the current docker directory: ls /var/lib/docker/
    2b) Make a backup - tar -zcC /var/lib docker > /mnt/pd0/var_lib_docker-backup-$(date +%s).tar.gz
  3. Move the /var/lib/docker directory to your new partition: mv /var/lib/docker /mnt/pd0/docker
  4. Make a symlink: ln -s /mnt/pd0/docker /var/lib/docker
  5. Take a peek at the directory structure to make sure it looks like it did before the mv: ls /var/lib/docker/ (note the trailing slash to resolve the symlink)
  6. Start docker back up service docker start
  7. restart your containers

I believe that in this guide you'll have a good explanation.

You can change Docker's storage base directory (where container and images go) using the -g option when starting the Docker daemon.

Ubuntu/Debian: edit your /etc/default/docker file with the -g option: DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4 -g /mnt"

Fedora/Centos: edit /etc/sysconfig/docker, and add the -g option in the other_args variable: ex. other_args="-g /var/lib/testdir". If there's more than one option, make sure you enclose them in " ". After a restart, (service docker restart) Docker should use the new directory.

Using a symlink is another method to change image storage.

Caution - These steps depend on your current /var/lib/docker being an actual directory (not a symlink to another location).

  1. Stop docker: service docker stop. Verify no docker process is running ps faux
  2. Double check docker really isn't running. Take a look at the current docker directory: ls /var/lib/docker/
    2b) Make a backup - tar -zcC /var/lib docker > /mnt/pd0/var_lib_docker-backup-$(date +%s).tar.gz
  3. Move the /var/lib/docker directory to your new partition: mv /var/lib/docker /mnt/pd0/docker
  4. Make a symlink: ln -s /mnt/pd0/docker /var/lib/docker
  5. Take a peek at the directory structure to make sure it looks like it did before the mv: ls /var/lib/docker/ (note the trailing slash to resolve the symlink)
  6. Start docker back up service docker start
  7. restart your containers
@tiangolo

This comment has been minimized.

Show comment
Hide comment
@tiangolo

tiangolo Sep 29, 2015

I was having some issues related to this, I was having permissions problems inside the containers after changing the directory. Namely, a mysql container couldn't run because it didn't have write permissions to the /tmp/ directory.

To test it, and to check if permissions were right or not, I ran a test container:

docker run -it ubuntu bash

And I checked the permissions for the /tmp/ directory:

ls -l /

Before moving the /var/lib/docker files, the permissions for /tmp/ (inside the container) were rwxrwxrwx, after moving the files and creating a test container like before, the permissions were rwxr--r--.


This is what worked for me in a VirtualBox test machine (this may help other people that arrive here).

I mounted an external disk following this http://dev-random.net/creating-mounting-partition-lager-2-tb-linux/ and this https://help.ubuntu.com/community/Fstab

The new directory ended up in: /media/disk2/var/lib/docker/

I assume you already have a directory with a similar location for your partition.

  • First, I stopped Docker
service docker stop

I wanted to make sure everything would work before moving or removing my Docker /var/lib/docker files, so I wanted to "copy" the files instead of "moving" them. And that was my main problem, the files were copied without the original permissions.

  • To copy the files preserving permissions I used rsync with the "archive" option
rsync -a /var/lib/docker/* /media/disk2/var/lib/docker/
  • Then I edited the /etc/default/docker file
nano /etc/default/docker
  • I edited and uncommented the line with
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
  • That line ended up like
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /media/disk2/var/lib/docker"
  • Then I started Docker again
service docker start

And I was able to re-start my old containers without loosing any data (I created a dummy mysql and wordpress, added changes and checked them after moving the files).

I was having some issues related to this, I was having permissions problems inside the containers after changing the directory. Namely, a mysql container couldn't run because it didn't have write permissions to the /tmp/ directory.

To test it, and to check if permissions were right or not, I ran a test container:

docker run -it ubuntu bash

And I checked the permissions for the /tmp/ directory:

ls -l /

Before moving the /var/lib/docker files, the permissions for /tmp/ (inside the container) were rwxrwxrwx, after moving the files and creating a test container like before, the permissions were rwxr--r--.


This is what worked for me in a VirtualBox test machine (this may help other people that arrive here).

I mounted an external disk following this http://dev-random.net/creating-mounting-partition-lager-2-tb-linux/ and this https://help.ubuntu.com/community/Fstab

The new directory ended up in: /media/disk2/var/lib/docker/

I assume you already have a directory with a similar location for your partition.

  • First, I stopped Docker
service docker stop

I wanted to make sure everything would work before moving or removing my Docker /var/lib/docker files, so I wanted to "copy" the files instead of "moving" them. And that was my main problem, the files were copied without the original permissions.

  • To copy the files preserving permissions I used rsync with the "archive" option
rsync -a /var/lib/docker/* /media/disk2/var/lib/docker/
  • Then I edited the /etc/default/docker file
nano /etc/default/docker
  • I edited and uncommented the line with
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
  • That line ended up like
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /media/disk2/var/lib/docker"
  • Then I started Docker again
service docker start

And I was able to re-start my old containers without loosing any data (I created a dummy mysql and wordpress, added changes and checked them after moving the files).

@ghostsquad

This comment has been minimized.

Show comment
Hide comment
@ghostsquad

ghostsquad Oct 15, 2015

I'm trying your method here using rsync and I've found that the destination is 4-6x larger than the source. Any ideas why that might be?

I'm trying your method here using rsync and I've found that the destination is 4-6x larger than the source. Any ideas why that might be?

@tiangolo

This comment has been minimized.

Show comment
Hide comment
@tiangolo

tiangolo Oct 16, 2015

Hmm, that's weird @ghostsquad, how are you measuring the sizes? What kind of file system you have in each disk?

Hmm, that's weird @ghostsquad, how are you measuring the sizes? What kind of file system you have in each disk?

@stefanleh

This comment has been minimized.

Show comment
Hide comment
@stefanleh

stefanleh Oct 17, 2015

Using current CentOS7 (3.10.0-229.14.1.el7.x86_64) it was necessary to modify /usr/lib/systemd/system/docker.service to get the -d /docker/dir/ parameter handed over to the executable:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/docker daemon -g /docker -H fd://
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity

[Install]
WantedBy=multi-user.target

Using current CentOS7 (3.10.0-229.14.1.el7.x86_64) it was necessary to modify /usr/lib/systemd/system/docker.service to get the -d /docker/dir/ parameter handed over to the executable:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/docker daemon -g /docker -H fd://
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity

[Install]
WantedBy=multi-user.target
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 17, 2015

Member

@stefanleh it's really discouraged to edit the docker.service file directly. To override default settings, use drop-in files, as explained here; https://docs.docker.com/articles/systemd/#custom-docker-daemon-options

Member

thaJeztah commented Oct 17, 2015

@stefanleh it's really discouraged to edit the docker.service file directly. To override default settings, use drop-in files, as explained here; https://docs.docker.com/articles/systemd/#custom-docker-daemon-options

@stefanleh

This comment has been minimized.

Show comment
Hide comment
@stefanleh

stefanleh Oct 17, 2015

@thaJeztah thank you for the hint.

Should'nt the EnvironmentFile=-/etc/sysconfig/docker and ExecStart=...... $other_args lines be in the original /usr/lib/systemd/system/docker.service then by default?

@thaJeztah thank you for the hint.

Should'nt the EnvironmentFile=-/etc/sysconfig/docker and ExecStart=...... $other_args lines be in the original /usr/lib/systemd/system/docker.service then by default?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 17, 2015

Member

@stefanleh no, various distros came back from using environment variables/files for configuration, that's why it isn't in there. However you are free to use them if you want using the method described.

Member

thaJeztah commented Oct 17, 2015

@stefanleh no, various distros came back from using environment variables/files for configuration, that's why it isn't in there. However you are free to use them if you want using the method described.

@stefanleh

This comment has been minimized.

Show comment
Hide comment
@stefanleh

stefanleh Oct 17, 2015

Ok, but then the use of /etc/sysconfig/docker in /etc/init.d/docker should be dropped as well.
This was the biggest cause of confusion for me. I started debugging there first and wondered some minutes why my changes are'nt reflected. (I'm Dev and usually relying on Ops for such tasks in my job)

Ok, but then the use of /etc/sysconfig/docker in /etc/init.d/docker should be dropped as well.
This was the biggest cause of confusion for me. I started debugging there first and wondered some minutes why my changes are'nt reflected. (I'm Dev and usually relying on Ops for such tasks in my job)

@skairali

This comment has been minimized.

Show comment
Hide comment

+1

@skairali

This comment has been minimized.

Show comment
Hide comment
@skairali

skairali Jan 16, 2016

tiangolo commented on 30 Sep 2015 -> Solved issues for me

tiangolo commented on 30 Sep 2015 -> Solved issues for me

@zedtux

This comment has been minimized.

Show comment
Hide comment
@zedtux

zedtux May 31, 2016

@tiangolo I tried the instructions you gave, but when I try to run my container I have the following:

DEBU[0057] sandbox set key processing took 121.496115ms for container 4459631a33df967d3a525688f6638529a59eb2f69b53847368f4737f48ef4853
ERRO[0057] containerd: start container                   error=oci runtime error: permission denied id=4459631a33df967d3a525688f6638529a59eb2f69b53847368f4737f48ef4853
...
ERRO[0058] Handler for POST /v1.23/containers/rtorrent-rutorrent/start returned error: Container command 'supervisord' could not be invoked.
$ docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

So it seems to not work.

zedtux commented May 31, 2016

@tiangolo I tried the instructions you gave, but when I try to run my container I have the following:

DEBU[0057] sandbox set key processing took 121.496115ms for container 4459631a33df967d3a525688f6638529a59eb2f69b53847368f4737f48ef4853
ERRO[0057] containerd: start container                   error=oci runtime error: permission denied id=4459631a33df967d3a525688f6638529a59eb2f69b53847368f4737f48ef4853
...
ERRO[0058] Handler for POST /v1.23/containers/rtorrent-rutorrent/start returned error: Container command 'supervisord' could not be invoked.
$ docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

So it seems to not work.

@asokani

This comment has been minimized.

Show comment
Hide comment

asokani commented Jun 1, 2016

I would recommend considering moving to ZFS! See https://docs.docker.com/engine/userguide/storagedriver/zfs-driver/

@tiangolo

This comment has been minimized.

Show comment
Hide comment
@tiangolo

tiangolo Jun 1, 2016

@zedtux it seems you're having permissions issues. Make sure you are using rsync -a to copy the files. That will ensure that all the permissions remain the same.

If you use rsync with different options or for example cp you need to (somehow) make sure you keep the original permissions.

tiangolo commented Jun 1, 2016

@zedtux it seems you're having permissions issues. Make sure you are using rsync -a to copy the files. That will ensure that all the permissions remain the same.

If you use rsync with different options or for example cp you need to (somehow) make sure you keep the original permissions.

@mondain

This comment has been minimized.

Show comment
Hide comment
@mondain

mondain Jun 9, 2016

I've tried the other solutions listed here and they fail in one way or another, so I've modified the one from @tiangolo and it works for me on
Ubuntu 16.04.

First stop Docker

service docker stop

Copy everything from /var/lib/docker to my new destination, which is /mnt/extra in my case

cd /var/lib
sudo rsync -a docker /mnt/extra/

Remove the docker directory

sudo rm -rf docker

Create a static link

sudo ln -s docker /mnt/extra/docker

Edit the /etc/default/docker file

sudo vi /etc/default/docker

Modify this line

#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"

Changing it to this

DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /mnt/extra/docker"

Restart Docker

service docker start

To test perms etc use this command

docker run -it ubuntu bash

If it all works, you're good to go!

mondain commented Jun 9, 2016

I've tried the other solutions listed here and they fail in one way or another, so I've modified the one from @tiangolo and it works for me on
Ubuntu 16.04.

First stop Docker

service docker stop

Copy everything from /var/lib/docker to my new destination, which is /mnt/extra in my case

cd /var/lib
sudo rsync -a docker /mnt/extra/

Remove the docker directory

sudo rm -rf docker

Create a static link

sudo ln -s docker /mnt/extra/docker

Edit the /etc/default/docker file

sudo vi /etc/default/docker

Modify this line

#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"

Changing it to this

DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /mnt/extra/docker"

Restart Docker

service docker start

To test perms etc use this command

docker run -it ubuntu bash

If it all works, you're good to go!

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jun 9, 2016

Member

It's also possible to use a daemon.json configuration file instead of /etc/default/docker. The /etc/default/docker is only used for systems running upstart, and not for systems running systemd, so is more portable. Also it allows reloading some configuration settings without restarting the daemon;

https://docs.docker.com/engine/reference/commandline/daemon/#daemon-configuration-file

Member

thaJeztah commented Jun 9, 2016

It's also possible to use a daemon.json configuration file instead of /etc/default/docker. The /etc/default/docker is only used for systems running upstart, and not for systems running systemd, so is more portable. Also it allows reloading some configuration settings without restarting the daemon;

https://docs.docker.com/engine/reference/commandline/daemon/#daemon-configuration-file

@zedtux

This comment has been minimized.

Show comment
Hide comment
@zedtux

zedtux Jun 10, 2016

@tiangolo I followed using copy/past your instructions.

zedtux commented Jun 10, 2016

@tiangolo I followed using copy/past your instructions.

@zedtux

This comment has been minimized.

Show comment
Hide comment
@zedtux

zedtux Jun 10, 2016

I will try the one from @mondain and let you know.

zedtux commented Jun 10, 2016

I will try the one from @mondain and let you know.

@mborho

This comment has been minimized.

Show comment
Hide comment
@mborho

mborho Jun 17, 2016

On Ubuntu 16.04 (using systemd) you have to change the systemd unit file /lib/systemd/system/docker.service and add "-g /data/docker/" to the ExecStart directive. Should look like this then:

ExecStart=/usr/bin/docker daemon -H fd:// -g /path/to/docker/

Don't forget to do systemctl daemon-reload afterwards...

mborho commented Jun 17, 2016

On Ubuntu 16.04 (using systemd) you have to change the systemd unit file /lib/systemd/system/docker.service and add "-g /data/docker/" to the ExecStart directive. Should look like this then:

ExecStart=/usr/bin/docker daemon -H fd:// -g /path/to/docker/

Don't forget to do systemctl daemon-reload afterwards...

@tugdualenligne

This comment has been minimized.

Show comment
Hide comment
@tugdualenligne

tugdualenligne Jun 25, 2016

For some reason, the advices in these posts do not work for me: I use Ubuntu 16.04 and tried

  • symlink
  • then deleted the symlink and tried the systemd unit file
  • then removed what I changed in systemd and tried through a direct CIFS fstab map

Nothing works

I get in the 2nd case (systemd -g option) a chown error in relation with the tmpdir;
and in the last case (fstab) the following error at docker daemon start:
level=fatal msg="Error starting daemon: chmod /var/lib/docker: permission denied"

Any help appreciated!

For some reason, the advices in these posts do not work for me: I use Ubuntu 16.04 and tried

  • symlink
  • then deleted the symlink and tried the systemd unit file
  • then removed what I changed in systemd and tried through a direct CIFS fstab map

Nothing works

I get in the 2nd case (systemd -g option) a chown error in relation with the tmpdir;
and in the last case (fstab) the following error at docker daemon start:
level=fatal msg="Error starting daemon: chmod /var/lib/docker: permission denied"

Any help appreciated!

@jerbia

This comment has been minimized.

Show comment
Hide comment
@jerbia

jerbia Jul 5, 2016

On ubuntu 16.04 I've used the steps described by @mondain, which worked for me.

There is a small typo in the "ln" command, should be:

cd /var/lib
sudo ln -s /mnt/extra/docker docker

jerbia commented Jul 5, 2016

On ubuntu 16.04 I've used the steps described by @mondain, which worked for me.

There is a small typo in the "ln" command, should be:

cd /var/lib
sudo ln -s /mnt/extra/docker docker

@zedtux

This comment has been minimized.

Show comment
Hide comment
@zedtux

zedtux Jul 10, 2016

@tiangolo and @mondain I tried what has been described by @mondain but it's not working still. :(

I have also tried to delete the docker folder and restart the daemon, it created the docker folder from scratch, then I ran the image (so Docker has downloaded again the image) and same error as reported in my previous comment.

I also have tried to chmod 777 -R my docker folder, same error.

zedtux commented Jul 10, 2016

@tiangolo and @mondain I tried what has been described by @mondain but it's not working still. :(

I have also tried to delete the docker folder and restart the daemon, it created the docker folder from scratch, then I ran the image (so Docker has downloaded again the image) and same error as reported in my previous comment.

I also have tried to chmod 777 -R my docker folder, same error.

@mvousden mvousden referenced this issue in computationalmodelling/virtualmicromagnetics Jul 22, 2016

Closed

CI: Change where docker stores its metadata #51

@ashleysommer

This comment has been minimized.

Show comment
Hide comment
@ashleysommer

ashleysommer Jul 25, 2016

I am having exactly the same issue as @zedtux and @tugdualenligne
I followed the instructions from @mondain exactly, and I still get this:

time="2016-07-25T13:20:55+10:00" level=error msg="containerd: start container" error="oci runtime error: permission denied" id=316deb1e9d05f8044d2759f58e83bf0f06e695f9a4b1ae011fec80f8ac22e331

In my case, I am moving from /var/lib/docker on an XFS partition (with the AUFS docker fs driver) to /mnt/data/docker on a BTRFS mount (with the BTRFS docker fs driver), I believe that is the cause of the problem.

EDIT: I fixed it.
For future googlers encountering this problem, it turns out the BTRFS linux fs driver puts noexec in the mounting options if you specify user or users in your mount options string. And it seems that containerd cannot run a container on a noexec location.

I fixed it by adding exec directive to the mount string after the user directive.

ashleysommer commented Jul 25, 2016

I am having exactly the same issue as @zedtux and @tugdualenligne
I followed the instructions from @mondain exactly, and I still get this:

time="2016-07-25T13:20:55+10:00" level=error msg="containerd: start container" error="oci runtime error: permission denied" id=316deb1e9d05f8044d2759f58e83bf0f06e695f9a4b1ae011fec80f8ac22e331

In my case, I am moving from /var/lib/docker on an XFS partition (with the AUFS docker fs driver) to /mnt/data/docker on a BTRFS mount (with the BTRFS docker fs driver), I believe that is the cause of the problem.

EDIT: I fixed it.
For future googlers encountering this problem, it turns out the BTRFS linux fs driver puts noexec in the mounting options if you specify user or users in your mount options string. And it seems that containerd cannot run a container on a noexec location.

I fixed it by adding exec directive to the mount string after the user directive.

@zedtux

This comment has been minimized.

Show comment
Hide comment
@zedtux

zedtux Jul 25, 2016

Awesome, thank you @ashleysommer, I guess I have the same issue!
I have plex running on the machine, and I got an error due to the noexec which I have removed.
I will try again now and I'll update the issue.

zedtux commented Jul 25, 2016

Awesome, thank you @ashleysommer, I guess I have the same issue!
I have plex running on the machine, and I got an error due to the noexec which I have removed.
I will try again now and I'll update the issue.

@danlamanna

This comment has been minimized.

Show comment
Hide comment
@danlamanna

danlamanna Jul 27, 2016

I had issues with the solutions suggested here on Fedora 24.

Substituting what people had said about setting DOCKER_OPTIONS with just OPTIONS worked for me.

So I modified /etc/sysconfig/docker to contain OPTIONS="-g /mnt".

I had issues with the solutions suggested here on Fedora 24.

Substituting what people had said about setting DOCKER_OPTIONS with just OPTIONS worked for me.

So I modified /etc/sysconfig/docker to contain OPTIONS="-g /mnt".

@eromoe

This comment has been minimized.

Show comment
Hide comment
@eromoe

eromoe Aug 22, 2016

On centos 6.5 docker 1.7 (I'm not allowed to update system or docker)

rsync -a /var/lib/docker/* /mnt/docker/data/ command never finish.

  • /var/lib/docker/ only 3.7G
  • /mnt/docker/data/ became larger and larger to 72G, and still got larger

I have to interrtpt the rsync and clean the folder. What should I do?

eromoe commented Aug 22, 2016

On centos 6.5 docker 1.7 (I'm not allowed to update system or docker)

rsync -a /var/lib/docker/* /mnt/docker/data/ command never finish.

  • /var/lib/docker/ only 3.7G
  • /mnt/docker/data/ became larger and larger to 72G, and still got larger

I have to interrtpt the rsync and clean the folder. What should I do?

@t-oster

This comment has been minimized.

Show comment
Hide comment

t-oster commented Aug 22, 2016

use rsync -a --sparse ... as suggested here https://groups.google.com/forum/#!topic/docker-user/3mUVu4dNJBU

@eromoe

This comment has been minimized.

Show comment
Hide comment
@eromoe

eromoe Aug 23, 2016

@t-oster --sparse didn't work

[root@server1 docker]# cd /var/lib/docker
[root@server1 docker]# du -csh *
2.7M    containers
3.6G    devicemapper
1.6M    graph
5.5M    init
28K     linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
92M     volumes
3.7G    total
[root@server1 docker]# rsync -a -sparse /var/lib/docker/* /mnt/docker/data/

Still got larger and larger:

[root@server1 data]# du -csh *
26G     devicemapper
436K    graph
4.0K    init
24K     linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
324K    volumes
26G     total
[root@server1 data]# du -csh *
33G     devicemapper
436K    graph
4.0K    init
24K     linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
324K    volumes
33G     total

eromoe commented Aug 23, 2016

@t-oster --sparse didn't work

[root@server1 docker]# cd /var/lib/docker
[root@server1 docker]# du -csh *
2.7M    containers
3.6G    devicemapper
1.6M    graph
5.5M    init
28K     linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
92M     volumes
3.7G    total
[root@server1 docker]# rsync -a -sparse /var/lib/docker/* /mnt/docker/data/

Still got larger and larger:

[root@server1 data]# du -csh *
26G     devicemapper
436K    graph
4.0K    init
24K     linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
324K    volumes
26G     total
[root@server1 data]# du -csh *
33G     devicemapper
436K    graph
4.0K    init
24K     linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
324K    volumes
33G     total
@t-oster

This comment has been minimized.

Show comment
Hide comment
@t-oster

t-oster Aug 23, 2016

does the target filesystem support sparse files? Also you need to delete the target file first, so that it gets created as sparse file.

t-oster commented Aug 23, 2016

does the target filesystem support sparse files? Also you need to delete the target file first, so that it gets created as sparse file.

@eromoe

This comment has been minimized.

Show comment
Hide comment
@eromoe

eromoe Aug 23, 2016

@t-oster

I have checked my filesystem.

df -T
/dev/vda1      ext4   20510332  16767576   2694232  87% /
tmpfs          tmpfs  16439744        60  16439684   1% /dev/shm
/dev/vdb1      ext3  309636900 206882212  87026060  71% /mnt

I search for ext3 sparse file support and filesystem support sparse file, not an article list all the filesystem which support sparse. The only related seems to be http://serverfault.com/questions/558761/best-linux-filesystem-for-sparse-files ,but it says All files systems that support inodes (ext2/3/4, xfs, btfs, etc) support files, use inodes instead of sparse , does that mean ext3 support sparse file?

Or it is the problem copy from ext4 to ext3 ?


And what do you mean target file ?

/mnt/docker/data/ is clean, no file in this dir
/var/lib/docker/ is docker images folder, I can't delete it

eromoe commented Aug 23, 2016

@t-oster

I have checked my filesystem.

df -T
/dev/vda1      ext4   20510332  16767576   2694232  87% /
tmpfs          tmpfs  16439744        60  16439684   1% /dev/shm
/dev/vdb1      ext3  309636900 206882212  87026060  71% /mnt

I search for ext3 sparse file support and filesystem support sparse file, not an article list all the filesystem which support sparse. The only related seems to be http://serverfault.com/questions/558761/best-linux-filesystem-for-sparse-files ,but it says All files systems that support inodes (ext2/3/4, xfs, btfs, etc) support files, use inodes instead of sparse , does that mean ext3 support sparse file?

Or it is the problem copy from ext4 to ext3 ?


And what do you mean target file ?

/mnt/docker/data/ is clean, no file in this dir
/var/lib/docker/ is docker images folder, I can't delete it

@t-oster

This comment has been minimized.

Show comment
Hide comment
@t-oster

t-oster Aug 23, 2016

should work. but in your line you wrote -sparse instead of --sparse

t-oster commented Aug 23, 2016

should work. but in your line you wrote -sparse instead of --sparse

@eromoe

This comment has been minimized.

Show comment
Hide comment
@eromoe

eromoe Aug 24, 2016

@t-oster
sorry for late reply, I have to switch to server2 which env just like server1.

You are right, rsync doesn't eat up disk any more with --sparse
[root@server2 docker]# rsync -a --sparse /var/lib/docker/* /mnt/docker/data/

But rsync still stumed,
/var/lib/docker is 2.3G, /mnt/docker/data/ just stop at 2.0G...

[root@server2 data]# cd /var/lib/docker
[root@server2 docker]# du -csh *
3.3M    containers
2.3G    devicemapper
1.5M    graph
5.5M    init
8.0K    linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
1004K   volumes
2.3G    total

[root@server2 data]# cd /mnt/docker/data/
[root@server2 data]# du -csh *
3.3M    containers
2.0G    devicemapper
508K    graph
4.0K    init
8.0K    linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
4.0K    volumes
2.0G    total

eromoe commented Aug 24, 2016

@t-oster
sorry for late reply, I have to switch to server2 which env just like server1.

You are right, rsync doesn't eat up disk any more with --sparse
[root@server2 docker]# rsync -a --sparse /var/lib/docker/* /mnt/docker/data/

But rsync still stumed,
/var/lib/docker is 2.3G, /mnt/docker/data/ just stop at 2.0G...

[root@server2 data]# cd /var/lib/docker
[root@server2 docker]# du -csh *
3.3M    containers
2.3G    devicemapper
1.5M    graph
5.5M    init
8.0K    linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
1004K   volumes
2.3G    total

[root@server2 data]# cd /mnt/docker/data/
[root@server2 data]# du -csh *
3.3M    containers
2.0G    devicemapper
508K    graph
4.0K    init
8.0K    linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
4.0K    volumes
2.0G    total
@t-oster

This comment has been minimized.

Show comment
Hide comment
@t-oster

t-oster Aug 24, 2016

it may still be complete, did you try starting docker after bind-mounting /mnt/docker over /var/lib/docker?

t-oster commented Aug 24, 2016

it may still be complete, did you try starting docker after bind-mounting /mnt/docker over /var/lib/docker?

@eromoe

This comment has been minimized.

Show comment
Hide comment
@eromoe

eromoe Aug 25, 2016

No, I interrupted rsync and used other_args="-g /mnt/docker/data/".
But docker images show nothing.

I think the key problem is rsync never finish.

eromoe commented Aug 25, 2016

No, I interrupted rsync and used other_args="-g /mnt/docker/data/".
But docker images show nothing.

I think the key problem is rsync never finish.

@eromoe

This comment has been minimized.

Show comment
Hide comment
@eromoe

eromoe Aug 26, 2016

My bad, I don't know can use -v to see rsync progress. rsync just too slow, and take serveral hours, some times my ssh connection timed out also disturb rsync.

eromoe commented Aug 26, 2016

My bad, I don't know can use -v to see rsync progress. rsync just too slow, and take serveral hours, some times my ssh connection timed out also disturb rsync.

@btorresgil

This comment has been minimized.

Show comment
Hide comment
@btorresgil

btorresgil Nov 23, 2016

Contributor

@thaJeztah Your last suggestion worked best. I'm on Ubuntu 16.04.1, but it shouldn't matter because this is a cross-distro solution.

Just put this json into /etc/docker/daemon.json:

{
  "graph": "/path/to/docker"
}

Worked for me and I didn't have to mess with upstart or systemd.

Contributor

btorresgil commented Nov 23, 2016

@thaJeztah Your last suggestion worked best. I'm on Ubuntu 16.04.1, but it shouldn't matter because this is a cross-distro solution.

Just put this json into /etc/docker/daemon.json:

{
  "graph": "/path/to/docker"
}

Worked for me and I didn't have to mess with upstart or systemd.

@danielhklein

This comment has been minimized.

Show comment
Hide comment
@danielhklein

danielhklein Jan 18, 2017

Any ideas on how to fix this issue on CentOS 7.3? Using Docker v1.13. I was able to start the docker service with no issues after installing with

curl -fsSL https://get.docker.com/ | sh

but after stopping the service and modifying the docker.service file to show

ExecStart=/usr/bin/dockerd daemon -g /mnt1/docker-data

then

systemctl daemon-reload

it hangs when I try systemctl start docker

I've also tried ExecStart=/usr/bin/dockerd --graph="/mnt/docker-data" --storage-driver=overlay

And now, even if I revert docker.service back to its previous state, it still hangs at launch...

edit: got it working! steps:

  1. ExecStart=/usr/bin/dockerd --graph="/mnt/docker-data" --storage-driver=overlay in /usr/lib/systemd/system/docker.service

  2. systemctl daemon-reload

  3. reboot

  4. systemctl start docker

danielhklein commented Jan 18, 2017

Any ideas on how to fix this issue on CentOS 7.3? Using Docker v1.13. I was able to start the docker service with no issues after installing with

curl -fsSL https://get.docker.com/ | sh

but after stopping the service and modifying the docker.service file to show

ExecStart=/usr/bin/dockerd daemon -g /mnt1/docker-data

then

systemctl daemon-reload

it hangs when I try systemctl start docker

I've also tried ExecStart=/usr/bin/dockerd --graph="/mnt/docker-data" --storage-driver=overlay

And now, even if I revert docker.service back to its previous state, it still hangs at launch...

edit: got it working! steps:

  1. ExecStart=/usr/bin/dockerd --graph="/mnt/docker-data" --storage-driver=overlay in /usr/lib/systemd/system/docker.service

  2. systemctl daemon-reload

  3. reboot

  4. systemctl start docker

@MaximeD MaximeD referenced this issue in MaximeD/docker-react-native Feb 5, 2017

Open

Image size is very very vey large #1

@sourcekick

This comment has been minimized.

Show comment
Hide comment
@zedtux

This comment has been minimized.

Show comment
Hide comment
@zedtux

zedtux Feb 26, 2017

@sourcekick I followed the steps from your article and it seems to work! 👍

zedtux commented Feb 26, 2017

@sourcekick I followed the steps from your article and it seems to work! 👍

@ensemblebd

This comment has been minimized.

Show comment
Hide comment
@ensemblebd

ensemblebd Jun 1, 2017

If you've already moved your folder using one of the numerous alternative topics on the internet, without preserving permissions, you likely will still receive:
"no space left on device, permission denied" error within the container (docker logs ).

Still haven't found a resolution... Even ran chmod and umask to give everyone full access all the time. So it can't be permissions right?
Am starting to wonder if it's the fact that the new data directory is on a separate mounted drive.

If you've already moved your folder using one of the numerous alternative topics on the internet, without preserving permissions, you likely will still receive:
"no space left on device, permission denied" error within the container (docker logs ).

Still haven't found a resolution... Even ran chmod and umask to give everyone full access all the time. So it can't be permissions right?
Am starting to wonder if it's the fact that the new data directory is on a separate mounted drive.

@andyneff

This comment has been minimized.

Show comment
Hide comment
@andyneff

andyneff Jun 26, 2017

Update: -g and --graph are deprecated in 17.05, and you should be using --data-root instead now.

Also, as @thaJeztah pointed out, customize systemd by using /etc/docker/daemon.json (which appears to still use graph?)

{
  "graph": "/some/other/dir"
}

Works on 17.05.0-ce

Update: -g and --graph are deprecated in 17.05, and you should be using --data-root instead now.

Also, as @thaJeztah pointed out, customize systemd by using /etc/docker/daemon.json (which appears to still use graph?)

{
  "graph": "/some/other/dir"
}

Works on 17.05.0-ce

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jun 26, 2017

Member

which appears to still use graph?

Both "graph" and "data-root" should work. Even though deprecated, the old ("graph") will still work (for backward compatibility), but is hidden to encourage people to use the (less confusingly named) "data-root" name (see #28696).

The docs at docs.docker.com usually reflect the latest "stable" release, which don't have this change yet

Member

thaJeztah commented Jun 26, 2017

which appears to still use graph?

Both "graph" and "data-root" should work. Even though deprecated, the old ("graph") will still work (for backward compatibility), but is hidden to encourage people to use the (less confusingly named) "data-root" name (see #28696).

The docs at docs.docker.com usually reflect the latest "stable" release, which don't have this change yet

@7yl4r 7yl4r referenced this issue in marinebon/sdg14-shiny Nov 17, 2017

Closed

mbon server out of disk space #6

@tmarplatt

This comment has been minimized.

Show comment
Hide comment
@tmarplatt

tmarplatt Jan 7, 2018

Is a symlink from /var/lib/docker necessary (any longer) when you've already set the data-root/graph option? Doesn't docker use this option alone to look up contents in a different folder?

Is a symlink from /var/lib/docker necessary (any longer) when you've already set the data-root/graph option? Doesn't docker use this option alone to look up contents in a different folder?

@swarupkm

This comment has been minimized.

Show comment
Hide comment
@swarupkm

swarupkm Jun 8, 2018

I was trying the above steps, but now i cant see any container/images
docker ps , docker ps -a , docker images are all empty.

However i have made sure both var/lib/docker and /new/path are having the data just in case.

Can anyone explain why I cannot see the containers?

My docker version is 1.31 , Fedora 28

swarupkm commented Jun 8, 2018

I was trying the above steps, but now i cant see any container/images
docker ps , docker ps -a , docker images are all empty.

However i have made sure both var/lib/docker and /new/path are having the data just in case.

Can anyone explain why I cannot see the containers?

My docker version is 1.31 , Fedora 28

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment