[graphdriver] prior storage driver "aufs" failed: invalid argument #14026

Closed
phobologic opened this Issue Jun 18, 2015 · 67 comments

Comments

Projects
None yet
@phobologic

When trying docker 1.7 (from the docker maintained repo) package on Ubuntu 14.04, I now get the following error on boot:

INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: invalid argument
FATA[0000] Error starting daemon: error initializing graphdriver: invalid argument
/var/run/docker.sock is up

This led me to this bug #7321, because we use btrfs for /var/lib/docker. I've since changed it to ext4, but I'm still getting the errors above.

root@ip-10-128-16-91:/etc/init# uname -a
Linux ip-10-128-16-91 3.13.0-55-generic #92-Ubuntu SMP Sun Jun 14 18:32:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@ip-10-128-16-91:/etc/init# mount | grep docker
/dev/xvdh on /var/lib/docker type ext4 (rw)
root@ip-10-128-16-91:/etc/init# docker --version
Docker version 1.7.0, build 0baf609
@phobologic

This comment has been minimized.

Show comment
Hide comment
@phobologic

phobologic Jun 18, 2015

Environment Details:

Fresh AWS c4.xlarge instance

Couldn't run docker info since the daemon wasn't able to start.

Steps to Reproduce:

  1. Install docker 1.7 from package repository on Ubuntu 14.04, when it attempts to start it fails.

Environment Details:

Fresh AWS c4.xlarge instance

Couldn't run docker info since the daemon wasn't able to start.

Steps to Reproduce:

  1. Install docker 1.7 from package repository on Ubuntu 14.04, when it attempts to start it fails.
@isntall

This comment has been minimized.

Show comment
Hide comment
@isntall

isntall Jun 19, 2015

I've just run into this issue too, on AWS.

isntall commented Jun 19, 2015

I've just run into this issue too, on AWS.

@PhE

This comment has been minimized.

Show comment
Hide comment
@PhE

PhE Jun 19, 2015

Same error here :

INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: driver not supported 
FATA[0000] Error starting daemon: error initializing graphdriver: driver not supported 

I am running docker in Linux Mint 17.1.
Docker was automatically updated from 1.6.2 to 1.7.
It worked fine before.

PhE commented Jun 19, 2015

Same error here :

INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: driver not supported 
FATA[0000] Error starting daemon: error initializing graphdriver: driver not supported 

I am running docker in Linux Mint 17.1.
Docker was automatically updated from 1.6.2 to 1.7.
It worked fine before.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jun 19, 2015

Contributor

@PhE you have a completely different error message and as such a different issue. Most likely you upgraded your kernel and did not install that kernels extras package, which includes the aufs kernel mods.

Contributor

cpuguy83 commented Jun 19, 2015

@PhE you have a completely different error message and as such a different issue. Most likely you upgraded your kernel and did not install that kernels extras package, which includes the aufs kernel mods.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jun 19, 2015

Contributor

@phobologic That error message is coming directly from the kernel. Seems there's some issue with it being able to setup the mounts.
Can you stop docker and check if you have any aufs mounts in /proc/mounts?

Contributor

cpuguy83 commented Jun 19, 2015

@phobologic That error message is coming directly from the kernel. Seems there's some issue with it being able to setup the mounts.
Can you stop docker and check if you have any aufs mounts in /proc/mounts?

@isntall

This comment has been minimized.

Show comment
Hide comment
@isntall

isntall Jun 19, 2015

My issue came from a kernel upgrade.
I was running

    # apt-get -y update
    # apt-get -y upgrade
    # apt-get -y dist-upgrade
    # apt-get -y install aufs-tools openjdk-7-jre curl wget git vim make php-pear \
php5-dev php5-curl python mc gawk ssh grep sudo htop nmon mysql-client php5-cli \
sqlite3 sysstat sysdig linux-headers-$(uname -r) lxc bsdtar
    # apt-get -y install linux-image-extra-$(uname -r)
    # modprobe aufs
    # apt-get clean

So i think the uname -r was using the current kernel and not the new kernel that the dist-upgrade. So aufs wasn't working properly.

isntall commented Jun 19, 2015

My issue came from a kernel upgrade.
I was running

    # apt-get -y update
    # apt-get -y upgrade
    # apt-get -y dist-upgrade
    # apt-get -y install aufs-tools openjdk-7-jre curl wget git vim make php-pear \
php5-dev php5-curl python mc gawk ssh grep sudo htop nmon mysql-client php5-cli \
sqlite3 sysstat sysdig linux-headers-$(uname -r) lxc bsdtar
    # apt-get -y install linux-image-extra-$(uname -r)
    # modprobe aufs
    # apt-get clean

So i think the uname -r was using the current kernel and not the new kernel that the dist-upgrade. So aufs wasn't working properly.

@benguillet

This comment has been minimized.

Show comment
Hide comment
@benguillet

benguillet Jun 19, 2015

@cpuguy83 I work with @phobologic , we do have an aufs mount in /proc/mounts:

/dev/disk/by-uuid/82e281c0-5930-4069-b6d7-799dd1e14e66 /var/lib/docker/aufs ext4 rw,relatime,discard,data=ordered 0 0

Any advice?

@cpuguy83 I work with @phobologic , we do have an aufs mount in /proc/mounts:

/dev/disk/by-uuid/82e281c0-5930-4069-b6d7-799dd1e14e66 /var/lib/docker/aufs ext4 rw,relatime,discard,data=ordered 0 0

Any advice?

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jun 19, 2015

Contributor

@benjamin-guillet With docker stopped, call unmount on /var/lib/docker/aufs, then try to start docker again.

Contributor

cpuguy83 commented Jun 19, 2015

@benjamin-guillet With docker stopped, call unmount on /var/lib/docker/aufs, then try to start docker again.

@benguillet

This comment has been minimized.

Show comment
Hide comment
@benguillet

benguillet Jun 19, 2015

@cpuguy83

root@*:~# stop docker
stop: Unknown instance:
root@*:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8115168 1782524   5897368  24% /
none                   4       0         4   0% /sys/fs/cgroup
udev             3816100      12   3816088   1% /dev
tmpfs             765956     356    765600   1% /run
none                5120       0      5120   0% /run/lock
none             3829776       0   3829776   0% /run/shm
none              102400       0    102400   0% /run/user
/dev/xvdh      104857600     512 102731520   1% /var/lib/docker
root@*:~# umount /var/lib/docker
root@*:~# umount /var/lib/docker/aufs
root@*:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8115168 1782532   5897360  24% /
none                   4       0         4   0% /sys/fs/cgroup
udev             3816100      12   3816088   1% /dev
tmpfs             765956     356    765600   1% /run
none                5120       0      5120   0% /run/lock
none             3829776       0   3829776   0% /run/shm
none              102400       0    102400   0% /run/user
root@*:~# mount /var/lib/docker
root@*:~# start docker
docker start/running, process 25124
root@*:~# docker ps
Cannot connect to the Docker daemon. Is 'docker -d' running on this host?
root@*:~#

Docker starts correctly and works if I umount /var/lib/docker. We use it with an external EBS volume.

@cpuguy83

root@*:~# stop docker
stop: Unknown instance:
root@*:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8115168 1782524   5897368  24% /
none                   4       0         4   0% /sys/fs/cgroup
udev             3816100      12   3816088   1% /dev
tmpfs             765956     356    765600   1% /run
none                5120       0      5120   0% /run/lock
none             3829776       0   3829776   0% /run/shm
none              102400       0    102400   0% /run/user
/dev/xvdh      104857600     512 102731520   1% /var/lib/docker
root@*:~# umount /var/lib/docker
root@*:~# umount /var/lib/docker/aufs
root@*:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8115168 1782532   5897360  24% /
none                   4       0         4   0% /sys/fs/cgroup
udev             3816100      12   3816088   1% /dev
tmpfs             765956     356    765600   1% /run
none                5120       0      5120   0% /run/lock
none             3829776       0   3829776   0% /run/shm
none              102400       0    102400   0% /run/user
root@*:~# mount /var/lib/docker
root@*:~# start docker
docker start/running, process 25124
root@*:~# docker ps
Cannot connect to the Docker daemon. Is 'docker -d' running on this host?
root@*:~#

Docker starts correctly and works if I umount /var/lib/docker. We use it with an external EBS volume.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jun 19, 2015

Contributor

@benjamin-guillet Your response is a little ambiguous. Is this resolved?

Contributor

cpuguy83 commented Jun 19, 2015

@benjamin-guillet Your response is a little ambiguous. Is this resolved?

@phobologic

This comment has been minimized.

Show comment
Hide comment
@phobologic

phobologic Jun 19, 2015

@cpuguy83 @benjamin-guillet Sorry - no, it doesn't fix it for us. Our setup includes mounting an EBS volume as /var/lib/docker. For some reason, even after unmounting that, then umounting /var/lib/docker/aufs then remounting /var/lib/docker we see the issues above. I feel like we're missing a step here - should we leave /var/lib/docker mounted and then unmount /var/lib/docker/aufs?

Thanks for all your help with this.

@cpuguy83 @benjamin-guillet Sorry - no, it doesn't fix it for us. Our setup includes mounting an EBS volume as /var/lib/docker. For some reason, even after unmounting that, then umounting /var/lib/docker/aufs then remounting /var/lib/docker we see the issues above. I feel like we're missing a step here - should we leave /var/lib/docker mounted and then unmount /var/lib/docker/aufs?

Thanks for all your help with this.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jun 19, 2015

Contributor

@phobologic Yes, if you are mounting /var/lib/docker elsewhere, definitely just unmount the aufs dir.
Generally Docker would unmount this on exit.

Contributor

cpuguy83 commented Jun 19, 2015

@phobologic Yes, if you are mounting /var/lib/docker elsewhere, definitely just unmount the aufs dir.
Generally Docker would unmount this on exit.

@phobologic

This comment has been minimized.

Show comment
Hide comment
@phobologic

phobologic Jun 19, 2015

@cpuguy83 Ok, we'll give that a try and see that it works. Thanks.

@cpuguy83 Ok, we'll give that a try and see that it works. Thanks.

@bkeroackdsc

This comment has been minimized.

Show comment
Hide comment
@bkeroackdsc

bkeroackdsc Jun 25, 2015

see https://gist.github.com/parente/025dcb2b9400a12d1a9f

This is another example of Docker having major showstopping issues with one of the most widely-used distros (see #3182 )

see https://gist.github.com/parente/025dcb2b9400a12d1a9f

This is another example of Docker having major showstopping issues with one of the most widely-used distros (see #3182 )

@phobologic

This comment has been minimized.

Show comment
Hide comment
@phobologic

phobologic Jun 25, 2015

In the end it looks like we had to manually remove /var/lib/docker/aufs after mounting the newly formatted external volume and starting docker1.7. I'm not sure why it would be getting created in the first place, and I'm not entirely sure what is creating it.

We also made sure to unmount /var/lib/docker/aufs before we mounted the new volume. I'm not entirely sure which of our steps finally fixed it for us, but I hope to have some time to narrow it down.

In the end it looks like we had to manually remove /var/lib/docker/aufs after mounting the newly formatted external volume and starting docker1.7. I'm not sure why it would be getting created in the first place, and I'm not entirely sure what is creating it.

We also made sure to unmount /var/lib/docker/aufs before we mounted the new volume. I'm not entirely sure which of our steps finally fixed it for us, but I hope to have some time to narrow it down.

@sleaze

This comment has been minimized.

Show comment
Hide comment
@sleaze

sleaze Aug 4, 2015

@phobologic thanks for the info, you just saved some bacon!

sleaze commented Aug 4, 2015

@phobologic thanks for the info, you just saved some bacon!

@Kelindar

This comment has been minimized.

Show comment
Hide comment
@Kelindar

Kelindar Aug 5, 2015

Had the same issue when upgrading to kernel 4.0.x on Ubuntu, where aufs is no longer supported and rm -rf /var/lib/docker/aufs did the trick.

Kelindar commented Aug 5, 2015

Had the same issue when upgrading to kernel 4.0.x on Ubuntu, where aufs is no longer supported and rm -rf /var/lib/docker/aufs did the trick.

@sdiemert

This comment has been minimized.

Show comment
Hide comment
@sdiemert

sdiemert Aug 21, 2015

@Kelindar thanks! That worked for me.

@Kelindar thanks! That worked for me.

@Toub

This comment has been minimized.

Show comment
Hide comment
@Toub

Toub Aug 30, 2015

Thanks @Kelindar

I had @PhE error message ("driver not supported ") on Debian Testing with kernel 4.0, and your solution worked for me.

Toub commented Aug 30, 2015

Thanks @Kelindar

I had @PhE error message ("driver not supported ") on Debian Testing with kernel 4.0, and your solution worked for me.

@phobologic

This comment has been minimized.

Show comment
Hide comment
@phobologic

phobologic Aug 31, 2015

So, I had this bite me again today. I was able to dig in some more. Here's some background for what happens on my servers:

  1. I build an AMI using ansible that installs most of my non-server specific software. This includes the docker ubuntu package, which comes with startup scripts.
  2. When I boot various servers, I check to see if there is a drive mounted @ /dev/xvdh. If there is, that is assumed to be for /var/lib/docker. I then:
  3. shut down docker (using the upstart init script)
  4. initialize (mkfs) the device
  5. mount it as /var/lib/docker
  6. Start docker

The issue seems to be that sometimes, when stopping docker, it does not umount not only /var/lib/docker/aufs, but a subdirectory (in this last case: /var/lib/docker/aufs/mnt/ebe69f270843f32a24b4c16343f8c312897d38128bd36df8b25cfe4442a1de3e).

Since I've mounted over /var/lib/docker already with the new device, it doesn't give me the ability to unmount leftover volumes. I was able to manually fix it by doing the following:

  1. stop docker (using the init script, though it was already down due to crashing repeatedly due to this bug).
  2. umount /var/lib/docker
  3. umount /var/lib/docker/aufs/mnt/ebe69f270843f32a24b4c16343f8c312897d38128bd36df8b25cfe4442a1de3e
  4. umount /var/lib/docker/aufs
  5. mount /var/lib/docker
  6. start docker (using the init script)

At this point I'm not sure how I can deal with this in a safe way without having to get manually involved, or writing a script to handle it. My current workflow, in ansible, goes like this:

- service: name=docker state=stopped
  when: new_docker_fs|changed
# Make absolutely sure aufs doesn't exist or docker freaks
# https://github.com/docker/docker/issues/14026
- shell: /bin/umount -t aufs -a -f
  ignore_errors: yes
  when: new_docker_fs|changed
- shell: /bin/umount -f /var/lib/docker/aufs
  ignore_errors: yes
  when: new_docker_fs|changed
- mount: name=/var/lib/docker src=/dev/xvdh fstype=ext4 state=mounted
  when: new_docker_fs|changed
- file: path=/var/lib/docker/aufs state=absent
  ignore_errors: yes
  when: new_docker_fs|changed

The change being that now I call out to the umount command directly and try to force the host to unmount every aufs mount it has. I'm not sure that this will actually work, but I thought I'd at least give an update as to where I am today just in case this bites anyone else, and in case this gives @cpuguy83 any other info that might help.

So, I had this bite me again today. I was able to dig in some more. Here's some background for what happens on my servers:

  1. I build an AMI using ansible that installs most of my non-server specific software. This includes the docker ubuntu package, which comes with startup scripts.
  2. When I boot various servers, I check to see if there is a drive mounted @ /dev/xvdh. If there is, that is assumed to be for /var/lib/docker. I then:
  3. shut down docker (using the upstart init script)
  4. initialize (mkfs) the device
  5. mount it as /var/lib/docker
  6. Start docker

The issue seems to be that sometimes, when stopping docker, it does not umount not only /var/lib/docker/aufs, but a subdirectory (in this last case: /var/lib/docker/aufs/mnt/ebe69f270843f32a24b4c16343f8c312897d38128bd36df8b25cfe4442a1de3e).

Since I've mounted over /var/lib/docker already with the new device, it doesn't give me the ability to unmount leftover volumes. I was able to manually fix it by doing the following:

  1. stop docker (using the init script, though it was already down due to crashing repeatedly due to this bug).
  2. umount /var/lib/docker
  3. umount /var/lib/docker/aufs/mnt/ebe69f270843f32a24b4c16343f8c312897d38128bd36df8b25cfe4442a1de3e
  4. umount /var/lib/docker/aufs
  5. mount /var/lib/docker
  6. start docker (using the init script)

At this point I'm not sure how I can deal with this in a safe way without having to get manually involved, or writing a script to handle it. My current workflow, in ansible, goes like this:

- service: name=docker state=stopped
  when: new_docker_fs|changed
# Make absolutely sure aufs doesn't exist or docker freaks
# https://github.com/docker/docker/issues/14026
- shell: /bin/umount -t aufs -a -f
  ignore_errors: yes
  when: new_docker_fs|changed
- shell: /bin/umount -f /var/lib/docker/aufs
  ignore_errors: yes
  when: new_docker_fs|changed
- mount: name=/var/lib/docker src=/dev/xvdh fstype=ext4 state=mounted
  when: new_docker_fs|changed
- file: path=/var/lib/docker/aufs state=absent
  ignore_errors: yes
  when: new_docker_fs|changed

The change being that now I call out to the umount command directly and try to force the host to unmount every aufs mount it has. I'm not sure that this will actually work, but I thought I'd at least give an update as to where I am today just in case this bites anyone else, and in case this gives @cpuguy83 any other info that might help.

@stefanteixeira

This comment has been minimized.

Show comment
Hide comment
@stefanteixeira

stefanteixeira Sep 8, 2015

Thanks @Kelindar! I also updated the linux kernel on Ubuntu and removing aufs saved me. :)

Thanks @Kelindar! I also updated the linux kernel on Ubuntu and removing aufs saved me. :)

@dchen1107 dchen1107 referenced this issue in kubernetes/kubernetes Sep 15, 2015

Closed

Running pods fails with AUFS mount errors #10959

@agnivade

This comment has been minimized.

Show comment
Hide comment
@agnivade

agnivade Sep 28, 2015

+1 to @Kelindar. I also had the same problem after upgrading 4.0. But now all my containers are gone :(

I don't think there is any way to get those back ?

+1 to @Kelindar. I also had the same problem after upgrading 4.0. But now all my containers are gone :(

I don't think there is any way to get those back ?

@GuillaumeSeren

This comment has been minimized.

Show comment
Hide comment
@GuillaumeSeren

GuillaumeSeren Nov 1, 2015

@Kelindar Thank you this was happened on Debian also, and removing '/var/lib/docker/aufs' did work.

@Kelindar Thank you this was happened on Debian also, and removing '/var/lib/docker/aufs' did work.

@resmo

This comment has been minimized.

Show comment
Hide comment
@resmo

resmo Nov 30, 2015

can also confirm @Kelindar solution worked for me. Debian stretch.

resmo commented Nov 30, 2015

can also confirm @Kelindar solution worked for me. Debian stretch.

@sagardixit84

This comment has been minimized.

Show comment
Hide comment
@sagardixit84

sagardixit84 Jan 19, 2016

I had this issue and removing /var/lib/docker/aufs did not help.
I had /var/lib/docker bind mounted to another location, which I had to remove and then docker started successfully

I had this issue and removing /var/lib/docker/aufs did not help.
I had /var/lib/docker bind mounted to another location, which I had to remove and then docker started successfully

@shivece

This comment has been minimized.

Show comment
Hide comment
@shivece

shivece Feb 7, 2016

After unwittingly upgrading to 1.10, I got this error starting the docker daemon:

sudo tail -10f /var/log/upstart/docker.log
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: driver not supported
FATA[0000] Error starting daemon: error initializing graphdriver: driver not supported

And @Kelindar trick worked for me:

rm -rf /var/lib/docker/aufs

shivece commented Feb 7, 2016

After unwittingly upgrading to 1.10, I got this error starting the docker daemon:

sudo tail -10f /var/log/upstart/docker.log
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: driver not supported
FATA[0000] Error starting daemon: error initializing graphdriver: driver not supported

And @Kelindar trick worked for me:

rm -rf /var/lib/docker/aufs

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 16, 2016

Member

@shivece what platform are you on? How did you install/upgrade docker (using a static binary or from the apt repositories?) Did you also upgrade your system, causing the aufs requirements to no longer be in place?

Member

thaJeztah commented Feb 16, 2016

@shivece what platform are you on? How did you install/upgrade docker (using a static binary or from the apt repositories?) Did you also upgrade your system, causing the aufs requirements to no longer be in place?

@CyborTronik

This comment has been minimized.

Show comment
Hide comment
@CyborTronik

CyborTronik Feb 22, 2016

Had the same issue.
Was resolved after:

   sudo rm -rf /var/lib/docker/aufs
   sudo service docker start

Solution found there: http://log.rowanto.com/broken-docker-in-debian-jessie/

Had the same issue.
Was resolved after:

   sudo rm -rf /var/lib/docker/aufs
   sudo service docker start

Solution found there: http://log.rowanto.com/broken-docker-in-debian-jessie/

@ET-Bent

This comment has been minimized.

Show comment
Hide comment
@ET-Bent

ET-Bent Feb 26, 2016

CAUTION!
rm -rf /var/lib/docker/aufs also deletes all docker containers...

ET-Bent commented Feb 26, 2016

CAUTION!
rm -rf /var/lib/docker/aufs also deletes all docker containers...

@zave

This comment has been minimized.

Show comment
Hide comment
@zave

zave Feb 29, 2016

@Kelindar Thank you! I'm on Ubuntu 14.04, 3.13.0-79-generic, and removing '/var/lib/docker/aufs' resolved the issue for me.

zave commented Feb 29, 2016

@Kelindar Thank you! I'm on Ubuntu 14.04, 3.13.0-79-generic, and removing '/var/lib/docker/aufs' resolved the issue for me.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 29, 2016

Member

@zave Ubuntu 14.04 should have Suppprt for aufs, and is the recommended driver; did you recently update the kernel? If so, you may want to reinstall the Linux-image-extra package

Member

thaJeztah commented Feb 29, 2016

@zave Ubuntu 14.04 should have Suppprt for aufs, and is the recommended driver; did you recently update the kernel? If so, you may want to reinstall the Linux-image-extra package

@zave

This comment has been minimized.

Show comment
Hide comment
@zave

zave Feb 29, 2016

I didn't think to do that, since the fix mentioned above took care of the problem, should I do
that in any case?

zave commented Feb 29, 2016

I didn't think to do that, since the fix mentioned above took care of the problem, should I do
that in any case?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 29, 2016

Member

@zave the "fix" mentioned is basically to get rid of an actual error; the fact that you previously ran docker with the aufs driver, but the driver doesn't work. Removing the aufs directory just nukes your containers and images, so that docker picks the next available driver (which may be vfs in a worst case scenario).

The Linux-image-extra package is tied to your kernel version, so has to be reinstalled after upgrading the kernel; check https://docs.docker.com/engine/installation/linux/ubuntulinux/#prerequisites-by-ubuntu-version on how to install it

Member

thaJeztah commented Feb 29, 2016

@zave the "fix" mentioned is basically to get rid of an actual error; the fact that you previously ran docker with the aufs driver, but the driver doesn't work. Removing the aufs directory just nukes your containers and images, so that docker picks the next available driver (which may be vfs in a worst case scenario).

The Linux-image-extra package is tied to your kernel version, so has to be reinstalled after upgrading the kernel; check https://docs.docker.com/engine/installation/linux/ubuntulinux/#prerequisites-by-ubuntu-version on how to install it

@zave

This comment has been minimized.

Show comment
Hide comment
@zave

zave Feb 29, 2016

@thaJeztah Ok, I'll do that. Thank you.

zave commented Feb 29, 2016

@thaJeztah Ok, I'll do that. Thank you.

@zave

This comment has been minimized.

Show comment
Hide comment
@zave

zave Mar 1, 2016

@thaJeztah I did it. All is well again. I remember having done that when I installed the Docker Engine. I'd forgotten all about that. Great recommendation. Thanks again.

zave commented Mar 1, 2016

@thaJeztah I did it. All is well again. I remember having done that when I installed the Docker Engine. I'd forgotten all about that. Great recommendation. Thanks again.

@schinzelh

This comment has been minimized.

Show comment
Hide comment
@schinzelh

schinzelh Mar 5, 2016

I think i have found a related issue:

I am running

"Docker version 1.10.2, build c3959b1" on
"Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-55-generic x86_64)"

All is fine when running Docker with default setup. But as soon as i change the Docker base storage directory via "-g" parameter in DOCKER_OPTS in /etc/default/docker i get the error

/var/run/docker.sock is up
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: invalid argument
FATA[0000] Error starting daemon: error initializing graphdriver: invalid argument

in my message log. I did not find a way around this yet.

I think i have found a related issue:

I am running

"Docker version 1.10.2, build c3959b1" on
"Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-55-generic x86_64)"

All is fine when running Docker with default setup. But as soon as i change the Docker base storage directory via "-g" parameter in DOCKER_OPTS in /etc/default/docker i get the error

/var/run/docker.sock is up
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: invalid argument
FATA[0000] Error starting daemon: error initializing graphdriver: invalid argument

in my message log. I did not find a way around this yet.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Mar 5, 2016

Member

@schinzelh note that /etc/default/docker (and DOCKER_OPTS) is only used for systems running upstart/sysvinit, not systems that use systemd. Have you checked if the options you provided are actually picked up / used by the daemon?

Member

thaJeztah commented Mar 5, 2016

@schinzelh note that /etc/default/docker (and DOCKER_OPTS) is only used for systems running upstart/sysvinit, not systems that use systemd. Have you checked if the options you provided are actually picked up / used by the daemon?

@schinzelh

This comment has been minimized.

Show comment
Hide comment
@schinzelh

schinzelh Mar 5, 2016

@thaJeztah According to https://docs.docker.com/engine/installation/linux/ubuntulinux/#configure-docker-to-start-on-boot Ubuntu 14.04 is using upstart and as I see the error message in

/var/log/upstart/docker.log

i think i am good here. I have used

DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4 -g /srv/docker"

and the directory is picked up by the daemon - but it fails with the error message above while creating

 /srv/docker/aufs

@thaJeztah According to https://docs.docker.com/engine/installation/linux/ubuntulinux/#configure-docker-to-start-on-boot Ubuntu 14.04 is using upstart and as I see the error message in

/var/log/upstart/docker.log

i think i am good here. I have used

DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4 -g /srv/docker"

and the directory is picked up by the daemon - but it fails with the error message above while creating

 /srv/docker/aufs
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Mar 5, 2016

Member

@schinzelh did you perhaps upgrade your system (and kernel)? In that case, check if the linux-image-extra package is installed properly for the kernel https://docs.docker.com/engine/installation/linux/ubuntulinux/#prerequisites-by-ubuntu-version

Member

thaJeztah commented Mar 5, 2016

@schinzelh did you perhaps upgrade your system (and kernel)? In that case, check if the linux-image-extra package is installed properly for the kernel https://docs.docker.com/engine/installation/linux/ubuntulinux/#prerequisites-by-ubuntu-version

@zave

This comment has been minimized.

Show comment
Hide comment
@zave

zave Mar 5, 2016

That sounds familiar. That's what solved my problem.

zave commented Mar 5, 2016

That sounds familiar. That's what solved my problem.

@schinzelh

This comment has been minimized.

Show comment
Hide comment
@schinzelh

schinzelh Mar 6, 2016

@thaJeztah i already did. All packages including linux.image-extra are up to date.

Like i wrote: If Docker is configured to use its default storage location in

/var/lib/docker

all is working 100% perfect. But if i move the location to e.g.

/srv/docker

via the -g parameter the daemon fails to start with

/var/run/docker.sock is up
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: invalid argument
FATA[0000] Error starting daemon: error initializing graphdriver: invalid argument

Would a AMI image be of any use for you to reproduce this issue?

@thaJeztah i already did. All packages including linux.image-extra are up to date.

Like i wrote: If Docker is configured to use its default storage location in

/var/lib/docker

all is working 100% perfect. But if i move the location to e.g.

/srv/docker

via the -g parameter the daemon fails to start with

/var/run/docker.sock is up
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: invalid argument
FATA[0000] Error starting daemon: error initializing graphdriver: invalid argument

Would a AMI image be of any use for you to reproduce this issue?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Mar 6, 2016

Member

Is there something special with /srv/docker; Is it a regular directory, a mount?

Member

thaJeztah commented Mar 6, 2016

Is there something special with /srv/docker; Is it a regular directory, a mount?

@schinzelh

This comment has been minimized.

Show comment
Hide comment
@schinzelh

schinzelh Mar 6, 2016

/srv is actually a mount of /dev/xvdh

ubuntu@ip-10-73-178-189:~$ lsblk
NAME  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
xvdb  202:16   0   840G  0 disk /mnt
**xvdh  202:112  0    40G  0 disk /srv**
xvda1 202:1    0    25G  0 disk /


ubuntu@ip-10-73-178-189:~$ mount
/dev/xvda1 on / type ext4 (rw,discard)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
/dev/xvdb on /mnt type ext3 (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
**/dev/xvdh on /srv type ext3 (rw)**
/mnt/home on /home type none (rw,bind)

/srv is actually a mount of /dev/xvdh

ubuntu@ip-10-73-178-189:~$ lsblk
NAME  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
xvdb  202:16   0   840G  0 disk /mnt
**xvdh  202:112  0    40G  0 disk /srv**
xvda1 202:1    0    25G  0 disk /


ubuntu@ip-10-73-178-189:~$ mount
/dev/xvda1 on / type ext4 (rw,discard)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
/dev/xvdb on /mnt type ext3 (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
**/dev/xvdh on /srv type ext3 (rw)**
/mnt/home on /home type none (rw,bind)
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Mar 6, 2016

Member

@schinzelh alright, one difference I see is ext3 vs ext4. Just to verify; if you use /var/lib/foobar as graphdir, does it work correctly?

If that works correctly, might be worth checking if things are missing, using the check-config.sh script

Member

thaJeztah commented Mar 6, 2016

@schinzelh alright, one difference I see is ext3 vs ext4. Just to verify; if you use /var/lib/foobar as graphdir, does it work correctly?

If that works correctly, might be worth checking if things are missing, using the check-config.sh script

@schinzelh

This comment has been minimized.

Show comment
Hide comment
@schinzelh

schinzelh Mar 7, 2016

@thaJeztah Thanks for looking into this. I finally found out that it was caused by user error: The mount was not done via /etc/fstab but via a script executed at server startup in /etc/rc.local

This caused a race condition between docker deamon startup and mount command on server boot, which corrupted the docker storage directory.

Thanks for your help 👍

@thaJeztah Thanks for looking into this. I finally found out that it was caused by user error: The mount was not done via /etc/fstab but via a script executed at server startup in /etc/rc.local

This caused a race condition between docker deamon startup and mount command on server boot, which corrupted the docker storage directory.

Thanks for your help 👍

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Mar 7, 2016

Member

@schinzelh oh, that explains things! Good to hear you got it resolved!

Member

thaJeztah commented Mar 7, 2016

@schinzelh oh, that explains things! Good to hear you got it resolved!

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Mar 7, 2016

Member

Alright, looking at the discussion above, I'm going to close this issue; this issue has been collecting different issues (resulting in the same "error" message). As far as I can see there's no bug involved here.

Feel free to comment here after closing, but please read the discussion above if your error can be the result of one of the things discussed above

Member

thaJeztah commented Mar 7, 2016

Alright, looking at the discussion above, I'm going to close this issue; this issue has been collecting different issues (resulting in the same "error" message). As far as I can see there's no bug involved here.

Feel free to comment here after closing, but please read the discussion above if your error can be the result of one of the things discussed above

@thaJeztah thaJeztah closed this Mar 7, 2016

@jmccann jmccann referenced this issue in jmccann/aircover-cookbook Apr 21, 2016

Open

Issue initializing AUFS #1

@dwu-zingbox

This comment has been minimized.

Show comment
Hide comment
@dwu-zingbox

dwu-zingbox Apr 29, 2016

My experience show docker doesn't like to run under a root directory, meaning /var/lib/docker or what is specified by -g cannot be a root directory of an EBS volume or a LVM of EBS volume. Giving docker one more level of directory in the same volume through -g solved the problem.

I have Docker version 1.10.2, build c3959b1 and ubuntu 14.04, Linux 3.13.0-79

# df
Filesystem                     1K-blocks    Used Available Use% Mounted on
...
/dev/mapper/data-docker         10190136   23144   9626320   1% /data/docker

DOCKER_OPTS="-g /data/docker/docker" works but DOCKER_OPTS="-g /data/docker" doesn't.

dwu-zingbox commented Apr 29, 2016

My experience show docker doesn't like to run under a root directory, meaning /var/lib/docker or what is specified by -g cannot be a root directory of an EBS volume or a LVM of EBS volume. Giving docker one more level of directory in the same volume through -g solved the problem.

I have Docker version 1.10.2, build c3959b1 and ubuntu 14.04, Linux 3.13.0-79

# df
Filesystem                     1K-blocks    Used Available Use% Mounted on
...
/dev/mapper/data-docker         10190136   23144   9626320   1% /data/docker

DOCKER_OPTS="-g /data/docker/docker" works but DOCKER_OPTS="-g /data/docker" doesn't.

@siliconcow

This comment has been minimized.

Show comment
Hide comment
@siliconcow

siliconcow Apr 29, 2016

Yep, weird bug but reproduced it on 14.04 w/ docker 1.6.2

Yep, weird bug but reproduced it on 14.04 w/ docker 1.6.2

@pmontrasio

This comment has been minimized.

Show comment
Hide comment
@pmontrasio

pmontrasio Jun 10, 2016

The solution by @thaJeztah worked for me. I upgraded to linux 3.13.0-88-generic and I had to sudo apt-get install linux-image-extra-3.13.0-88-generic. Started docker and it works.

Maybe the error in the log file

[graphdriver] prior storage driver "aufs" failed: driver not supported
Error starting daemon: error initializing graphdriver: driver not supported

should hint to that package. How about adding a line like this?

You could have to upgrade the linux-image-extra to the version matching your kernel.
Example: sudo apt-get install linux-image-extra-3.13.0-88-generic

Better if with the appropriate kernel version and the appropriate package manager according to the distribution.

The solution by @thaJeztah worked for me. I upgraded to linux 3.13.0-88-generic and I had to sudo apt-get install linux-image-extra-3.13.0-88-generic. Started docker and it works.

Maybe the error in the log file

[graphdriver] prior storage driver "aufs" failed: driver not supported
Error starting daemon: error initializing graphdriver: driver not supported

should hint to that package. How about adding a line like this?

You could have to upgrade the linux-image-extra to the version matching your kernel.
Example: sudo apt-get install linux-image-extra-3.13.0-88-generic

Better if with the appropriate kernel version and the appropriate package manager according to the distribution.

@jamesfryer

This comment has been minimized.

Show comment
Hide comment
@jamesfryer

jamesfryer Aug 7, 2016

Same issue, sudo apt-get install linux-image-extra-3.13.0-88-generic fixed it for me.

Same issue, sudo apt-get install linux-image-extra-3.13.0-88-generic fixed it for me.

@jocull

This comment has been minimized.

Show comment
Hide comment
@jocull

jocull Sep 12, 2016

This happens to me all the time. It seems like it happens after running dist-upgrades in Ubuntu, and I always have to go in and run this:

sudo apt-get install linux-headers-$(uname -r) linux-image-extra-$(uname -r)

jocull commented Sep 12, 2016

This happens to me all the time. It seems like it happens after running dist-upgrades in Ubuntu, and I always have to go in and run this:

sudo apt-get install linux-headers-$(uname -r) linux-image-extra-$(uname -r)
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Sep 12, 2016

Member

@jocull correct; if a dist-upgrade upgrades the kernel, chances are you need to install the correct version of the linux-image-extra package for the updated kernel. I think that installing the linux-image-extra-virtual package would help in that situation (and was recently added to the docs, to match what the install script installs; #25614)

Member

thaJeztah commented Sep 12, 2016

@jocull correct; if a dist-upgrade upgrades the kernel, chances are you need to install the correct version of the linux-image-extra package for the updated kernel. I think that installing the linux-image-extra-virtual package would help in that situation (and was recently added to the docs, to match what the install script installs; #25614)

@jocull

This comment has been minimized.

Show comment
Hide comment
@jocull

jocull Sep 12, 2016

I will try that and see if it makes a difference! Thank you!

jocull commented Sep 12, 2016

I will try that and see if it makes a difference! Thank you!

@MouhamedAymen

This comment has been minimized.

Show comment
Hide comment
@MouhamedAymen

MouhamedAymen Sep 17, 2016

I had the same problem, after updating to 1.12.1 and rebooting.
I solved it by removing /var/lib/docker/overlay (or just renaming it smth else, just in case, you know) and restarting docker. I have no idea why it worked, but glad it did.

I had the same problem, after updating to 1.12.1 and rebooting.
I solved it by removing /var/lib/docker/overlay (or just renaming it smth else, just in case, you know) and restarting docker. I have no idea why it worked, but glad it did.

@toudidel

This comment has been minimized.

Show comment
Hide comment
@toudidel

toudidel Sep 26, 2016

I had the same problem if I wanted to changed my docker images location.
I performed solution from http://stackoverflow.com/a/34731550 and added file /etc/systemd/system/docker.service.d/docker-storage.conf with content:

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --graph="/mnt"

After restarting the system I had the problem. If I delete this file and return to default location (/var/lib/docker) everything is OK.

I had the same problem if I wanted to changed my docker images location.
I performed solution from http://stackoverflow.com/a/34731550 and added file /etc/systemd/system/docker.service.d/docker-storage.conf with content:

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --graph="/mnt"

After restarting the system I had the problem. If I delete this file and return to default location (/var/lib/docker) everything is OK.

@mathieufrh

This comment has been minimized.

Show comment
Hide comment
@mathieufrh

mathieufrh Oct 11, 2016

aufs is not supported by latest kernels, so you should use overlayfs instead.
I just edit the /etc/systemd/system/docker.service.d/docker.conf this way:

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --storage-driver=overlay

Then do:

# systemctl stop docker.service
# systemctl daemon-reload
# systemctl start docker.service

aufs is not supported by latest kernels, so you should use overlayfs instead.
I just edit the /etc/systemd/system/docker.service.d/docker.conf this way:

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --storage-driver=overlay

Then do:

# systemctl stop docker.service
# systemctl daemon-reload
# systemctl start docker.service
@f3l1x

This comment has been minimized.

Show comment
Hide comment
@f3l1x

f3l1x Nov 29, 2016

I'm running Linux Mint 18 and I've updated from kernel 4.4.0 to 4.8.11. (29.11.2016).

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2016-11-29 13:50:20 CET; 8s ago
     Docs: https://docs.docker.com
 Main PID: 9128 (code=exited, status=1/FAILURE)

Nov 29 13:50:19 XPS systemd[1]: Starting Docker Application Container Engine...
Nov 29 13:50:19 XPS dockerd[9128]: time="2016-11-29T13:50:19.400041011+01:00" level=info msg="libcontainerd: new containerd process, pid: 9135"
Nov 29 13:50:20 XPS dockerd[9128]: time="2016-11-29T13:50:20.406203559+01:00" level=error msg="[graphdriver] prior storage driver \"aufs\" failed: driver not supported"
Nov 29 13:50:20 XPS dockerd[9128]: time="2016-11-29T13:50:20.406360943+01:00" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"
Nov 29 13:50:20 XPS systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 13:50:20 XPS systemd[1]: Failed to start Docker Application Container Engine.
Nov 29 13:50:20 XPS systemd[1]: docker.service: Unit entered failed state.
Nov 29 13:50:20 XPS systemd[1]: docker.service: Failed with result 'exit-code'.

This helps me out.

 sudo rm -rf /var/lib/docker/aufs
 sudo service docker start

f3l1x commented Nov 29, 2016

I'm running Linux Mint 18 and I've updated from kernel 4.4.0 to 4.8.11. (29.11.2016).

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2016-11-29 13:50:20 CET; 8s ago
     Docs: https://docs.docker.com
 Main PID: 9128 (code=exited, status=1/FAILURE)

Nov 29 13:50:19 XPS systemd[1]: Starting Docker Application Container Engine...
Nov 29 13:50:19 XPS dockerd[9128]: time="2016-11-29T13:50:19.400041011+01:00" level=info msg="libcontainerd: new containerd process, pid: 9135"
Nov 29 13:50:20 XPS dockerd[9128]: time="2016-11-29T13:50:20.406203559+01:00" level=error msg="[graphdriver] prior storage driver \"aufs\" failed: driver not supported"
Nov 29 13:50:20 XPS dockerd[9128]: time="2016-11-29T13:50:20.406360943+01:00" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"
Nov 29 13:50:20 XPS systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 13:50:20 XPS systemd[1]: Failed to start Docker Application Container Engine.
Nov 29 13:50:20 XPS systemd[1]: docker.service: Unit entered failed state.
Nov 29 13:50:20 XPS systemd[1]: docker.service: Failed with result 'exit-code'.

This helps me out.

 sudo rm -rf /var/lib/docker/aufs
 sudo service docker start
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 29, 2016

Member

Be aware that sudo rm -rf /var/lib/docker/aufs removes all existing images and containers. Basically it's a "factory reset"

Member

thaJeztah commented Nov 29, 2016

Be aware that sudo rm -rf /var/lib/docker/aufs removes all existing images and containers. Basically it's a "factory reset"

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Nov 29, 2016

Contributor

And the real fix is to re-install the aufs kernel module when you upgrade your kernel.

Contributor

cpuguy83 commented Nov 29, 2016

And the real fix is to re-install the aufs kernel module when you upgrade your kernel.

@jocull

This comment has been minimized.

Show comment
Hide comment
@jocull

jocull Dec 29, 2016

Just to follow up on @thaJeztah's comment, installing the -virtual packages did not correct the issue on my end.

This could be related to using Dokku, or maybe not. I really don't know, but it still breaks every time I run a system upgrade.

jocull commented Dec 29, 2016

Just to follow up on @thaJeztah's comment, installing the -virtual packages did not correct the issue on my end.

This could be related to using Dokku, or maybe not. I really don't know, but it still breaks every time I run a system upgrade.

@bogste

This comment has been minimized.

Show comment
Hide comment
@bogste

bogste Feb 4, 2017

Thank you @Kelindar for the answer at docker#14026 (comment)

I just upgraded the Kernel to 4.9 and was getting

[graphdriver] prior storage driver aufs failed: driver not supported

Moved the /var/lib/docker/aufs folder to my $HOME_DIR/Backups/docker (just in case) and was able to run service docker stop/start. Now it starts up as it should!

bogste commented Feb 4, 2017

Thank you @Kelindar for the answer at docker#14026 (comment)

I just upgraded the Kernel to 4.9 and was getting

[graphdriver] prior storage driver aufs failed: driver not supported

Moved the /var/lib/docker/aufs folder to my $HOME_DIR/Backups/docker (just in case) and was able to run service docker stop/start. Now it starts up as it should!

@Vadorequest

This comment has been minimized.

Show comment
Hide comment
@Vadorequest

Vadorequest Oct 9, 2017

Had the same issue, having an OVH VPS with pre-installed docker, I was getting this annoying error message. Deleting aufs directory did the trick, I was able to start docker service again after that.
I don't even know why I got this bug in the first place, maybe OVH did something with their VPS kernel on the background.

Had the same issue, having an OVH VPS with pre-installed docker, I was getting this annoying error message. Deleting aufs directory did the trick, I was able to start docker service again after that.
I don't even know why I got this bug in the first place, maybe OVH did something with their VPS kernel on the background.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 9, 2017

Member

Yes, if they upgraded the kernel and did not install the required kernel modules after that, then aufs would no longer start. Docker does not automatically pick the next available driver in that situation, because then you won't see your existing images and containers, so it deliberately fails to start to give you an option to resolve the situation.

Member

thaJeztah commented Oct 9, 2017

Yes, if they upgraded the kernel and did not install the required kernel modules after that, then aufs would no longer start. Docker does not automatically pick the next available driver in that situation, because then you won't see your existing images and containers, so it deliberately fails to start to give you an option to resolve the situation.

@tahonaPL

This comment has been minimized.

Show comment
Hide comment
@tahonaPL

tahonaPL May 2, 2018

I had this error after kernel update from 4.4.1 to 4.4.129 (Mint 18.1) . I couldn't revive docker-engine so I installed docker-ce and start working:

apt purge docker-engine
rm -Rf /var/lib/docker

and then:

https://coytar.wordpress.com/2017/03/23/docker-ce-on-linux-mint-18-1/

tahonaPL commented May 2, 2018

I had this error after kernel update from 4.4.1 to 4.4.129 (Mint 18.1) . I couldn't revive docker-engine so I installed docker-ce and start working:

apt purge docker-engine
rm -Rf /var/lib/docker

and then:

https://coytar.wordpress.com/2017/03/23/docker-ce-on-linux-mint-18-1/

@prinsherbert

This comment has been minimized.

Show comment
Hide comment
@prinsherbert

prinsherbert May 8, 2018

 # apt-get -y install aufs-tools openjdk-7-jre curl wget git vim make php-pear \
 php5-dev php5-curl python mc gawk ssh grep sudo htop nmon mysql-client php5-cli \
 sqlite3 sysstat sysdig linux-headers-$(uname -r) lxc bsdtar

What do php, python, mysql, sqlite3 and java have to do with this issue?

prinsherbert commented May 8, 2018

 # apt-get -y install aufs-tools openjdk-7-jre curl wget git vim make php-pear \
 php5-dev php5-curl python mc gawk ssh grep sudo htop nmon mysql-client php5-cli \
 sqlite3 sysstat sysdig linux-headers-$(uname -r) lxc bsdtar

What do php, python, mysql, sqlite3 and java have to do with this issue?

@prinsherbert

This comment has been minimized.

Show comment
Hide comment
@prinsherbert

prinsherbert May 8, 2018

@Kelindar , thanks, I gave you a thumbs up and down because I think moving is better than deleting.

sudo mv /var/lib/docker/aufs /root/var-lib-docker-aufs-backup`date +%Y-%m-%d`

@Kelindar , thanks, I gave you a thumbs up and down because I think moving is better than deleting.

sudo mv /var/lib/docker/aufs /root/var-lib-docker-aufs-backup`date +%Y-%m-%d`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment