New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[graphdriver] prior storage driver "aufs" failed: invalid argument #14026

Closed
phobologic opened this Issue Jun 18, 2015 · 67 comments

Comments

Projects
None yet
@phobologic
Copy link

phobologic commented Jun 18, 2015

When trying docker 1.7 (from the docker maintained repo) package on Ubuntu 14.04, I now get the following error on boot:

INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: invalid argument
FATA[0000] Error starting daemon: error initializing graphdriver: invalid argument
/var/run/docker.sock is up

This led me to this bug #7321, because we use btrfs for /var/lib/docker. I've since changed it to ext4, but I'm still getting the errors above.

root@ip-10-128-16-91:/etc/init# uname -a
Linux ip-10-128-16-91 3.13.0-55-generic #92-Ubuntu SMP Sun Jun 14 18:32:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@ip-10-128-16-91:/etc/init# mount | grep docker
/dev/xvdh on /var/lib/docker type ext4 (rw)
root@ip-10-128-16-91:/etc/init# docker --version
Docker version 1.7.0, build 0baf609
@phobologic

This comment has been minimized.

Copy link

phobologic commented Jun 18, 2015

Environment Details:

Fresh AWS c4.xlarge instance

Couldn't run docker info since the daemon wasn't able to start.

Steps to Reproduce:

  1. Install docker 1.7 from package repository on Ubuntu 14.04, when it attempts to start it fails.
@isntall

This comment has been minimized.

Copy link

isntall commented Jun 19, 2015

I've just run into this issue too, on AWS.

@PhE

This comment has been minimized.

Copy link

PhE commented Jun 19, 2015

Same error here :

INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: driver not supported 
FATA[0000] Error starting daemon: error initializing graphdriver: driver not supported 

I am running docker in Linux Mint 17.1.
Docker was automatically updated from 1.6.2 to 1.7.
It worked fine before.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Jun 19, 2015

@PhE you have a completely different error message and as such a different issue. Most likely you upgraded your kernel and did not install that kernels extras package, which includes the aufs kernel mods.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Jun 19, 2015

@phobologic That error message is coming directly from the kernel. Seems there's some issue with it being able to setup the mounts.
Can you stop docker and check if you have any aufs mounts in /proc/mounts?

@isntall

This comment has been minimized.

Copy link

isntall commented Jun 19, 2015

My issue came from a kernel upgrade.
I was running

    # apt-get -y update
    # apt-get -y upgrade
    # apt-get -y dist-upgrade
    # apt-get -y install aufs-tools openjdk-7-jre curl wget git vim make php-pear \
php5-dev php5-curl python mc gawk ssh grep sudo htop nmon mysql-client php5-cli \
sqlite3 sysstat sysdig linux-headers-$(uname -r) lxc bsdtar
    # apt-get -y install linux-image-extra-$(uname -r)
    # modprobe aufs
    # apt-get clean

So i think the uname -r was using the current kernel and not the new kernel that the dist-upgrade. So aufs wasn't working properly.

@benguillet

This comment has been minimized.

Copy link

benguillet commented Jun 19, 2015

@cpuguy83 I work with @phobologic , we do have an aufs mount in /proc/mounts:

/dev/disk/by-uuid/82e281c0-5930-4069-b6d7-799dd1e14e66 /var/lib/docker/aufs ext4 rw,relatime,discard,data=ordered 0 0

Any advice?

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Jun 19, 2015

@benjamin-guillet With docker stopped, call unmount on /var/lib/docker/aufs, then try to start docker again.

@benguillet

This comment has been minimized.

Copy link

benguillet commented Jun 19, 2015

@cpuguy83

root@*:~# stop docker
stop: Unknown instance:
root@*:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8115168 1782524   5897368  24% /
none                   4       0         4   0% /sys/fs/cgroup
udev             3816100      12   3816088   1% /dev
tmpfs             765956     356    765600   1% /run
none                5120       0      5120   0% /run/lock
none             3829776       0   3829776   0% /run/shm
none              102400       0    102400   0% /run/user
/dev/xvdh      104857600     512 102731520   1% /var/lib/docker
root@*:~# umount /var/lib/docker
root@*:~# umount /var/lib/docker/aufs
root@*:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8115168 1782532   5897360  24% /
none                   4       0         4   0% /sys/fs/cgroup
udev             3816100      12   3816088   1% /dev
tmpfs             765956     356    765600   1% /run
none                5120       0      5120   0% /run/lock
none             3829776       0   3829776   0% /run/shm
none              102400       0    102400   0% /run/user
root@*:~# mount /var/lib/docker
root@*:~# start docker
docker start/running, process 25124
root@*:~# docker ps
Cannot connect to the Docker daemon. Is 'docker -d' running on this host?
root@*:~#

Docker starts correctly and works if I umount /var/lib/docker. We use it with an external EBS volume.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Jun 19, 2015

@benjamin-guillet Your response is a little ambiguous. Is this resolved?

@phobologic

This comment has been minimized.

Copy link

phobologic commented Jun 19, 2015

@cpuguy83 @benjamin-guillet Sorry - no, it doesn't fix it for us. Our setup includes mounting an EBS volume as /var/lib/docker. For some reason, even after unmounting that, then umounting /var/lib/docker/aufs then remounting /var/lib/docker we see the issues above. I feel like we're missing a step here - should we leave /var/lib/docker mounted and then unmount /var/lib/docker/aufs?

Thanks for all your help with this.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Jun 19, 2015

@phobologic Yes, if you are mounting /var/lib/docker elsewhere, definitely just unmount the aufs dir.
Generally Docker would unmount this on exit.

@phobologic

This comment has been minimized.

Copy link

phobologic commented Jun 19, 2015

@cpuguy83 Ok, we'll give that a try and see that it works. Thanks.

@bkeroackdsc

This comment has been minimized.

Copy link

bkeroackdsc commented Jun 25, 2015

see https://gist.github.com/parente/025dcb2b9400a12d1a9f

This is another example of Docker having major showstopping issues with one of the most widely-used distros (see #3182 )

@phobologic

This comment has been minimized.

Copy link

phobologic commented Jun 25, 2015

In the end it looks like we had to manually remove /var/lib/docker/aufs after mounting the newly formatted external volume and starting docker1.7. I'm not sure why it would be getting created in the first place, and I'm not entirely sure what is creating it.

We also made sure to unmount /var/lib/docker/aufs before we mounted the new volume. I'm not entirely sure which of our steps finally fixed it for us, but I hope to have some time to narrow it down.

@sleaze

This comment has been minimized.

Copy link

sleaze commented Aug 4, 2015

@phobologic thanks for the info, you just saved some bacon!

@Kelindar

This comment has been minimized.

Copy link

Kelindar commented Aug 5, 2015

Had the same issue when upgrading to kernel 4.0.x on Ubuntu, where aufs is no longer supported and rm -rf /var/lib/docker/aufs did the trick.

@sdiemert

This comment has been minimized.

Copy link

sdiemert commented Aug 21, 2015

@Kelindar thanks! That worked for me.

@Toub

This comment has been minimized.

Copy link

Toub commented Aug 30, 2015

Thanks @Kelindar

I had @PhE error message ("driver not supported ") on Debian Testing with kernel 4.0, and your solution worked for me.

@phobologic

This comment has been minimized.

Copy link

phobologic commented Aug 31, 2015

So, I had this bite me again today. I was able to dig in some more. Here's some background for what happens on my servers:

  1. I build an AMI using ansible that installs most of my non-server specific software. This includes the docker ubuntu package, which comes with startup scripts.
  2. When I boot various servers, I check to see if there is a drive mounted @ /dev/xvdh. If there is, that is assumed to be for /var/lib/docker. I then:
  3. shut down docker (using the upstart init script)
  4. initialize (mkfs) the device
  5. mount it as /var/lib/docker
  6. Start docker

The issue seems to be that sometimes, when stopping docker, it does not umount not only /var/lib/docker/aufs, but a subdirectory (in this last case: /var/lib/docker/aufs/mnt/ebe69f270843f32a24b4c16343f8c312897d38128bd36df8b25cfe4442a1de3e).

Since I've mounted over /var/lib/docker already with the new device, it doesn't give me the ability to unmount leftover volumes. I was able to manually fix it by doing the following:

  1. stop docker (using the init script, though it was already down due to crashing repeatedly due to this bug).
  2. umount /var/lib/docker
  3. umount /var/lib/docker/aufs/mnt/ebe69f270843f32a24b4c16343f8c312897d38128bd36df8b25cfe4442a1de3e
  4. umount /var/lib/docker/aufs
  5. mount /var/lib/docker
  6. start docker (using the init script)

At this point I'm not sure how I can deal with this in a safe way without having to get manually involved, or writing a script to handle it. My current workflow, in ansible, goes like this:

- service: name=docker state=stopped
  when: new_docker_fs|changed
# Make absolutely sure aufs doesn't exist or docker freaks
# https://github.com/docker/docker/issues/14026
- shell: /bin/umount -t aufs -a -f
  ignore_errors: yes
  when: new_docker_fs|changed
- shell: /bin/umount -f /var/lib/docker/aufs
  ignore_errors: yes
  when: new_docker_fs|changed
- mount: name=/var/lib/docker src=/dev/xvdh fstype=ext4 state=mounted
  when: new_docker_fs|changed
- file: path=/var/lib/docker/aufs state=absent
  ignore_errors: yes
  when: new_docker_fs|changed

The change being that now I call out to the umount command directly and try to force the host to unmount every aufs mount it has. I'm not sure that this will actually work, but I thought I'd at least give an update as to where I am today just in case this bites anyone else, and in case this gives @cpuguy83 any other info that might help.

@stefanteixeira

This comment has been minimized.

Copy link

stefanteixeira commented Sep 8, 2015

Thanks @Kelindar! I also updated the linux kernel on Ubuntu and removing aufs saved me. :)

@agnivade

This comment has been minimized.

Copy link

agnivade commented Sep 28, 2015

+1 to @Kelindar. I also had the same problem after upgrading 4.0. But now all my containers are gone :(

I don't think there is any way to get those back ?

@GuillaumeSeren

This comment has been minimized.

Copy link

GuillaumeSeren commented Nov 1, 2015

@Kelindar Thank you this was happened on Debian also, and removing '/var/lib/docker/aufs' did work.

@resmo

This comment has been minimized.

Copy link

resmo commented Nov 30, 2015

can also confirm @Kelindar solution worked for me. Debian stretch.

@sagardixit84

This comment has been minimized.

Copy link

sagardixit84 commented Jan 19, 2016

I had this issue and removing /var/lib/docker/aufs did not help.
I had /var/lib/docker bind mounted to another location, which I had to remove and then docker started successfully

@shivece

This comment has been minimized.

Copy link

shivece commented Feb 7, 2016

After unwittingly upgrading to 1.10, I got this error starting the docker daemon:

sudo tail -10f /var/log/upstart/docker.log
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: driver not supported
FATA[0000] Error starting daemon: error initializing graphdriver: driver not supported

And @Kelindar trick worked for me:

rm -rf /var/lib/docker/aufs

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Feb 16, 2016

@shivece what platform are you on? How did you install/upgrade docker (using a static binary or from the apt repositories?) Did you also upgrade your system, causing the aufs requirements to no longer be in place?

@CyborTronik

This comment has been minimized.

Copy link

CyborTronik commented Feb 22, 2016

Had the same issue.
Was resolved after:

   sudo rm -rf /var/lib/docker/aufs
   sudo service docker start

Solution found there: http://log.rowanto.com/broken-docker-in-debian-jessie/

@ET-Bent

This comment has been minimized.

Copy link

ET-Bent commented Feb 26, 2016

CAUTION!
rm -rf /var/lib/docker/aufs also deletes all docker containers...

@schinzelh

This comment has been minimized.

Copy link

schinzelh commented Mar 7, 2016

@thaJeztah Thanks for looking into this. I finally found out that it was caused by user error: The mount was not done via /etc/fstab but via a script executed at server startup in /etc/rc.local

This caused a race condition between docker deamon startup and mount command on server boot, which corrupted the docker storage directory.

Thanks for your help 👍

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Mar 7, 2016

@schinzelh oh, that explains things! Good to hear you got it resolved!

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Mar 7, 2016

Alright, looking at the discussion above, I'm going to close this issue; this issue has been collecting different issues (resulting in the same "error" message). As far as I can see there's no bug involved here.

Feel free to comment here after closing, but please read the discussion above if your error can be the result of one of the things discussed above

@dwu-zingbox

This comment has been minimized.

Copy link

dwu-zingbox commented Apr 29, 2016

My experience show docker doesn't like to run under a root directory, meaning /var/lib/docker or what is specified by -g cannot be a root directory of an EBS volume or a LVM of EBS volume. Giving docker one more level of directory in the same volume through -g solved the problem.

I have Docker version 1.10.2, build c3959b1 and ubuntu 14.04, Linux 3.13.0-79

# df
Filesystem                     1K-blocks    Used Available Use% Mounted on
...
/dev/mapper/data-docker         10190136   23144   9626320   1% /data/docker

DOCKER_OPTS="-g /data/docker/docker" works but DOCKER_OPTS="-g /data/docker" doesn't.

@siliconcow

This comment has been minimized.

Copy link

siliconcow commented Apr 29, 2016

Yep, weird bug but reproduced it on 14.04 w/ docker 1.6.2

@pmontrasio

This comment has been minimized.

Copy link

pmontrasio commented Jun 10, 2016

The solution by @thaJeztah worked for me. I upgraded to linux 3.13.0-88-generic and I had to sudo apt-get install linux-image-extra-3.13.0-88-generic. Started docker and it works.

Maybe the error in the log file

[graphdriver] prior storage driver "aufs" failed: driver not supported
Error starting daemon: error initializing graphdriver: driver not supported

should hint to that package. How about adding a line like this?

You could have to upgrade the linux-image-extra to the version matching your kernel.
Example: sudo apt-get install linux-image-extra-3.13.0-88-generic

Better if with the appropriate kernel version and the appropriate package manager according to the distribution.

@jamesfryer

This comment has been minimized.

Copy link

jamesfryer commented Aug 7, 2016

Same issue, sudo apt-get install linux-image-extra-3.13.0-88-generic fixed it for me.

@jocull

This comment has been minimized.

Copy link

jocull commented Sep 12, 2016

This happens to me all the time. It seems like it happens after running dist-upgrades in Ubuntu, and I always have to go in and run this:

sudo apt-get install linux-headers-$(uname -r) linux-image-extra-$(uname -r)
@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Sep 12, 2016

@jocull correct; if a dist-upgrade upgrades the kernel, chances are you need to install the correct version of the linux-image-extra package for the updated kernel. I think that installing the linux-image-extra-virtual package would help in that situation (and was recently added to the docs, to match what the install script installs; #25614)

@jocull

This comment has been minimized.

Copy link

jocull commented Sep 12, 2016

I will try that and see if it makes a difference! Thank you!

@MouhamedAymen

This comment has been minimized.

Copy link

MouhamedAymen commented Sep 17, 2016

I had the same problem, after updating to 1.12.1 and rebooting.
I solved it by removing /var/lib/docker/overlay (or just renaming it smth else, just in case, you know) and restarting docker. I have no idea why it worked, but glad it did.

@toudidel

This comment has been minimized.

Copy link

toudidel commented Sep 26, 2016

I had the same problem if I wanted to changed my docker images location.
I performed solution from http://stackoverflow.com/a/34731550 and added file /etc/systemd/system/docker.service.d/docker-storage.conf with content:

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --graph="/mnt"

After restarting the system I had the problem. If I delete this file and return to default location (/var/lib/docker) everything is OK.

@mathieufrh

This comment has been minimized.

Copy link

mathieufrh commented Oct 11, 2016

aufs is not supported by latest kernels, so you should use overlayfs instead.
I just edit the /etc/systemd/system/docker.service.d/docker.conf this way:

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --storage-driver=overlay

Then do:

# systemctl stop docker.service
# systemctl daemon-reload
# systemctl start docker.service
@f3l1x

This comment has been minimized.

Copy link

f3l1x commented Nov 29, 2016

I'm running Linux Mint 18 and I've updated from kernel 4.4.0 to 4.8.11. (29.11.2016).

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2016-11-29 13:50:20 CET; 8s ago
     Docs: https://docs.docker.com
 Main PID: 9128 (code=exited, status=1/FAILURE)

Nov 29 13:50:19 XPS systemd[1]: Starting Docker Application Container Engine...
Nov 29 13:50:19 XPS dockerd[9128]: time="2016-11-29T13:50:19.400041011+01:00" level=info msg="libcontainerd: new containerd process, pid: 9135"
Nov 29 13:50:20 XPS dockerd[9128]: time="2016-11-29T13:50:20.406203559+01:00" level=error msg="[graphdriver] prior storage driver \"aufs\" failed: driver not supported"
Nov 29 13:50:20 XPS dockerd[9128]: time="2016-11-29T13:50:20.406360943+01:00" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"
Nov 29 13:50:20 XPS systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 13:50:20 XPS systemd[1]: Failed to start Docker Application Container Engine.
Nov 29 13:50:20 XPS systemd[1]: docker.service: Unit entered failed state.
Nov 29 13:50:20 XPS systemd[1]: docker.service: Failed with result 'exit-code'.

This helps me out.

 sudo rm -rf /var/lib/docker/aufs
 sudo service docker start
@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Nov 29, 2016

Be aware that sudo rm -rf /var/lib/docker/aufs removes all existing images and containers. Basically it's a "factory reset"

@cpuguy83

This comment has been minimized.

Copy link
Contributor

cpuguy83 commented Nov 29, 2016

And the real fix is to re-install the aufs kernel module when you upgrade your kernel.

@jocull

This comment has been minimized.

Copy link

jocull commented Dec 29, 2016

Just to follow up on @thaJeztah's comment, installing the -virtual packages did not correct the issue on my end.

This could be related to using Dokku, or maybe not. I really don't know, but it still breaks every time I run a system upgrade.

@bogste

This comment has been minimized.

Copy link

bogste commented Feb 4, 2017

Thank you @Kelindar for the answer at #14026 (comment)

I just upgraded the Kernel to 4.9 and was getting

[graphdriver] prior storage driver aufs failed: driver not supported

Moved the /var/lib/docker/aufs folder to my $HOME_DIR/Backups/docker (just in case) and was able to run service docker stop/start. Now it starts up as it should!

@Vadorequest

This comment has been minimized.

Copy link

Vadorequest commented Oct 9, 2017

Had the same issue, having an OVH VPS with pre-installed docker, I was getting this annoying error message. Deleting aufs directory did the trick, I was able to start docker service again after that.
I don't even know why I got this bug in the first place, maybe OVH did something with their VPS kernel on the background.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Oct 9, 2017

Yes, if they upgraded the kernel and did not install the required kernel modules after that, then aufs would no longer start. Docker does not automatically pick the next available driver in that situation, because then you won't see your existing images and containers, so it deliberately fails to start to give you an option to resolve the situation.

@tahonaPL

This comment has been minimized.

Copy link

tahonaPL commented May 2, 2018

I had this error after kernel update from 4.4.1 to 4.4.129 (Mint 18.1) . I couldn't revive docker-engine so I installed docker-ce and start working:

apt purge docker-engine
rm -Rf /var/lib/docker

and then:

https://coytar.wordpress.com/2017/03/23/docker-ce-on-linux-mint-18-1/

@prinsherbert

This comment has been minimized.

Copy link

prinsherbert commented May 8, 2018

 # apt-get -y install aufs-tools openjdk-7-jre curl wget git vim make php-pear \
 php5-dev php5-curl python mc gawk ssh grep sudo htop nmon mysql-client php5-cli \
 sqlite3 sysstat sysdig linux-headers-$(uname -r) lxc bsdtar

What do php, python, mysql, sqlite3 and java have to do with this issue?

@prinsherbert

This comment has been minimized.

Copy link

prinsherbert commented May 8, 2018

@Kelindar , thanks, I gave you a thumbs up and down because I think moving is better than deleting.

sudo mv /var/lib/docker/aufs /root/var-lib-docker-aufs-backup`date +%Y-%m-%d`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment