Base Device UUID verification failed. (docker 1.7.1, centos7) docker fails to start

[gaga-github]

I setup the docker-storage (defaults) and ran docker fine, then after a few days the server failed to startup new containers, docker ps hangs, so I tried a restart:

This started after docker ps was hanging and docker needed to be forced off systemctl stop docker

Sep 16 14:23:34 mesos1-02d systemd[1]: Stopping Docker Application Container Engine...
Sep 16 14:23:34 mesos1-02d docker[1298]: time="2015-09-16T10:23:34.271646355-04:00" level=info msg="Processing signal 'terminated'"
Sep 16 14:23:49 mesos1-02d docker[1298]: time="2015-09-16T10:23:49.272446023-04:00" level=error msg="Force shutdown daemon"
Sep 16 14:25:04 mesos1-02d systemd[1]: docker.service stopping timed out. Killing.

startup fails:

Sep 16 14:25:45 mesos1-02d systemd[1]: Starting Docker Application Container Engine...
Sep 16 14:26:34 mesos1-02d systemd[1]: docker.service still around after SIGKILL. Ignoring.
Sep 16 14:26:34 mesos1-02d systemd[1]: Unit docker.service entered failed state.
Sep 16 14:26:34 mesos1-02d systemd[1]: Starting Docker Application Container Engine...
Sep 16 14:26:34 mesos1-02d docker[3822]: time="2015-09-16T14:26:34.344342394Z" level=info msg="Listening for HTTP on unix (/var/run/docker.sock)"
Sep 16 14:26:34 mesos1-02d docker[3822]: time="2015-09-16T14:26:34.438982069Z" level=fatal msg="Error starting daemon: error initializing graphdriver: Base Device UUID verification failed.
Sep 16 14:26:34 mesos1-02d systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Sep 16 14:26:34 mesos1-02d systemd[1]: Failed to start Docker Application Container Engine.
Sep 16 14:26:34 mesos1-02d systemd[1]: Unit docker.service entered failed state.

I see this when I ran docker -d -D

DEBU[0000] Error device setupBaseImage: Base Device UUID verification failed. Possibly using a different thin pool then last invocation:Error running DeviceCreate (ActivateDevice) dm_task_run failed
ERRO[0000] [graphdriver] prior storage driver "devicemapper" failed: Base Device UUID verification failed. Possibly using a different thin pool then last invocation:Error running DeviceCreate (ActivateDevice) dm_task_run failed
FATA[0000] Error starting daemon: error initializing graphdriver: Base Device UUID verification failed. Possibly using a different thin pool then last invocation:Error running DeviceCreate (ActivateDevice) dm_task_run failed

I'm wondering if there is a lock or a 0 byte file causing this.

docker version
Client version: 1.7.1
Client API version: 1.19
Package Version (client): docker-1.7.1-108.el7.centos.x86_64
Go version (client): go1.4.2
Git commit (client): 3043001/1.7.1
OS/Arch (client): linux/amd64
Cannot connect to the Docker daemon. Is 'docker -d' running on this host?

docker info fails (obviously)

uname -a
Linux mesos1-02d 3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[GordonTheTurtle]

Hi!

Please read this important information about creating issues.

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

This is an automated, informational response.

Thank you.

For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

---

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:
uname -a:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

[gaga-github]

Looks like udev changed the UUID, I saw this after I rebooted the vm:

mesos1-02d devicemapper]# pvscan
WARNING: Device for PV ZhIcTl-hdo3-oT7e-3GZ1-ssOJ-2qmv-esXPLl not found or rejected by a filter.
WARNING: Device for PV ZhIcTl-hdo3-oT7e-3GZ1-ssOJ-2qmv-esXPLl not found or rejected by a filter.
PV /dev/sda2 VG centos lvm2 [15.51 GiB / 28.00 MiB free]
PV unknown device VG centos lvm2 [30.00 GiB / 0 free]
PV /dev/sdb1 VG centos lvm2 [30.00 GiB / 30.00 GiB free]
PV /dev/sdc1 VG dockerVG lvm2 [30.00 GiB / 96.00 MiB free]
Total: 4 [105.50 GiB] / in use: 4 [105.50 GiB] / in no VG: 0 [0 ]

So, I just removed the "missing" and sdb1, cleared out /var/lib/docker/ (might not be needed), ran docker-storage-setup, all good.

I guess this is a bug in udev not docker.

[ghost]

As my last resort, I removed docker and re-installed and this problem went away. I have no idea why that worked.

[mrpatrick]

+1 I also had the same issue (CentOS 7.1.1503, Docker 1.7.1) after running a yum update - @tsharpe thanks for the tip. Uninstalling / re-installing also fixed the issue for me as well.

[tobowers]

I just had the same issue on centos 7.1, docker 1.8.2

[mrjoshuap]

I noticed this also happens when you pull images and then try to configure storage after the fact. As @gaga-github mentioned, rm -rf /var/lib/docker/* cleared the error up for me though I did not have to reinstall.

[timbunce]

I just hit this. sudo sh -c 'rm -r /var/lib/docker/*' then sudo systemctl start docker worked for me.
Duplicate of #15721 I presume.

[thaJeztah]

Yup, looks very much like it.

I'll go ahead and close this, please continue the discussion in #15721