Docker service is failing to start with latest build of the vagrant box

[LalatenduMohanty]

Read #68 (comment)
Scratch build: https://cbs.centos.org/koji/taskinfo?taskID=18983

CC @navidshaikh

sudo systemctl status docker
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
  Drop-In: /usr/lib/systemd/system/docker.service.d
           └─flannel.conf
   Active: failed (Result: exit-code) since Wed 2015-08-19 11:31:22 EDT; 5min ago
     Docs: http://docs.docker.com
  Process: 814 ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE)
 Main PID: 814 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/docker.service

Aug 19 11:31:22 localhost.localdomain docker[814]: time="2015-08-19T11:31:22.503471246-04:00" level=info msg="Listening for HTTP on unix (/var/run/docker.sock)"
Aug 19 11:31:22 localhost.localdomain docker[814]: time="2015-08-19T11:31:22.544865763-04:00" level=fatal msg="Error starting daemon: error initializing graphdriver: Unknown option dm.fs"
Aug 19 11:31:22 localhost.localdomain systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Aug 19 11:31:22 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine.
Aug 19 11:31:22 localhost.localdomain systemd[1]: Unit docker.service entered failed state.

[jasonbrooks]

The devicemapper arguments in /etc/sysconfig/docker-storage changed somewhat. It should look like:

DOCKER_STORAGE_OPTIONS=--storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/vg001-docker--pool --storage-opt dm.use_deferred_removal=true

More info: https://github.com/docker/docker/blob/master/man/docker.1.md

[LalatenduMohanty]

@jasonbrooks Thanks for the pointer. It looks like docker cmd line does not need data and metadata logical volume path and just need path to docker--pool. So do I need to create a lv for docker-pool ?

Added later:
@jasonbrooks Never mind I got the answer from https://github.com/docker/docker/blob/master/man/docker.1.md#dmmetadatadev

[navidshaikh]

@LalatenduMohanty : As I pointed in #68 (comment) ; there are multiple entries of variable DOCKER_STORAGE_OPTIONS in /etc/sysconfig/docker-storage, after commenting out unnecessary one, the docker service starts.
I think we need to figure out how /etc/sysconfig/docker-storage is populated with multiple entries of DOCKER_STORAGE_OPTIONS variable and fix it. Here at line no: 59 in our kickstart file, DOCKER_STORAGE_OPTIONS variable is updated.

[navidshaikh]

@jasonbrooks : I tried updating the the variable however the service is failing with

msg="Error starting daemon: error initializing graphdriver: Non existing device vg001-docker--pool"

I think, I am missing some preliminary step(s) before updating the /etc/sysconfig/docker?
Is device vg001-docker-pool supposed to be created before hand or the service / devicemapper should create it?

[vagrant@localhost ~]$ rpm -q docker
docker-1.7.1-108.el7.centos.x86_64

[vagrant@localhost ~]$ cat /etc/sysconfig/docker-storage

# DOCKER_STORAGE_OPTIONS = --storage-opt dm.metadatadev=/dev/mylogvol/my-docker-metadata --storage-opt dm.datadev=/dev/mylogvol/my-docker-data

DOCKER_STORAGE_OPTIONS=--storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/vg001-docker--pool --storage-opt dm.use_deferred_removal=true

[vagrant@localhost ~]$ sudo systemctl restart docker ; sudo systemctl status docker -l
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
  Drop-In: /usr/lib/systemd/system/docker.service.d
           └─flannel.conf
   Active: failed (Result: exit-code) since Thu 2015-08-20 02:30:06 EDT; 12ms ago
     Docs: http://docs.docker.com
  Process: 19140 ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE)
 Main PID: 19140 (code=exited, status=1/FAILURE)

Aug 20 02:30:06 localhost.localdomain systemd[1]: Starting Docker Application Container Engine...
Aug 20 02:30:06 localhost.localdomain docker[19140]: time="2015-08-20T02:30:06.767464326-04:00" level=fatal msg="Error starting daemon: error initializing graphdriver: Non existing device vg001-docker--pool"
Aug 20 02:30:06 localhost.localdomain systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Aug 20 02:30:06 localhost.localdomain systemd[1]: Failed to start Docker Application Container Engine.
Aug 20 02:30:06 localhost.localdomain systemd[1]: Unit docker.service entered failed state.

[vagrant@localhost ~]$ ls /dev/mapper/
control  vg001-docker--data  vg001-docker--meta  vg001-root

[LalatenduMohanty]

@navidshaikh Yes, I have seen your comment in #68 (comment) . However @jasonbrooks also pointed some more stuff in #69 (comment) . Looks like docker storage options has changed. So trying to understand it.

[LalatenduMohanty]

@navidshaikh So it seems we need a separate thinpool for the docker pool.

[LalatenduMohanty]

from man docker-storage-setup -> lvm2 version should be same or higher than lvm2-2.02.112 for lvm thin pool functionality to work properly.

The current versions in C7 are more then lvm2-2.02.112

[vagrant@localhost ~]$ rpm -qa | grep lvm
lvm2-libs-2.02.115-3.el7_1.1.x86_64
lvm2-2.02.115-3.el7_1.1.x86_64

[jasonbrooks]

# echo "DOCKER_STORAGE_OPTIONS=--storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/vg001-docker--pool --storage-opt dm.use_deferred_removal=true" > /etc/sysconfig/docker-storage
# lvremove /dev/vg001/docker-meta
# lvremove /dev/vg001/docker-data
# docker-storage-setup
# systemctl start docker
# docker info