docker-storage-setup.service fails on all nodes.

[vhosakot]

I see the docker-storage-setup.service service failed on all the nodes. Looks like a disk space issue? Restarting docker-storage-setup.service on the nodes did not help.

kube{2,3,4} show There is not enough free space in volume group VolGroup00 to create data volume of size MIN_DATA_SIZE=2G. I have not dug into or googled this error yet. Don't know if it affects k8s. I'll submit a PR if I find the solution. I'm using CentOS+libvirt.

# vagrant ssh kube1
[vagrant@kube1 ~]$ sudo systemctl --failed
  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
● docker-storage-setup.service loaded failed failed Docker Storage Setup

[vagrant@kube1 ~]$ sudo systemctl status -l docker-storage-setup.service
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2017-02-13 02:55:18 UTC; 1 day 1h ago
 Main PID: 24417 (code=exited, status=1/FAILURE)
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

# vagrant ssh kube2
[vagrant@kube2 ~]$ sudo systemctl --failed
  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
● docker-storage-setup.service loaded failed failed Docker Storage Setup

[vagrant@kube2 ~]$ sudo systemctl status -l docker-storage-setup.service
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2017-02-13 02:55:19 UTC; 1 day 1h ago
 Main PID: 24416 (code=exited, status=1/FAILURE)
Feb 13 02:55:17 kube2 systemd[1]: Starting Docker Storage Setup...
Feb 13 02:55:19 kube2 docker-storage-setup[24416]: ERROR: There is not enough free space in volume group VolGroup00 to create data volume of size MIN_DATA_SIZE=2G.
Feb 13 02:55:19 kube2 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Feb 13 02:55:19 kube2 systemd[1]: Failed to start Docker Storage Setup.
Feb 13 02:55:19 kube2 systemd[1]: Unit docker-storage-setup.service entered failed state.
Feb 13 02:55:19 kube2 systemd[1]: docker-storage-setup.service failed.

# vagrant ssh kube3
[vagrant@kube3 ~]$ sudo systemctl --failed
  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
● docker-storage-setup.service loaded failed failed Docker Storage Setup

[vagrant@kube3 ~]$ sudo systemctl status -l docker-storage-setup.service
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2017-02-13 02:55:17 UTC; 1 day 1h ago
 Main PID: 24411 (code=exited, status=1/FAILURE)
Feb 13 02:55:16 kube3 systemd[1]: Starting Docker Storage Setup...
Feb 13 02:55:17 kube3 docker-storage-setup[24411]: ERROR: There is not enough free space in volume group VolGroup00 to create data volume of size MIN_DATA_SIZE=2G.
Feb 13 02:55:17 kube3 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Feb 13 02:55:17 kube3 systemd[1]: Failed to start Docker Storage Setup.
Feb 13 02:55:17 kube3 systemd[1]: Unit docker-storage-setup.service entered failed state.
Feb 13 02:55:17 kube3 systemd[1]: docker-storage-setup.service failed.

# vagrant ssh kube4
[vagrant@kube4 ~]$ sudo systemctl --failed
  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
● docker-storage-setup.service loaded failed failed Docker Storage Setup

[vagrant@kube4 ~]$ sudo sudo systemctl status -l docker-storage-setup.service
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2017-02-13 02:55:18 UTC; 1 day 1h ago
 Main PID: 24411 (code=exited, status=1/FAILURE)
Feb 13 02:55:17 kube4 systemd[1]: Starting Docker Storage Setup...
Feb 13 02:55:18 kube4 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Feb 13 02:55:18 kube4 docker-storage-setup[24411]: ERROR: There is not enough free space in volume group VolGroup00 to create data volume of size MIN_DATA_SIZE=2G.
Feb 13 02:55:18 kube4 systemd[1]: Failed to start Docker Storage Setup.
Feb 13 02:55:18 kube4 systemd[1]: Unit docker-storage-setup.service entered failed state.
Feb 13 02:55:18 kube4 systemd[1]: docker-storage-setup.service failed.

Restarting docker-storage-setup.service on the nodes did not help.

# vagrant ssh kube1
[vagrant@kube1 ~]$ sudo systemctl restart docker-storage-setup.service
Job for docker-storage-setup.service failed because the control process exited with error code. See "systemctl status docker-storage-setup.service" and "journalctl -xe" for details.

[vagrant@kube1 ~]$ sudo systemctl status -l docker-storage-setup.service
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2017-02-14 04:12:23 UTC; 4s ago
  Process: 23430 ExecStart=/usr/bin/docker-storage-setup (code=exited, status=1/FAILURE)
 Main PID: 23430 (code=exited, status=1/FAILURE)
Feb 14 04:12:20 kube1 systemd[1]: Starting Docker Storage Setup...
Feb 14 04:12:23 kube1 docker-storage-setup[23430]: ERROR: Docker has been previously configured for use with devicemapper graph driver. Not creating a new thin pool as existing docker metadata will fail to work with it. Manual cleanup is required before this will succeed.
Feb 14 04:12:23 kube1 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Feb 14 04:12:23 kube1 docker-storage-setup[23430]: INFO: Docker state can be reset by stopping docker and by removing /var/lib/docker directory. This will destroy existing docker images and containers and all the docker metadata.
Feb 14 04:12:23 kube1 systemd[1]: Failed to start Docker Storage Setup.
Feb 14 04:12:23 kube1 systemd[1]: Unit docker-storage-setup.service entered failed state.
Feb 14 04:12:23 kube1 systemd[1]: docker-storage-setup.service failed.

# vagrant ssh kube2
[vagrant@kube2 ~]$ sudo systemctl restart docker-storage-setup.service
Job for docker-storage-setup.service failed because the control process exited with error code. See "systemctl status docker-storage-setup.service" and "journalctl -xe" for details.

[vagrant@kube2 ~]$ sudo systemctl status -l docker-storage-setup.service
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2017-02-14 04:13:05 UTC; 2s ago
  Process: 13159 ExecStart=/usr/bin/docker-storage-setup (code=exited, status=1/FAILURE)
 Main PID: 13159 (code=exited, status=1/FAILURE)
Feb 14 04:13:03 kube2 systemd[1]: Starting Docker Storage Setup...
Feb 14 04:13:05 kube2 docker-storage-setup[13159]: ERROR: Docker has been previously configured for use with devicemapper graph driver. Not creating a new thin pool as existing docker metadata will fail to work with it. Manual cleanup is required before this will succeed.
Feb 14 04:13:05 kube2 docker-storage-setup[13159]: INFO: Docker state can be reset by stopping docker and by removing /var/lib/docker directory. This will destroy existing docker images and containers and all the docker metadata.
Feb 14 04:13:05 kube2 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Feb 14 04:13:05 kube2 systemd[1]: Failed to start Docker Storage Setup.
Feb 14 04:13:05 kube2 systemd[1]: Unit docker-storage-setup.service entered failed state.
Feb 14 04:13:05 kube2 systemd[1]: docker-storage-setup.service failed.

# vagrant ssh kube3
[vagrant@kube3 ~]$ sudo systemctl restart docker-storage-setup.service
Job for docker-storage-setup.service failed because the control process exited with error code. See "systemctl status docker-storage-setup.service" and "journalctl -xe" for details.

[vagrant@kube3 ~]$ sudo systemctl status -l docker-storage-setup.service
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2017-02-14 04:13:23 UTC; 3s ago
  Process: 24423 ExecStart=/usr/bin/docker-storage-setup (code=exited, status=1/FAILURE)
 Main PID: 24423 (code=exited, status=1/FAILURE)
Feb 14 04:13:21 kube3 systemd[1]: Starting Docker Storage Setup...
Feb 14 04:13:24 kube3 docker-storage-setup[24423]: ERROR: Docker has been previously configured for use with devicemapper graph driver. Not creating a new thin pool as existing docker metadata will fail to work with it. Manual cleanup is required before this will succeed.
Feb 14 04:13:24 kube3 docker-storage-setup[24423]: INFO: Docker state can be reset by stopping docker and by removing /var/lib/docker directory. This will destroy existing docker images and containers and all the docker metadata.
Feb 14 04:13:23 kube3 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Feb 14 04:13:23 kube3 systemd[1]: Failed to start Docker Storage Setup.
Feb 14 04:13:23 kube3 systemd[1]: Unit docker-storage-setup.service entered failed state.
Feb 14 04:13:23 kube3 systemd[1]: docker-storage-setup.service failed.

# vagrant ssh kube4
[vagrant@kube4 ~]$ sudo systemctl restart docker-storage-setup.service
Job for docker-storage-setup.service failed because the control process exited with error code. See "systemctl status docker-storage-setup.service" and "journalctl -xe" for details.

[vagrant@kube4 ~]$ sudo sudo systemctl status -l docker-storage-setup.service
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2017-02-14 04:13:41 UTC; 2s ago
  Process: 15536 ExecStart=/usr/bin/docker-storage-setup (code=exited, status=1/FAILURE)
 Main PID: 15536 (code=exited, status=1/FAILURE)
Feb 14 04:13:39 kube4 systemd[1]: Starting Docker Storage Setup...
Feb 14 04:13:41 kube4 docker-storage-setup[15536]: ERROR: Docker has been previously configured for use with devicemapper graph driver. Not creating a new thin pool as existing docker metadata will fail to work with it. Manual cleanup is required before this will succeed.
Feb 14 04:13:41 kube4 docker-storage-setup[15536]: INFO: Docker state can be reset by stopping docker and by removing /var/lib/docker directory. This will destroy existing docker images and containers and all the docker metadata.
Feb 14 04:13:41 kube4 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Feb 14 04:13:41 kube4 systemd[1]: Failed to start Docker Storage Setup.
Feb 14 04:13:41 kube4 systemd[1]: Unit docker-storage-setup.service entered failed state.
Feb 14 04:13:41 kube4 systemd[1]: docker-storage-setup.service failed.

[v1k0d3n]

@vhosakot there's been a PR that was submitted that allows users to increase the disk size: https://github.com/att-comdev/halcyon-vagrant-kubernetes/blob/master/config.rb#L4

would that help?

[vhosakot]

@v1k0d3n Borne Mace (bmace) confirmed in the #openstack-kolla IRC channel that this issue is not seen.

Turns out my vagrant host has low disk space, and I'm surprised why vagrant does not complain with an error when the host has disk space less than the sum of the disk spaces of all the VMs booted by vagrant. For example, if vagrant boots four VMs each with 10 GB disk and the vagrant host does not have at least >=40 GB disk space, I'd expect vagrant to complain with an error like:

Not enough disk space on host to boot four VMs each with 10 GB disk.

Once I collect all the info, I might open a bug against vagrant to check the host's disk space before booting the VMs. I will also increase the disk space on my vagrant host.

This issue can be closed.

[v1k0d3n]

awesome. thanks @vhosakot!