System broken after upgrade: /var/lib/docker contains several valid graphdrivers

[fxkr]

RancherOS Version: (ros os version)
v1.1.3 --> sudo ros os upgrade --> v1.2.0

Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.)
VM (on qemu/kvm/libvirt) in a test lab (not a production system)

The user docker daemon is not running and /var/log/docker.log from point of current boot reads like this (variations of last two lines repeat a number of times):

time="2018-03-07T21:40:59.176080912Z" level=info msg="Processing signal 'terminated'" 
time="2018-03-07T21:42:12.687316284Z" level=info msg="libcontainerd: new containerd process, pid: 1203" 
time="2018-03-07T21:42:12.698463713Z" level=warning msg="libcontainerd: unknown container 6563f2319bf5f53ad00bf95ad667306da3d545a7a1c5d17c654a5f77d59ba214" 
Error starting daemon: error initializing graphdriver: /var/lib/docker contains several valid graphdrivers: overlay, overlay2; Please cleanup or explicitly choose storage driver (-s <DRIVER>)

[niusmallnan]

Please refer to this: #2194 (comment)

[fxkr]

So according to #2194, it should keep working if /var/lib/docker/overlay2 doesn't exist. I would expect this directory to not exist after the ros upgrade. However it does exist now:

[root@rancher docker]# find /var/lib/docker/overlay2/  
/var/lib/docker/overlay2/
/var/lib/docker/overlay2/l

I suppose I can recover by removing either of them, which works fine for me. Feel free to close this issue if you're not interested in fixing your upgrade path. Otherwise let me know if there's anything I can help you with.

[stefanlasiewski]

@niusmallnan Can you clarify which is the correct way forward?

The Docker docs say that overlay2 is preferred, but my RancherOS instance has both an overlay and an overlay2 directory. overlay2 is mostly empty, except for the presence of the /l file , while 'overlay' contains all the Docker storage assets:

[rancher@server-01 ~]$ sudo find /var/lib/docker/overlay
/var/lib/docker/overlay
/var/lib/docker/overlay/1f112dab0d356e957ecf05142287b7e2246bf892429f86aae9edd895644501d7
...
[rancher@server-01 ~]$ sudo find /var/lib/docker/overlay2
/var/lib/docker/overlay2
/var/lib/docker/overlay2/l
[rancher@server-01 ~]$

[niusmallnan]

@stefanlasiewski From docker 17.06+(maybe 17.03+), Docker can automatically detect storage drivers.
But If you use an earlier docker version(maybe 1.12.6), you must choose one storage driver.

Maybe ros config set rancher.docker.storage_driver overlay, then system-docker restart docker

[stefanlasiewski]

Ah I see. https://github.com/rancher/vagrant is using Docker 1.12.6.