Safe upgrade of docker when using dokku (aufs storage driver)

[dblock]

Description of problem

I have an old Linux distro and trying to update the kernel from 3.13 to 4.x. After that upgrade docker wouldn't start with the error initializing graphdriver: driver not supported error.

I found several posts that say "aufs is not supported anymore, use overlay".

https://itecnote.com/tecnote/docker-prior-storage-driver-aufs-failed-driver-not-supported-error-starting-daemon-error-initializing-graphdriver-driver-not-supported/
https://stackoverflow.com/questions/33357824/prior-storage-driver-aufs-failed-driver-not-supported-error-starting-daemon
https://stackoverflow.com/questions/52617727/cannot-start-docker-daemon-graphdriver-issue

Okay, those tell you to rm -rf /var/lib/docker/aufs and "re-recreate containers". But many people, especially in this post are asking what that means. They talk about some docker data is "temporary", but this doesn't seem right. My data is in mongodb, I use the MongoDB plugin with dokku, mounted on an external volume.

What is the correct upgrade procedure with dokku to change docker aufs to overlay? Is rm -rf /var/lib/docker/aufs safe? What will I have to do after I do this to "recreate containers" with dokku? Do I need to save and restore existing containers?

Steps to reproduce

Upgrade Linux kernel from 3.13 to 4.x, be DOA.

dokku report $APP_NAME

N/A

Additional information

uname -a
Linux dblock-plum 3.13.0-71-generic #114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

dokku --version
dokku version 0.19.12

docker info
Containers: 112
 Running: 18
 Paused: 0
 Stopped: 94
Images: 867
Server Version: 17.05.0-ce
Storage Driver: aufs
 Root Dir: /mnt/dblock_plum_volume_1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 2156
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 apparmor
Kernel Version: 3.13.0-71-generic
Operating System: Ubuntu 16.04.7 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.67GiB
Name: dblock-plum
ID: WYMM:UXKU:SC3Z:U2GV:5Y4V:CPD2:4N4V:LJQJ:K45E:MN3L:QY5W:NMQ7
Docker Root Dir: /mnt/dblock_plum_volume_1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Output of failing commands after running: dokku trace:off

No response

Output of failing commands after running: dokku trace:on

No response

[josegonzalez]

Note: this isn't a bug with Dokku, but a request for support on Docker (kernal/os?) upgrades.

A few things:

• Always take a backup.
• Ubuntu 16.04 is no longer supported, so trying to upgrade just the kernel on the OSS is certainly going to be painful and I don't think it'll work as is.
• It might be easier to just use a fresh server than try to upgrade Ubuntu 16.04 to the latest stable LTS (22.04 as of this writing).
• We have docs on doing that here, which may be helpful for migrating servers (you can test it out :) ).
• The official dokku plugins store data in /var/lib/dokku/services.
• As far as I know, only ephemeral container data is stored in that aufs path.

Hope that helps. Good luck on your upgrade path, whatever that may be.