Failed to restart daemon after 24.0.2 -> 24.0.3 upgrade #45898
Labels
area/daemon
area/volumes
kind/bug
Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.
version/24.0
Comments
kotso
added
kind/bug
|
the panic is caused by scripts updating running container network configs. I worked around this:
However, restarting the docker service causes the issue again, so I recommend staying on the previous version. |
|
For already broken systems, we used |
|
thank you @tomsiewert @calvinbui If anyone found solution to apply this update without restarting containers, please share. |
|
|
This happened to me already on 3 systems (have 2 more to offer for debugging before i have no test envs left). Do you need any more info? docker info before it happened: running the upgrade with "restore" with: reboot # worked fine (all containers up and socket working)
# note this is "just" a build system do not perform the next 2 steps if your data has some meaning
# docker ps | awk '($1 != "CONTAINER"){print $1}' | xargs docker stop | xargs docker rm
# docker system prune -af --volumes
sudo apt update && sudo apt upgrade && sudo apt autoremove
# restore containers with build configdocker info from another server (same config, after upgrade + "restore") Also happened on my Ubuntu 23.04 Workstation. The containers seem fine when in the failed upgrade state (tested by checking some service ports that i could find with
|
|
It seems after rolling back the update with On ubuntu/debian the previously installed version can be found with |
|
Debug logs, as well as information on what kinds of volumes you have mounted would be helpful. Are you making use of any Volume plugins or Swarm CSI drivers, or are they all 'local' volumes? |
|
No Swarm CSI drivers, no volume drivers (afaik): (No (gitlab) runner caches where present on my local workstation only local volumes as well) |
|
I think the problem is here; https://github.com/moby/moby/blob/1d9c8619cded4657af1529779c5771127e8ad0e7/daemon/volumes.go#L243-L252C2 That function only looks at the volume if it has Driver set and Volume is And because there's no error, the code continues with the branch below that function call, which includes printing a log message; Lines 18 to 25 in 1d9c861 |
|
Agreed; though I'm not familiar enough with the volume code to know what the correct thing to do is. Naively, I think it's possibly always creating a Edit: no, the driver for local volumes is |
It's empty for bind-mounts etc, which don't need restoring |
|
See Lines 170 to 174 in 1d9c861 |
|
Issue fixed with 24.0.4 update. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
After update on all hosts (20+, all that I'm running) docker daemon is failed.
error is:
I'm running daemon with config:
any chance to manage restart daemon without restarting containers?
Reproduce
systemctl restart docker
Expected behavior
systemd service to start
docker version
docker info
Additional Info
cant provide docker info as service do not start, I have containers running on servers.
If i reboot server (i.e. containers get stopped, docker starts without issues)
The text was updated successfully, but these errors were encountered: