docker.service fails to start after upgrade from 24.0.2 to 24.0.3

[CamZie]

Hello,

We have encountered issues with our docker installation after the upgrade from 24.0.2 to 24.0.3 on the following systems:

• Debian 10
• Debian 11
• Ubuntu 20.04
• Ubuntu 22.04

According to the logs, it looks like the upgrade failed but we could not find the reason why:

~$ dpkg -l | grep docker-ce
iF  docker-ce                            5:24.0.3-1~ubuntu.20.04~focal     amd64        Docker: the open-source application container engine

When running the upgrade command it fails when starting the docker service:

$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up docker-ce (5:24.0.3-1~ubuntu.20.04~focal) ...
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "restart" failed.
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/docker.service.d
             └─service-overrides.conf
     Active: activating (auto-restart) (Result: exit-code) since Fri 2023-07-07 10:12:43 CEST; 23ms ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
    Process: 3601344 ExecStart=/usr/bin/dockerd $OPTIONS $DOCKER_STORAGE_OPTIONS (code=exited, status=2)
   Main PID: 3601344 (code=exited, status=2)

Jul 07 10:12:43 systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 07 10:12:43 systemd[1]: Failed to start Docker Application Container Engine.
dpkg: error processing package docker-ce (--configure):
 installed docker-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)

Tried to uninstall docker-ce and install it but that also did not help. For now we downgraded the servers that are affected to 24.0.2 and it works fine.

[CamZie]

I think we have found the cause of the issue. It was the following /etc/docker/daemon.json parameter that is causing this:

{ 
  "live-restore": true
}

We have removed this option and rebooted the server and after that we could successfully run systemctl restart docker without any issues.

We tested the other way around by adding this again in the daemon.json, rebooted the server and then ran the restart command but it fails:

$ sudo systemctl restart docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

According to the logs it is restarting too quickly

Jul  7 11:24:46 host systemd[1]: docker.service: Start request repeated too quickly.
Jul  7 11:24:46 host systemd[1]: docker.service: Failed with result 'exit-code'.
Jul  7 11:24:46 host systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.

Is there a way we can use this parameter without causing conflicts with the systemctl script? Downgrading the version to 24.0.2 works but we would like to avoid to use older versions of Docker.

[thaJeztah]

Thanks for reporting; this looks to be a duplicate of;

• Failed to restart daemon after 24.0.2 -> 24.0.3 upgrade moby/moby#45898
• dockerd segfaults when patching 24.0.2 to 24.0.3 with live-restore enabled moby/moby#45907

This was a regression in 24.0.3, and a fix was implemented in;

• daemon: daemon.prepareMountPoints(): fix panic if mount is not a volume moby/moby#45902
• backported to the 24.0 branch in; [24.0 backport] daemon: daemon.prepareMountPoints(): fix panic if mount is not a volume moby/moby#45903

This should be fixed in 24.0.4; https://github.com/moby/moby/releases/tag/v24.0.4