[BUG] Named volume fails to mount after reboot

[valentinomariotto]

Description

Hi,
I have a basic EMQX stack as described in this docker-compose.yml file:

networks:
  proxy:
    external: true
  emqx:

volumes:
  emqx-data:
    external: true
  emqx-db:
    external: true

services:
  emqx1:
    image: emqx/emqx:latest
    container_name: emqx
    restart: unless-stopped
    depends_on:
      - db
    volumes:
      - emqx-data:/opt/emqx/data
      - /DOCKER/emqx/log:/opt/emqx/log
    networks:
      - proxy
      - emqx
    ports:
      - ## redacted
    labels:
      # traefik stuff

  db:
    image: mysql:lts
    container_name: emqx-db
    restart: unless-stopped
    volumes:
      - emqx-db:/var/lib/mysql
      - /DOCKER/emqx/config/emqx-users_schema.sql:/docker-entrypoint-initdb.d/setup.sql
    networks:
      - emqx
    ports:
      - #redacted
    env_file: stack.env

I start my stack with docker compose up -d
I need the stack to restart when the host is restarted for scheduled updates.

The problem is that when the host is restarted, the docker daemon restarts all the other containers and stacks, but this one specifically, MOST TIMES does not use its named volumes. Sometimes it works correctly.
New volumes are used, and since they're empty, I lose the EMQX configuration.

If I then run

docker compose down
docker compose up -d

Everything works again.

These are the volumes in question:

# docker volume inspect emqx-data
[
    {
        "CreatedAt": "2024-10-03T10:35:04+02:00",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/emqx-data/_data",
        "Name": "emqx-data",
        "Options": null,
        "Scope": "local"
    }
]
# docker volume inspect emqx-db
[
    {
        "CreatedAt": "2024-10-03T10:35:01+02:00",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/emqx-db/_data",
        "Name": "emqx-db",
        "Options": null,
        "Scope": "local"
    }
]

This is the emqx container running correctly.

# docker inspect 8920fe95927b
      ...
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/DOCKER/emqx/log",
                "Destination": "/opt/emqx/log",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "emqx-data",
                "Source": "/var/lib/docker/volumes/emqx-data/_data",
                "Destination": "/opt/emqx/data",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
...
# docker inspect 8e5d7ae3fc59
...
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/DOCKER/emqx/config/emqx-users_schema.sql",
                "Destination": "/docker-entrypoint-initdb.d/setup.sql",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "emqx-db",
                "Source": "/var/lib/docker/volumes/emqx-db/_data",
                "Destination": "/var/lib/mysql",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
...

I rebooted 3 times. On the first 2 attempts it worked. On the third it failed. I collected this data:

# docker inspect 8920fe95927b
      ...
        "Mounts": [
            {
                "Type": "volume",
                "Name": "emqx-data",
                "Source": "/var/lib/docker/volumes/emqx-data/_data",
                "Destination": "/opt/emqx/data",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/DOCKER/emqx/log",
                "Destination": "/opt/emqx/log",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
...

# docker inspect volume emqx-data
[
    {
        "CreatedAt": "2024-10-03T10:35:04+02:00",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/emqx-data/_data",
        "Name": "emqx-data",
        "Options": null,
        "Scope": "local"
    }
]
Error: No such object: volume

# docker inspect 8e5d7ae3fc59
...
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/DOCKER/emqx/config/emqx-users_schema.sql",
                "Destination": "/docker-entrypoint-initdb.d/setup.sql",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "emqx-db",
                "Source": "/var/lib/docker/volumes/emqx-db/_data",
                "Destination": "/var/lib/mysql",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
...

# docker inspect volume emqx-db
[
    {
        "Id": "8e5d7ae3fc5960080b9cd65f45b5b8df16b7db5ce61990d06de0e46ffec67517",
        "Created": "2025-05-29T08:28:51.483248553Z",
        "Path": "docker-entrypoint.sh",
        "Args": [
            "mysqld"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 1484,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2025-05-29T10:02:06.298292564Z",
            "FinishedAt": "2025-05-29T10:01:39.319825285Z"
        },
        "Image": "sha256:d668aee95b72c7f64adce1bdadcf5d7ba3013105a74cd5fde1d75e0ed33e7b86",
        "ResolvConfPath": "/var/lib/docker/containers/8e5d7ae3fc5960080b9cd65f45b5b8df16b7db5ce61990d06de0e46ffec67517/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/8e5d7ae3fc5960080b9cd65f45b5b8df16b7db5ce61990d06de0e46ffec67517/hostname",
        "HostsPath": "/var/lib/docker/containers/8e5d7ae3fc5960080b9cd65f45b5b8df16b7db5ce61990d06de0e46ffec67517/hosts",
        "LogPath": "/var/lib/docker/containers/8e5d7ae3fc5960080b9cd65f45b5b8df16b7db5ce61990d06de0e46ffec67517/8e5d7ae3fc5960080b9cd65f45b5b8df16b7db5ce61990d06de0e46ffec67517-json.log",
        "Name": "/emqx-db",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/DOCKER/emqx/config/emqx-users_schema.sql:/docker-entrypoint-initdb.d/setup.sql:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "emqx_emqx",
            "PortBindings": {
                "3306/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "13306"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "volume",
                    "Source": "emqx-db",
                    "Target": "/var/lib/mysql",
                    "VolumeOptions": {}
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware",
                "/sys/devices/virtual/powercap"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/573d00cf92f4882f7160b99f78c6e9a3c98fed114fa29b4e7fbb1e7445c1be3c-init/diff:/var/lib/docker/overlay2/fe5217e7967a45e374d977a18ccd02eb639908531516cea0de5a6e4f9bb3fcdc/diff:/var/lib/docker/overlay2/d07aff2cb5dcca8dd066ac9a5cd6f83e243e9c70343503da96d1cd0b907f726d/diff:/var/lib/docker/overlay2/52d163fc3a4363e788c614f5c0342467d897a500f80aef2abc3ce2f867ff2ea2/diff:/var/lib/docker/overlay2/576cf7a6684f288ec13555fa694d40a19d6870f96813b12babd46d662a32b35c/diff:/var/lib/docker/overlay2/74be0202682fee4a13d7a7a98a62119583e11368c930ed5004d652cdf268619f/diff:/var/lib/docker/overlay2/6e9a938dc9fd3b247304b9961be9a30fd9f29265fbaeb0c0ee4432e8b0983537/diff:/var/lib/docker/overlay2/44f77f82275ca9936a6e7af866bb4bae08347c2dc710e8c19d146315dbe0f72e/diff:/var/lib/docker/overlay2/7457bfd8444df26ab2e1f2b029d66a2803cb74e8b533be3883d01c1cf59c8dc1/diff:/var/lib/docker/overlay2/ad59e608516da202fc9d0fc9300c33cde4bfd29cd2d8e98fee45451ac5040612/diff:/var/lib/docker/overlay2/cd50c55546b8d21e9215026c348e6529bf5eec066d5c72c076441784b8b1870d/diff",
                "MergedDir": "/var/lib/docker/overlay2/573d00cf92f4882f7160b99f78c6e9a3c98fed114fa29b4e7fbb1e7445c1be3c/merged",
                "UpperDir": "/var/lib/docker/overlay2/573d00cf92f4882f7160b99f78c6e9a3c98fed114fa29b4e7fbb1e7445c1be3c/diff",
                "WorkDir": "/var/lib/docker/overlay2/573d00cf92f4882f7160b99f78c6e9a3c98fed114fa29b4e7fbb1e7445c1be3c/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/DOCKER/emqx/config/emqx-users_schema.sql",
                "Destination": "/docker-entrypoint-initdb.d/setup.sql",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "emqx-db",
                "Source": "/var/lib/docker/volumes/emqx-db/_data",
                "Destination": "/var/lib/mysql",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "8e5d7ae3fc59",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "3306/tcp": {},
                "33060/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "MYSQL_ROOT_PASSWORD=ONDKaX8uBNsmDY64htsy",
                "MYSQL_DATABASE=emqx-users",
                "MYSQL_USER=emqx",
                "MYSQL_PASSWORD=LsAx6rBjNvme2Uv7rmzV",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GOSU_VERSION=1.17",
                "MYSQL_MAJOR=8.4",
                "MYSQL_VERSION=8.4.2-1.el9",
                "MYSQL_SHELL_VERSION=8.4.1-1.el9"
            ],
            "Cmd": [
                "mysqld"
            ],
            "Image": "mysql:lts",
            "Volumes": {
                "/var/lib/mysql": {}
            },
            "WorkingDir": "/",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "b78c92905d2353efc20faee7289ed4d1d0ee20fe2a26e54e6db6213b01debb71",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:d668aee95b72c7f64adce1bdadcf5d7ba3013105a74cd5fde1d75e0ed33e7b86",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "emqx",
                "com.docker.compose.project.config_files": "/DOCKER/emqx/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/DOCKER/emqx",
                "com.docker.compose.service": "db",
                "com.docker.compose.version": "2.29.1"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "426475a0dff85de906cd09ea9cdb5dfe71b4102d020b592003d38c896a831d5f",
            "SandboxKey": "/var/run/docker/netns/426475a0dff8",
            "Ports": {
                "3306/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "13306"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "13306"
                    }
                ],
                "33060/tcp": null
            },
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "emqx_emqx": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "emqx-db",
                        "db"
                    ],
                    "MacAddress": "02:42:ac:13:00:03",
                    "DriverOpts": null,
                    "NetworkID": "e702abf9d80f387490bb880b3183d6b30342e95aa1cf8324e0ebe25980682d49",
                    "EndpointID": "9238ce7a442b80abf8ffe5ffe5a4ef6f611b90cf51d94bdfd668c23d9b5c08f8",
                    "Gateway": "172.19.0.1",
                    "IPAddress": "172.19.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DNSNames": [
                        "emqx-db",
                        "db",
                        "8e5d7ae3fc59"
                    ]
                }
            }
        }
    }
]
Error: No such object: volume

It looks like docker is confusing the volume with the container!

Steps To Reproduce

Unsure. This is what I did:

1. create named volumes
2. create docker compose stack
3. start with docker compose up -d
4. reboot the host
5. Half the times, it will fail to load the volumes.

Compose Version

Docker Compose version v2.29.1

bash: docker-compose: command not found

Docker Environment

Client: Docker Engine - Community
 Version:    27.1.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 5
  Running: 5
  Paused: 0
  Stopped: 0
 Images: 9
 Server Version: 27.1.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-37-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.887GiB
 Name: srv-docker-mqtt
 ID: 5c14a396-9ba5-4241-9c82-670ffb1c2b1a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

I'm renaming the emqx-db container.

[valentinomariotto]

I think this was indeed a race condition given by the volume and the container having the same name.

Docker doesn't seem to be checking the object type.

[valentinomariotto]

Renaming the volume fixes the volume/container ambiguity, but does not fix the reboot problem. Upon reboot, EMQX still failed to use the saved configuration, but now the information returned by inspect is correct:

rge@srv-docker-mqtt:~$ sudo docker volume inspect emqx-db
[
    {
        "CreatedAt": "2024-10-03T10:35:01+02:00",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/emqx-db/_data",
        "Name": "emqx-db",
        "Options": null,
        "Scope": "local"
    }
]

[ndeloof]

Docker Compose is not involved on system reboot, it just asks engine to create containers by API, then Docker engine is in charge. According to your description, container is well set with mount "Source": "/var/lib/docker/volumes/emqx-data/_data", "Destination": "/opt/emqx/data" even when things fail, so I can't explain why container would not have access to volume content, and how a docker compose down && docker compose up (which basically stop and starts container) fixed the issue.

Can you please check if a simple docker stop emqx && docker start emqx also "fix" this issue after a volume mount failure on reboot ?

[ndeloof]

Such an issue should be reported on github.com/moby/moby