No mount from running container into new container

[mredd]

Description

Since the update to v4.28.0, I can no longer mount a path from one container into another one.

We use a dev container with all the development tools and host compiler installed and from this dev container, we call the cross-compiler in another container. Worked like a charm for a year now, but broke with the update to 4.28.0.

Reproduce

Use this minimal Dockerfile:

FROM debian:bookworm-slim

RUN apt-get update \
 && apt-get install -y curl ca-certificates gnupg

RUN curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg \
 && echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian bookworm stable" | tee /etc/apt/sources.list.d/docker.list

RUN apt-get update \
 && apt-get -y install docker-ce-cli

1. Verify directory contents. In my case:

   > ls .
   Dockerfile

2. Build a test image:

   > docker build -t docker-test .

3. Run the test container and verify the mount into the test container:

   > docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock -v ${PWD}:/home/user/docker-test docker-test
   root@b7ece1b4a828:/# ls /home/user/docker-test
   Dockerfile

4. From within the test container, start another container with mounts:

   root@b7ece1b4a828:/# docker run --rm -v /home/user/docker-test:/home/user/docker-test debian:bookworm-slim ls -l /home/user/docker-test
   total 0

   In the second container, the mount point is empty

This is what the mount points look like in the containers:

# first container
> mount | fgrep /home/user/docker-test
/dev/sde on /home/user/docker-test type ext4 (rw,relatime,discard,errors=remount-ro,data=ordered)

# second container
> docker run --rm -v /home/user/docker-test:/home/user/docker-test debian:bookworm-slim mount | fgrep /home/user/docker-test
overlay on /home/user/docker-test type overlay (rw,relatime,lowerdir=/containers/services/01-docker/lower,upperdir=/containers/services/01-docker/tmp/upper,workdir=/containers/services/01-docker/tmp/work)

Expected behavior

The mount point (/home/user/docker-test in this example) should not be empty in the second container.

docker version

Client:
 Cloud integration: v1.0.35+desktop.11
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:00 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:25 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.6-desktop.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     /usr/local/lib/docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.22
    Path:     /usr/local/lib/docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /usr/local/lib/docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.5.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-scout
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 29
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.133.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 7.76GiB
 Name: docker-desktop
 ID: 5b262c69-62be-4ec9-8fe8-5fae249d4a36
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Diagnostics ID

84E9CF9F-2B60-42D5-97F5-21008A76AD3D/20240320131439

Additional Info

No response

Edit: Fixed a typo in repro step 3.

[mredd]

Might be related to #13969

[mredd]

Help! We are seeing a wave rolling through the company. When people have to restart their PCs, Docker Desktop automatically installs version 4.28.0 and they are doomed. A rollback to earlier versions does not fix the issue. We have tried downgrades all the way down to Docker Desktop version 4.25.0.
Once the issue pops up, we haven't figured out a way to fix it.

[smorgan-brainomix]

I have been hit by exactly this and #13969 today. I spent an embarrassingly long time messing about with permissions etc. before rolling back Docker Desktop because the failure did not coincide with the update to 4.28.0. Possibly it may have been triggered by doing an apt dist-upgrade on my Ubuntu 22.04 WSL environment on Friday (I only started the computer up again this morning), but I cannot be certain.

@mredd I have managed to roll back to 4.27.2 and get my environment working by doing a complete clean of everything Docker related before reinstalling. The steps I took were:

• Uninstall Docker Desktop 4.28.0
• Remove the Docker distros from WSL
  • wsl --unregister docker-desktop
  • wsl --unregister docker-desktop-data
• Delete everything in:
  • C:\Users\myname\AppData\Local\Docker
  • C:\Users\myname\AppData\Roaming\Docker
  • C:\Users\myname\AppData\Roaming\Docker Desktop
• Restart Windows for good luck
• Download and install Docker Desktop 4.27.2 from https://docs.docker.com/desktop/release-notes/#4272

This did lose all my images, configuration etc. but it now works.

[mredd]

Hi @smorgan-brainomix, somehow I missed the notification. Sorry.

After giving different versions a try and seeing that they worked at a few machines while other versions worked at other machines, we finally "solved" this problem by abandoning Docker Desktop completely. Some of us are still using docker CE installed directly in a Ubuntu WSL. All data is within the WSL, no ties to the Windows host system. But most of us have migrated to a native Linux.

We don't use Docker to provide any services. We only use it for per-project development environments (specific compilers and tools per project).