Possible bug docker-in-docker? could not connect to docker daemon

[mandrasch]

Select Topic Area

Bug

Body

Hey,

I'm experimenting with docker-in-docker and codespaces.

The .devcontainer.json:

{
  "image": "mcr.microsoft.com/devcontainers/universal:2",
  "features": {
    "ghcr.io/devcontainers/features/docker-in-docker:2": {},
    "ghcr.io/ddev/ddev/install-ddev:latest": {}
  },

1 times out of 15 I get the following error, it seems random:

2023-08-16 19:07:58.801Z: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Only a full rebuild solves it or starting a completely new codespace.

Update 16.08.2023: Full error log here Full output: https://gist.github.com/mandrasch/3b001bccdb8ab9ab75ebd7ddc6973727

time="2023-08-16T19:07:55.475998600Z" level=error msg="containerd did not exit successfully" error="exit status 1" module=libcontainerd
failed to start daemon: error while opening volume store metadata database: timeout
x\"..." type=io.containerd.runtime.v1

Update 16.08.2023: Issue found, missed it first time devcontainers/features#634

Repo: https://github.com/mandrasch/ddev-craftcms-vite

Thanks in advance for any hints, much appreciated!

Best regards,
Matthias

[nathan-hello]

in all of my .devcontainer.json files, i add a mount for the .sock file in the host machine to the one in the docker. This way any docker commands i run is actually being ran on the host machine. this also means any devcontainer + docker shenanigans i run into, it's because of the host machine and not the devcontainer, which is easier to control (though kind of not with codespaces but you get what i mean). anyways, here's the mount.

    "mounts": [
        "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock"
    ],

[rfay]

Hmm, one thing that you (and DDEV) should know is that docker is provided with no fiddling in Codespaces. With no devcontainer.json at all, you get docker running by default when opening any repo.

[mandrasch]

Thanks very much, @rfay!

Since other devs experience it in the issue below as well, at least we can confirm it has nothing to do with adding another feature like ddev.

devcontainers/features#634

Unfortunately I just found the issue now, did not find it when I first searched through the issues for github.com/devcontainers/features/ ... Must have missed it.

[rfay]

Congrats! Looks like that will be in next release, and other people have the problem.

[mandrasch]

👍 The fix was published in docker-in-docker 2.3.0, which was already used in my broken example from today. But hopefully there will be another patch in future.

[rfay]

It does claim that, but I guess you should chime up in the issue and explain that it's not fixed :(

[othman922]

[samruddhikhandale]

Hi 👋

Responded on the devcontainers/features#634 issue with more details.

Hi @mandrasch 👋

In your dev container, docker is added by the universal image. See https://github.com/devcontainers/images/blob/main/src/universal/.devcontainer/devcontainer.json#L64-L66.

Hence, adding the docker-in-docker Feature has no impact as the docker-init script is old. See https://gist.github.com/mandrasch/3b001bccdb8ab9ab75ebd7ddc6973727#file-creation-log-txt-L168. (I think this behavior should be updated.)

The prod image was built with Feature version 2.2.1 (released on Aug 3rd) which does not contain the retry logic. I'll work on releasing a new universal image which would be built with v2.3.1.

Let me know if that makes sense.

I'll work on releasing a new universal image which would be built with v2.3.1.

Opened devcontainers/images#705. In the meanwhile, @mandrasch can you use the dev image? (mcr.microsoft.com/devcontainers/universal:dev). Let me know if that helps!

Also, can you remove the docker-in-docker Feature from your dev container as that one is unnecessary (given that the universal image already installs it)?

[mandrasch]

Just for documentation:

For all people interested in this, I also responded in the issue: devcontainers/features#634 (comment)