can't create unix socket /var/run/docker.sock: is a directory #30348
Comments
|
Hm, what happened here is that when bind-mounting files or directories from the host, the host path is automatically created by docker if it doesn't exist (we tried deprecating that behavior, but there are many people relying on this; see #21666, and issues linked from that). If the path ( I'm not sure though how the container / bind-mount could be created before the daemon was "up" (and the socket created). ping @cpuguy83 any idea? |
|
It's a race condition. |
|
We faced the same: we tried to rm -rf /var/run/docker.sock, but while starting daemon, same thing happened again. |
|
I have also observed this on my setup.
|
|
Not creating the For this case (see my earlier comment #30348 (comment)), would bind-mounting |
|
I am also hitting this while upgrading from 1.12 to 17.03. My work around, added some code to docker's upstart init to remove the dir: |
|
Hi. It's been a while since I posted this issue. In the meantime, we decided to use a systemd.socket to manage a |
|
Disregard my upstart init mod. After talking more about this with @thaJeztah, bad things can happen with that approach. Instead, I am going to make sure any container that needs access to the docker socket, bind-mount /var/run instead of /var/run/docker.sock. |
|
The script is probably "ok" to fix the daemon not starting, but won't prevent containers from running into issues. I haven't tried @stuszynski's approach, perhaps it's worth a try if you're running systemd. |
|
I confirm the same problem encountered with both |
|
I ran into the same issue this morning. We do have containers with All I did was Manually removing the directory was the workaround for me. |
Don't create source directory while the daemon is being shutdown, fix moby#30348
|
Note that the pull-request that closed this issue (#33330) only prevents one possible reason a directory is created when bind-mounting |
|
I suspect most, myself included, are seeing the race condition on startup with containers that have a restart policy, rather than on shutdown as PR #33330 solves. Is there another issue to follow, should this one be reopened, or should a new issue be created for the other scenarios? |
|
There's probably other issues mentioning this issue, but I can reopen for now. The remaining problem is that containers are started before the API is up (thus, the socket not being there), in which case starting the container creates the directory. Possibly @piontec's proposal (#30348 (comment)) should be investigated, if someone is interested to look into that possibility |
… #30348 If a container mount the socket the daemon is listening on into container while the daemon is being shutdown, the socket will not exist on the host, then daemon will assume it's a directory and create it on the host, this will cause the daemon can't start next time. fix issue moby/moby#30348 To reproduce this issue, you can add following code ``` --- a/daemon/oci_linux.go +++ b/daemon/oci_linux.go @@ -8,6 +8,7 @@ import ( "sort" "strconv" "strings" + "time" "github.com/Sirupsen/logrus" "github.com/docker/docker/container" @@ -666,7 +667,8 @@ func (daemon *Daemon) createSpec(c *container.Container) (*libcontainerd.Spec, e if err := daemon.setupIpcDirs(c); err != nil { return nil, err } - + fmt.Printf("===please stop the daemon===\n") + time.Sleep(time.Second * 2) ms, err := daemon.setupMounts(c) if err != nil { return nil, err ``` step1 run a container which has `--restart always` and `-v /var/run/docker.sock:/sock` ``` $ docker run -ti --restart always -v /var/run/docker.sock:/sock busybox / # ``` step2 exit the the container ``` / # exit ``` and kill the daemon when you see ``` ===please stop the daemon=== ``` in the daemon log The daemon can't restart again and fail with `can't create unix socket /var/run/docker.sock: is a directory`. Signed-off-by: Lei Jitang <leijitang@huawei.com> (cherry picked from commit 7318eba) Signed-off-by: Eli Uriegas <eli.uriegas@docker.com> Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
|
Is there any workaround for this that I can use in the meantime ? |
|
I saw the same issue and rebooting the Linux helped for me. removing the |
… #30348 If a container mount the socket the daemon is listening on into container while the daemon is being shutdown, the socket will not exist on the host, then daemon will assume it's a directory and create it on the host, this will cause the daemon can't start next time. fix issue moby/moby#30348 To reproduce this issue, you can add following code ``` --- a/daemon/oci_linux.go +++ b/daemon/oci_linux.go @@ -8,6 +8,7 @@ import ( "sort" "strconv" "strings" + "time" "github.com/Sirupsen/logrus" "github.com/docker/docker/container" @@ -666,7 +667,8 @@ func (daemon *Daemon) createSpec(c *container.Container) (*libcontainerd.Spec, e if err := daemon.setupIpcDirs(c); err != nil { return nil, err } - + fmt.Printf("===please stop the daemon===\n") + time.Sleep(time.Second * 2) ms, err := daemon.setupMounts(c) if err != nil { return nil, err ``` step1 run a container which has `--restart always` and `-v /var/run/docker.sock:/sock` ``` $ docker run -ti --restart always -v /var/run/docker.sock:/sock busybox / # ``` step2 exit the the container ``` / # exit ``` and kill the daemon when you see ``` ===please stop the daemon=== ``` in the daemon log The daemon can't restart again and fail with `can't create unix socket /var/run/docker.sock: is a directory`. Signed-off-by: Lei Jitang <leijitang@huawei.com> Upstream-commit: 7318eba Component: engine
|
had the same issue on |
* Update docs/submodules/oom/offline-installer.git from branch 'master'
to 05b9001fa01b3f1076a5a21f063ca40421a66333
- Merge "Improving docker restart handler"
- Improving docker restart handler
There is a bug in docker which leads to not properly
shutdown service preventing subsequent startup.
moby/moby#30348
This commit is preventing this problem to appear.
Change-Id: I29505610bd9954af01d73264e5414fdb2b9ac99d
Issue-ID: OOM-1735
Signed-off-by: Michal Ptacek <m.ptacek@partner.samsung.com>
|
I am still facing this problem, and I am surprised there seems to be no solution yet (or I could not find it). This always catches me cold after an "almost finish, quickly to a reboot"maintenance session... |
|
@torwag I felt like this has been fixed. I haven't seen it for probably 18 months. Are you running a current version of Docker? |
|
@stefanlasiewski yes I do and I still face the problem. @kylefransham your workaround seems to work, but it comes with some drawback, as many of those containers, which need the docker.sock, have the path also added somewhere in their own config. Thus, it requires to change those configs too. |
|
We face the same issue these days with docker-ce 18.09.6. The thing is we are offering a kubernetes cluster with containerd as runtime, rather than docker, and before we setup docker on the host, users deploy pods mounting hostPath volume of /var/run/docker.sock in purpose of docker-in-container cases. We can manually remove the directory on the host and bring up docker daemon. However, I wonder there is still risk retriggering such issue. |
|
@jiuchen1986 Absolutely the issue can be re-triggered. |
ghost
mentioned this issue
… #30348 If a container mount the socket the daemon is listening on into container while the daemon is being shutdown, the socket will not exist on the host, then daemon will assume it's a directory and create it on the host, this will cause the daemon can't start next time. fix issue moby/moby#30348 To reproduce this issue, you can add following code ``` --- a/daemon/oci_linux.go +++ b/daemon/oci_linux.go @@ -8,6 +8,7 @@ import ( "sort" "strconv" "strings" + "time" "github.com/Sirupsen/logrus" "github.com/docker/docker/container" @@ -666,7 +667,8 @@ func (daemon *Daemon) createSpec(c *container.Container) (*libcontainerd.Spec, e if err := daemon.setupIpcDirs(c); err != nil { return nil, err } - + fmt.Printf("===please stop the daemon===\n") + time.Sleep(time.Second * 2) ms, err := daemon.setupMounts(c) if err != nil { return nil, err ``` step1 run a container which has `--restart always` and `-v /var/run/docker.sock:/sock` ``` $ docker run -ti --restart always -v /var/run/docker.sock:/sock busybox / # ``` step2 exit the the container ``` / # exit ``` and kill the daemon when you see ``` ===please stop the daemon=== ``` in the daemon log The daemon can't restart again and fail with `can't create unix socket /var/run/docker.sock: is a directory`. Signed-off-by: Lei Jitang <leijitang@huawei.com> (cherry picked from commit 7318eba) Signed-off-by: Eli Uriegas <eli.uriegas@docker.com> Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
… #30348 If a container mount the socket the daemon is listening on into container while the daemon is being shutdown, the socket will not exist on the host, then daemon will assume it's a directory and create it on the host, this will cause the daemon can't start next time. fix issue moby/moby#30348 To reproduce this issue, you can add following code ``` --- a/daemon/oci_linux.go +++ b/daemon/oci_linux.go @@ -8,6 +8,7 @@ import ( "sort" "strconv" "strings" + "time" "github.com/Sirupsen/logrus" "github.com/docker/docker/container" @@ -666,7 +667,8 @@ func (daemon *Daemon) createSpec(c *container.Container) (*libcontainerd.Spec, e if err := daemon.setupIpcDirs(c); err != nil { return nil, err } - + fmt.Printf("===please stop the daemon===\n") + time.Sleep(time.Second * 2) ms, err := daemon.setupMounts(c) if err != nil { return nil, err ``` step1 run a container which has `--restart always` and `-v /var/run/docker.sock:/sock` ``` $ docker run -ti --restart always -v /var/run/docker.sock:/sock busybox / # ``` step2 exit the the container ``` / # exit ``` and kill the daemon when you see ``` ===please stop the daemon=== ``` in the daemon log The daemon can't restart again and fail with `can't create unix socket /var/run/docker.sock: is a directory`. Signed-off-by: Lei Jitang <leijitang@huawei.com> (cherry picked from commit 7318eba) Signed-off-by: Eli Uriegas <eli.uriegas@docker.com> Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
|
Today I figured out that I had two odd settings in my system, which I was not aware of.
The second point might contribute to the race condition. If docker.sock is enabled systemd will take care of the socket even before dockerd and any container is active. Thus, the socket is there already and there is no chance that a container creates a folder instead. the standard service and sock files are set-up so that docker.service will get started AFTER docker.socket. The first point was noticed as the docker.socket file pointed to /run/docker.sock, which is ok, if /var/run is just a symlink of /run. If not dockerd and all the container looking for /var/run/docker.sock (the standard path) will not find it. I stopped all services, renamed the original folder After a reboot all dynamic content was placed into Now I have no problem with the socket creation anymore at boot time. Hope this helps others as well. |
… #30348 If a container mount the socket the daemon is listening on into container while the daemon is being shutdown, the socket will not exist on the host, then daemon will assume it's a directory and create it on the host, this will cause the daemon can't start next time. fix issue moby/moby#30348 To reproduce this issue, you can add following code ``` --- a/daemon/oci_linux.go +++ b/daemon/oci_linux.go @@ -8,6 +8,7 @@ import ( "sort" "strconv" "strings" + "time" "github.com/Sirupsen/logrus" "github.com/docker/docker/container" @@ -666,7 +667,8 @@ func (daemon *Daemon) createSpec(c *container.Container) (*libcontainerd.Spec, e if err := daemon.setupIpcDirs(c); err != nil { return nil, err } - + fmt.Printf("===please stop the daemon===\n") + time.Sleep(time.Second * 2) ms, err := daemon.setupMounts(c) if err != nil { return nil, err ``` step1 run a container which has `--restart always` and `-v /var/run/docker.sock:/sock` ``` $ docker run -ti --restart always -v /var/run/docker.sock:/sock busybox / # ``` step2 exit the the container ``` / # exit ``` and kill the daemon when you see ``` ===please stop the daemon=== ``` in the daemon log The daemon can't restart again and fail with `can't create unix socket /var/run/docker.sock: is a directory`. Signed-off-by: Lei Jitang <leijitang@huawei.com> (cherry picked from commit 7318eba) Signed-off-by: Eli Uriegas <eli.uriegas@docker.com> Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
… #30348 If a container mount the socket the daemon is listening on into container while the daemon is being shutdown, the socket will not exist on the host, then daemon will assume it's a directory and create it on the host, this will cause the daemon can't start next time. fix issue moby/moby#30348 To reproduce this issue, you can add following code ``` --- a/daemon/oci_linux.go +++ b/daemon/oci_linux.go @@ -8,6 +8,7 @@ import ( "sort" "strconv" "strings" + "time" "github.com/Sirupsen/logrus" "github.com/docker/docker/container" @@ -666,7 +667,8 @@ func (daemon *Daemon) createSpec(c *container.Container) (*libcontainerd.Spec, e if err := daemon.setupIpcDirs(c); err != nil { return nil, err } - + fmt.Printf("===please stop the daemon===\n") + time.Sleep(time.Second * 2) ms, err := daemon.setupMounts(c) if err != nil { return nil, err ``` step1 run a container which has `--restart always` and `-v /var/run/docker.sock:/sock` ``` $ docker run -ti --restart always -v /var/run/docker.sock:/sock busybox / # ``` step2 exit the the container ``` / # exit ``` and kill the daemon when you see ``` ===please stop the daemon=== ``` in the daemon log The daemon can't restart again and fail with `can't create unix socket /var/run/docker.sock: is a directory`. Signed-off-by: Lei Jitang <leijitang@huawei.com> Upstream-commit: 7318eba Component: engine
… #30348 If a container mount the socket the daemon is listening on into container while the daemon is being shutdown, the socket will not exist on the host, then daemon will assume it's a directory and create it on the host, this will cause the daemon can't start next time. fix issue moby/moby#30348 To reproduce this issue, you can add following code ``` --- a/daemon/oci_linux.go +++ b/daemon/oci_linux.go @@ -8,6 +8,7 @@ import ( "sort" "strconv" "strings" + "time" "github.com/Sirupsen/logrus" "github.com/docker/docker/container" @@ -666,7 +667,8 @@ func (daemon *Daemon) createSpec(c *container.Container) (*libcontainerd.Spec, e if err := daemon.setupIpcDirs(c); err != nil { return nil, err } - + fmt.Printf("===please stop the daemon===\n") + time.Sleep(time.Second * 2) ms, err := daemon.setupMounts(c) if err != nil { return nil, err ``` step1 run a container which has `--restart always` and `-v /var/run/docker.sock:/sock` ``` $ docker run -ti --restart always -v /var/run/docker.sock:/sock busybox / # ``` step2 exit the the container ``` / # exit ``` and kill the daemon when you see ``` ===please stop the daemon=== ``` in the daemon log The daemon can't restart again and fail with `can't create unix socket /var/run/docker.sock: is a directory`. Signed-off-by: Lei Jitang <leijitang@huawei.com> (cherry picked from commit 7318eba) Signed-off-by: Eli Uriegas <eli.uriegas@docker.com> Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
|
@thaJeztah I have tried to but I get |
|
You can't remove those if docker is running (or containers) are running (which could be if the daemon has To fix the issue with |
|
@thaJeztah Thank you, I think I have identified the cause. The problem seems to be this $ sudo -i service docker status
docker start/running, process 10847
$ docker info
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?So if I try then if I try $ sudo dockerd
INFO[0000] libcontainerd: previous instance of containerd still alive (3097)
WARN[0000] failed to rename /mnt/docker/tmp for background deletion: %!s(<nil>). Deleting synchronously
INFO[0000] [graphdriver] using prior storage driver: overlay2
WARN[0000] libcontainerd: unknown container 22731aae271f9e380def120714ff025df434c96ab189221bdb93241b59fff077
WARN[0000] libcontainerd: unknown container 22731aae271f9e380def120714ff025df434c96ab189221bdb93241b59fff077
Error starting daemon: error while opening volume store metadata database: timeoutSo I tried to clean up a bit any previous instances: ps axf | grep docker | grep -v grep | awk '{print "kill -9 " $1}' | sudo sh and now I get $ sudo dockerd
INFO[0000] libcontainerd: new containerd process, pid: 25085
WARN[0001] failed to rename /mnt/docker/tmp for background deletion: %!s(<nil>). Deleting synchronously
INFO[0001] [graphdriver] using prior storage driver: overlay2
INFO[0001] Graph migration to content-addressability took 0.00 seconds
WARN[0001] Your kernel does not support cgroup rt period
WARN[0001] Your kernel does not support cgroup rt runtime
INFO[0001] Loading containers: start.
ERRO[0001] Failed to load container 05f727bf4221a0391e4a6cbb241d74683b44ed680fe4137b9fb18b48e2e11b45: open /mnt/docker/containers/05f727bf4221a0391e4a6cbb241d74683b44ed680fe4137b9fb18b48e2e11b45/config.v2.json: no such file or directory
...
ERRO[0002] get ubuntu_fs-17368c5e: no such volume
ERRO[0002] get ubuntu_fs-17368c5e: no such volume
ERRO[0002] get ubuntu_fs-17368c5e: no such volume
INFO[0002] Loading containers: done.
INFO[0002] Daemon has completed initialization
INFO[0002] Docker daemon commit=89658be graphdriver=overlay2 version=17.05.0-ce
INFO[0002] API listen on /var/run/docker.sock Now $ docker info
Containers: 9
Running: 1
Paused: 0
Stopped: 8
Images: 219
Server Version: 17.05.0-ce
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: efs local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-initbut still |
Does it work if you run with What version of compose are you running, and what version do you have specified in your docker-compose file? I see you're running a really old version of docker (17.05) that reached EOL three years ago; I'd highly recommend upgrading to a more current version as that version has known, unpatched vulnerabilities |
|
I got hit with this issue while upgrading from The only container that mounted Not sure how to solve this, but it seems strange that the containers in "restart-mode" would start before the docker-daemon is listening to the socket? Maybe something can be built for docker to wait for that prior to launching containers? Or add the workaround mentioned by @stuszynski in the upstream packaging :) |
The daemon and API are separate bits in the code; the daemon may be up, but the API not yet listening. There's also scenarios (e.g. the
The systemd socket approach (#30348 (comment)) is already in use in all current versions of docker: https://github.com/docker/docker-ce-packaging/blob/a5db88ae1a64189e79d97f780f91e5c852d0ef3f/systemd/docker.service#L6-L13 The default is for the docker daemon to use There may be one fix for the systemd unit file related to this, that hasn't shipped yet; docker/docker-ce-packaging#575 (not sure if it addresses this particular issue, but might help) |
|
I just ran into this issue today, seems that I had a container use the |
When I mount /var/run instead of /var/run/docker.sock the docker commands work in the container, however this variable does not appear to give the container's host ip anymore: host.docker.internal |
|
Relevant StackOverflow answer that can land you here on this bug page: https://stackoverflow.com/a/62209937/10534510 |
and others
Description
Steps to reproduce the issue:
Describe the results you received:
Docker can't boot up after a restart. An error from journal:
At this point
/var/run/docker.sockis indeed, a directory. (wut?)Describe the results you expected:
To restart Docker without an error
Additional information you deem important (e.g. issue happens only occasionally):
This happened from time to time upon restart of
dockerddaemon. After I removed this directory manually, Docker boots up easily creating a socket, but after several restarts, this issue came back.Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
AWS EC2 instance
The text was updated successfully, but these errors were encountered: