Cannot connect to the Docker daemon at unix:///var/run/docker.sock.

[leo-unglaub]

Hey,
since a few days ago i get the following error when using this plugin to build docker images itself.

latest: Pulling from plugins/docker
Digest: sha256:e18e6f848cbeb88f83699e7a1f4e500e0b60930c9c400237b1e7371ef6d70963
Status: Downloaded newer image for plugins/docker:latest
+ /usr/local/bin/dockerd --data-root /var/lib/docker --host=unix:///var/run/docker.sock
Unable to reach Docker Daemon after 15 attempts.
Detected registry credentials
+ /usr/local/bin/docker version
Client:
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:45:09 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
exit status 1

I am running in the latest Alpine Linux host (Linux build 6.5.8-0-edge #1-Alpine SMP PREEMPT_DYNAMIC Fri, 20 Oct 2023 10:41:43 +0000 x86_64 Linux). I also verified that docker is running and the socket is available in the requestested location /var/run/docker.sock.

The used drone configuration:

kind: pipeline
name: leounglaub/alpine

steps:
- name: do-base
  image: plugins/docker
  settings:
    username:
      from_secret: DOCKER_USERNAME
    password:
      from_secret: DOCKER_PASSWORD
    repo: leounglaub/alpine
    tags: base
    dockerfile: alpine/base/Dockerfile

What went wrong here? Does soneone has an idea when this bug was introduced?
Thanks

[sjsf]

Same problem here running drone/drone:2.21.0 and drone/drone-runner-docker:1.8.3 with docker 24.0.7 on Alpine Linux v3.19. Did anybody find a solution to this? Or at least understood what the problem is?

[sjsf]

The timing of this indicates it is related to the OS upgrade (Alpine 3.18 -> 3.19). Also, I see that running a plain dind (docker-in-docker), it fails with some older images which are based on alpine 3.18 but work when they are based on 3.19. The only difference in the logs is this:

ip: can't find device 'ip_tables'
modprobe: can't change directory to '/lib/modules': No such file or directory
...
INFO[2024-01-03T14:51:53.156726749Z] unable to detect if iptables supports xlock: 'iptables --wait -L -n': `modprobe: can't change directory to '/lib/modules': No such file or directory
iptables v1.8.9 (legacy): can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.`  error="exit status 3"
INFO[2024-01-03T14:51:53.214336112Z] stopping healthcheck following graceful shutdown  module=libcontainerd
INFO[2024-01-03T14:51:53.214437896Z] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=moby
INFO[2024-01-03T14:51:53.214544837Z] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: modprobe: can't change directory to '/lib/modules': No such file or directory
iptables v1.8.9 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

This somehow points to some issues with iptables. Obviously the host didn't change, it's available in both cases. I noticed though that there is a difference in the iptables versions in both images:

/ # iptables --version
iptables v1.8.9 (legacy)

/ # iptables --version
iptables v1.8.10 (nf_tables)

Maybe this helps somebody to fully understand the reason. In any case, #418 fixes the issue for me.

[mamazary]

There's a problem with the pugins/gar but when I use plugins/gcr it worked fine.
I am switching from gcr to gar due to deprecation.

Here's the log while using plugins/gcr :

latest: Pulling from plugins/gcr
Digest: sha256:4b9d0eea47414a5402d5b9690faeebd9b5804f8387807a925ebcd939de6cfa75
Status: Image is up to date for plugins/gcr:latest
+ /usr/local/bin/dockerd --data-root /var/lib/docker --host=unix:///var/run/docker.sock
Detected registry credentials
+ /usr/local/bin/docker version
Client:
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:45:09 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:49:54 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
+ /usr/local/bin/docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 6.1.58+
 Operating System: Alpine Linux v3.15 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.63GiB
 Name: 5d894379f715
 ID: :STRIPPED:
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

And this one when I used plugins/gar :

latest: Pulling from plugins/gar
Digest: sha256:de9bad2216ee996be01393003acb4f85a957c53a1aaa5748130320dda38235c2
Status: Image is up to date for plugins/gar:latest
+ /usr/local/bin/dockerd --data-root /var/lib/docker --host=unix:///var/run/docker.sock
Unable to reach Docker Daemon after 15 attempts.
Detected registry credentials
+ /usr/local/bin/docker version
Client:
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:45:09 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
exit status 1
time="2024-02-26T07:41:18Z" level=fatal msg="exit status 1"

this is the yaml file :

steps:
  - name: build and push image
    image: plugins/gar
    settings:
      repo: gcp-project-id/image-repo
      registry: asia.gcr.io
      tags: ${DRONE_TAG}
      json_key:
        from_secret: GAR_TOKEN
      build_args_from_env: 
      - BITBUCKET_USER
      - BITBUCKET_TOKEN
    environment:
      BITBUCKET_USER:
        from_secret: BITBUCKET_USER
      BITBUCKET_TOKEN:
        from_secret: BITBUCKET_TOKEN

I am running the drone server and runner on kubernetes using helm chart

[smartfin]

As a temporary solution you can rollback to tag 20.17.4 for Drone plugins plugins/docker, plugins/ecr, plugins/gcr, etc