Error "link /bin/[ /bin/[[: operation not supported" when using Docker:dind container to load image and push to image service

[spoonysnail]

Description

I am trying to use Alibaba Cloud OSS (Object Storage Service) as the Docker Engine storage volume and leverage a Docker:dind (Docker-in-Docker) container to perform image extraction and push operations to an image registry.

To achieve this, I have created a Pod where the dind container starts Docker Engine and the clean-ci container performs the docker load and docker push operations.

However, when executing the docker load command, I encounter the following error: "ApplyLayer exit status 1 stdout: stderr: link /bin/[ /bin/[[: operation not supported".

This error seems to be related to an unsupported link operation, possibly caused by the underlying operating system not supporting certain operations within the Docker:dind container.

I would appreciate any guidance, suggestions, or best practices on how to resolve this error and successfully load the image and push it to the image service while utilizing Alibaba Cloud OSS as the Docker Engine storage volume. Thank you.

Reproduce

The YAML file of the pod is similar to the following:

apiVersion: v1
kind: Pod
metadata:
  name: clean-ci
spec:
  containers:
  - name: dind
    image: 'docker:stable-dind'
    command:
    - dockerd
    - --host=unix:///var/run/docker.sock
    - --host=tcp://0.0.0.0:8000
    - --data-root=/oss_volume/docker-data
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /var/run
      name: cache-dir
    - mountPath: /oss_volume
      name: oss-pvc
  - name: clean-ci
    image: 'docker:stable'
    command: ["/bin/sh", "-c"]
    args: ["docker load -i xxx.tar; docker push xxxx"]
    volumeMounts:
    - mountPath: /var/run
      name: cache-dir
  volumes:
  - name: cache-dir
    emptyDir: {}
  - name: oss-pvc
    persistentVolumeClaim:
      claimName: my-oss-pvc

Expected behavior

No response

docker version

Client: Docker Engine - Community
 Version:           19.03.14
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        5eb3275
 Built:             Tue Dec  1 19:14:24 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.14
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       5eb3275
  Built:            Tue Dec  1 19:21:08 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.9
  GitCommit:        ea765aba0d05254012b0b9e595e995c09186427f
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

docker info

Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.14
 Storage Driver: vfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ea765aba0d05254012b0b9e595e995c09186427f
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.91-27.3.al7.x86_64
 Operating System: Alpine Linux v3.12 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.12GiB
 Name: clean-ci2
 ID: HNU6:VVWV:KF2B:ITWH:E2Q5:HZ2N:Z2AU:AX2O:ME65:MHZC:R6XY:CSK2
 Docker Root Dir: /ai4s-image-registry/test0907
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: API is accessible on http://0.0.0.0:8000 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional Info

No response

[thaJeztah]

Hm.. not exactly sure, as there's quite some possible moving parts here.

First of all; the docker:stable image has been deprecated, which is why you get a very old version of the docker engine; you can use the docker:dind tag instead, or a specific version (such as docker:24.0-dind; https://hub.docker.com/_/docker/

As to the error you're seeing; if could be filesystem related; I see the daemon picked vfs as filesystem (which I would not recommend), possibly because it's using the container's filesystem which doesn't support overlayfs; if you mount a volume at /var/lib/docker that's on a regular filesystem then possibly that part would work.

I'll move this ticket to a discussion, because this is more a support / question than a bug-report.