failed to convert whiteout file: operation not supported: unknown

[sheey11]

I am trying to install docker on my OpenWRT machine, docker pull shows error message below:

$ docker pull python:3.9-alpine
3.9-alpine: Pulling from library/python
af6eaf76a39c: Pull complete
cdba2f686752: Extracting [==================================================>]  624.9kB/624.9kB
bcae47abf4b5: Download complete
b279af43ee8b: Download complete
512e88ffaaf1: Download complete
failed to register layer: operation not supported

then I turn to containerd, this is what bothers me now:

$ ctr --debug --address /var/run/docker/containerd/containerd.sock images pull docker.io/library/python:3.9-alpine
DEBU[0000] fetching                                      image="docker.io/library/python:3.9-alpine"
DEBU[0000] resolving                                     host=registry-1.docker.io
DEBU[0000] do request                                    host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.13 request.method=HEAD url="https://registry-1.docker.io/v2/library/python/manifests/3.9-alpine"
DEBU[0002] fetch response received                       host=registry-1.docker.io response.header.content-length=157 response.header.content-type=application/json response.header.date="Mon, 06 Mar 2023 14:50:02 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=125.69.189.242 response.header.strict-transport-security="max-age=31536000" response.header.www-authenticate="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/python:pull\"" response.status="401 Unauthorized" url="https://registry-1.docker.io/v2/library/python/manifests/3.9-alpine"
DEBU[0002] Unauthorized                                  header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/python:pull\"" host=registry-1.docker.io
DEBU[0002] do request                                    host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.13 request.method=HEAD url="https://registry-1.docker.io/v2/library/python/manifests/3.9-alpine"
DEBU[0007] fetch response received                       host=registry-1.docker.io response.header.content-length=1645 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Mon, 06 Mar 2023 14:50:08 GMT" response.header.docker-content-digest="sha256:a02fee195c815e1a1038513af6be324a4e401667bb10eb1dfdbbeda2d780748c" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=125.69.189.242 response.header.etag="\"sha256:a02fee195c815e1a1038513af6be324a4e401667bb10eb1dfdbbeda2d780748c\"" response.header.ratelimit-limit="100;w=21600" response.header.ratelimit-remaining="86;w=21600" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" url="https://registry-1.docker.io/v2/library/python/manifests/3.9-alpine"DEBU[0007] resolved                                      desc.digest="sha256:a02fee195c815e1a1038513af6be324a4e401667bb10eb1dfdbbeda2d780748c" host=registry-1.docker.io
DEBU[0007] fetch                                         digest="sha256:a02fee195c815e1a1038513af6be324a4e401667bb10eb1dfdbbeda2d780748c" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=1645
DEBU[0007] fetch                                         digest="sha256:e1f27cb750564f7e713d6483f11cfef6dbd4c9bb1142d9030457adab87140b6b" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1368
DEBU[0007] fetch                                         digest="sha256:cdba2f6867522ee9d2e8dcc9b1ad3ec5bafe75da976a3e613ff26cbc98738b48" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=624860
DEBU[0007] fetch                                         digest="sha256:512e88ffaaf18f89120107ee9f184aed79dfdeea7ffe0c5e183464fa4db16189" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=2879418
DEBU[0007] fetch                                         digest="sha256:a236c715ca8a0c920fd54c20d3ed5e1bed8cab560ea3fd590c4e3fb0c0680829" mediatype=application/vnd.docker.container.image.v1+json size=7325
DEBU[0007] fetch                                         digest="sha256:af6eaf76a39c2d3e7e0b8a0420486e3df33c4027d696c076a99a3d0ac09026af" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=3261959
DEBU[0007] fetch                                         digest="sha256:b279af43ee8b4beba1e30b95c4890396463eb4606004b6b670222e17af8b582f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=230
DEBU[0007] fetch                                         digest="sha256:bcae47abf4b57779b5ab0771fd1a7768a667be1c0d37a1d6a1967cb14b63c91f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11538498
DEBU[0007] unpacking                                     image="docker.io/library/python:3.9-alpine"
unpacking linux/arm64/v8 sha256:a02fee195c815e1a1038513af6be324a4e401667bb10eb1dfdbbeda2d780748c...
INFO[0008] apply failure, attempting cleanup             error="failed to extract layer sha256:2b6d33d15c9f2a4c4620f9b30afde93cdc72b3a4d50177d3b4ab5f9eaa02d60f: failed to convert whiteout file \"etc/ca-certificates/.wh..wh..opq\": operation not supported: unknown" key="extract-709964109-BcSI sha256:88f2b18b2ddcaec1663a9ae85612c3f9a51582fc5798176d9860608fcfef5764"
ctr: failed to extract layer sha256:2b6d33d15c9f2a4c4620f9b30afde93cdc72b3a4d50177d3b4ab5f9eaa02d60f: failed to convert whiteout file "etc/ca-certificates/.wh..wh..opq": operation not supported: unknown

I searched internet but I can't find anything useful, maybe related:

• support or disable direct unpack when running in userns #3762
  He got operation not permitted, but I got operation not supported, and the pr apply: use naive applier when running in UserNS #3763 is merged at containerd 1.3.0
• running containerd and its client in a container #5522
• openwrt forum helps me diagnosis

---

$ containerd --version
containerd github.com/containerd/containerd 1.6.13
$ dockerd --version
Docker version 20.10.22, build 42c8b31
$ df -T -h
Filesystem           Type            Size      Used Available Use% Mounted on
/dev/root            ext4          102.3M     64.4M     35.9M  64% /rom
tmpfs                tmpfs         494.5M      7.6M    486.9M   2% /tmp
/dev/mmcblk0p3       ext4           28.9G    887.3M     26.6G   3% /overlay
overlayfs:/overlay   overlay        28.9G    887.3M     26.6G   3% /
tmpfs                tmpfs         512.0K         0    512.0K   0% /dev
overlayfs:/overlay   overlay        28.9G    887.3M     26.6G   3% /opt/docker

$ cat /var/run/docker/containerd/containerd.toml
version = 0
root = "/opt/docker/containerd/daemon"
state = "/var/run/docker/containerd/daemon" # /var is tmpfs
plugin_dir = ""
disabled_plugins = ["cri"]
oom_score = 0

[grpc]
  address = "/var/run/docker/containerd/containerd.sock"
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[ttrpc]
  address = ""
  uid = 0
  gid = 0

[debug]
  address = "/var/run/docker/containerd/containerd-debug.sock"
  uid = 0
  gid = 0
  level = "debug"
  format = "json"

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = "/opt/docker/runc"
    no_shim = false
    shim_debug = false

[sheey11]

this is weired, today morning, I'm still trying to figure out problem, I tried docker pull and it suddenly works. Here's what I'm done so far:

1. I found /opt/docker/runc is missing, idk if it is related to this problem, so I go ln -s ${which runc} /opt/docker/runc, but the problem still, so I deleted the symlink.
2. I modified [debug] section of containerd.toml to level = "trace", format = "text", then

   $ /etc/init.d/dockerd stop
   $ containerd --config /var/run/docker/containerd/containerd.toml --log-level trace
     # nothing useful, the only error is that `unpigz` not found and falling back to `gzip`
   $ /etc/init.d/dockerd start
     # and then docker pull works
   $ reboot
     # still works

I have no idea why and how it is solved, but for anyone came across this problem, I hope these "steps" works for you.