Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Debian] Rootless Docker 20.10.6 overlay2 error: ApplyLayer exit status 1 stdout #42302

Open
gymnae opened this issue Apr 16, 2021 · 7 comments

Comments

@gymnae
Copy link

gymnae commented Apr 16, 2021

<EDIT> (by @AkihiroSuda)

Workaround: use fuse-overlayfs

curl -o $HOME/bin/fuse-overlayfs -fsSL https://github.com/containers/fuse-overlayfs/releases/download/v1.4.0/fuse-overlayfs-$(uname -m)

chmod +x $HOME/bin/fuse-overlayfs

echo '{"storage-driver": "fuse-overlayfs"}' > ~/.config/docker/daemon.json

systemctl --user restart docker

</EDIT>


Description

Even with #42188 in place, I'm facing issues with overlay2 fs driver in a docker rootless environment

Steps to reproduce the issue:

  1. Install docker rootless via package management - in my case debian 10
  2. $ docker pull ghost:latest
  3. See error: failed to register layer: ApplyLayer exit status 1 stdout: stderr: unlinkat /tmp/v8-compile-cache-0/8.4.371.19-node.18: input/output error

Describe the results you received:

Every time I try, I receive the error describe above:
failed to register layer: ApplyLayer exit status 1 stdout: stderr: unlinkat /tmp/v8-compile-cache-0/8.4.371.19-node.18: input/output error

Describe the results you expected:
Pulling and extracting the image without error

Additional information you deem important (e.g. issue happens only occasionally):
Issue is reproducable with fs overlay2 it does not appear with fs fuse-overlayfs

Output of docker version:

Docker version 20.10.6

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 8
  Running: 8
  Paused: 0
  Stopped: 0
 Images: 18
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 5.10.0-0.bpo.4-amd64
 Operating System: Debian GNU/Linux 10 (buster) (error determining if containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 10
 Total Memory: 9.728GiB
 Name: <hostname>
 ID: Y5Z7:5PCZ:P5VR:DQFO:JJDD:RAC3:X4BR:CQ4U:4LTE:KFTO:HAET:IWCA
 Docker Root Dir: /home/<username>/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
Running on a Debian 10 VPS with kernel 5.10 from back ports, host OS unknown

@gymnae
Copy link
Author

gymnae commented Apr 16, 2021

Facing the same issue with wireguard:

Using default tag: latest
latest: Pulling from linuxserver/wireguard
6942b413fb6b: Pull complete
f391502faaaf: Pull complete
063202cdb66e: Pull complete
10bd9c606ee7: Pull complete
a0bb3791992f: Pull complete
fbb83093d80c: Extracting [==================================================>]   11.8MB/11.8MB
a9ad653b3f46: Download complete
44d4eebb0db5: Download complete
53338848dfd6: Download complete
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: unlinkat /tmp/patch/etc/s6/init: input/output error

@gymnae gymnae changed the title Rootless Docker 20.10.6 overlay2 error Rootless Docker 20.10.6 overlay2 error: ApplyLayer exit status 1 stdout Apr 16, 2021
@AkihiroSuda
Copy link
Member

Is this new in 20.10.6?

@gymnae
Copy link
Author

gymnae commented Apr 17, 2021

I think so. I didn't even get that far before your bug fix for overlay2 :)

@AkihiroSuda
Copy link
Member

AkihiroSuda commented Apr 19, 2021

This seems specific to Debian?

I can repro the issue with Debian 10 (Linux 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19)), but can't with Ubuntu 20.10 (Linux 5.8.0-48-generic #54-Ubuntu SMP Fri Mar 19 14:25:20 UTC 2021).

cc @zhsj

@AkihiroSuda AkihiroSuda changed the title Rootless Docker 20.10.6 overlay2 error: ApplyLayer exit status 1 stdout [Debian] Rootless Docker 20.10.6 overlay2 error: ApplyLayer exit status 1 stdout Apr 19, 2021
@zhsj
Copy link
Contributor

zhsj commented Apr 19, 2021

This seems specific to Debian?

I can repro the issue with Debian 10 (Linux 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19)), but can't with Ubuntu 20.10 (Linux 5.8.0-48-generic #54-Ubuntu SMP Fri Mar 19 14:25:20 UTC 2021).

IMO, the Debian patch to userns overlayfs is just broken. See my reply on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969223

It's better not to advertise rootless overlayfs feature for Debian. Let's just wait the 5.11 kernel.

@gymnae
Copy link
Author

gymnae commented Apr 19, 2021

Waiting for 5.11 has a downside that it will most likely affect Debian 10 users.
They won’t be offered this Kernel via backports, which might affect server admins.
I might have to go back to fuse-overlayfs, which feels slower, so far.

AkihiroSuda added a commit to AkihiroSuda/rootlesscontaine.rs that referenced this issue Apr 23, 2021
Debian variant (before kernel 5.11) of kernel-mode rootless overlayfs seems broken, while Ubuntu variant seems fine.

moby/moby#42302

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/rootlesscontaine.rs that referenced this issue Apr 23, 2021
Debian variant (before kernel 5.11) of kernel-mode rootless overlayfs seems broken, while Ubuntu variant seems fine.

moby/moby#42302

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
@gymnae
Copy link
Author

gymnae commented Dec 3, 2021

Another workaround which works for me is to use a third-party kernel with Debian Buster (currently name old-stable). This allows to use Kernels higher than 5.10.
I am running liquorix kernel and was able to switch to overlay2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants