Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support or disable direct unpack when running in userns #3762

Closed
AkihiroSuda opened this issue Oct 19, 2019 · 5 comments · Fixed by #3763
Closed

support or disable direct unpack when running in userns #3762

AkihiroSuda opened this issue Oct 19, 2019 · 5 comments · Fixed by #3763
Labels
kind/bug

Comments

@AkihiroSuda
Copy link
Member

Description

Direct unpack added in #3528 (v1.3.0) doesn't work well when running in user namespace:

E1019 19:24:27.024977   21918 remote_runtime.go:200] CreateContainer in sandbox "941524a9dca443253b954cc64183131f5cc89054cd0dd70f4ab342671cd1bf1f" from runtime service failed: rpc error: code = Unknown desc = failed to create containerd container: error unpacking image: failed to extract layer sha256:d635f458a6f8a4f3dd57a597591ab8977588a5a477e0a68027d18612a248906f: failed to convert whiteout file "etc/ca-certificates/.wh..wh..opq": operation not permitted: unknown
E1019 19:24:27.025143   21918 kuberuntime_manager.go:783] container start failed: CreateContainerError: failed to create containerd container: error unpacking image: failed to extract layer sha256:d635f458a6f8a4f3dd57a597591ab8977588a5a477e0a68027d18612a248906f: failed to convert whiteout file "etc/ca-certificates/.wh..wh..opq": operation not permitted: unknown

The error happens because whiteout cannot be created with mknod in user namespace.

Moby implementation supports creating whiteout in a hacky alternative way:

https://github.com/moby/moby/blob/a8b04b17fd37ed797e34bea6534d307929c6337b/pkg/archive/archive_linux.go#L122-L175

// mknodChar0Overlay creates 0,0 char device by mounting overlayfs and unlinking.
// This function can be used for creating 0,0 char device in userns on Ubuntu.
//
// Steps:
// * Mkdir lower,upper,merged,work
// * Create lower/dummy
// * Mount overlayfs
// * Unlink merged/dummy
// * Unmount overlayfs
// * Make sure a 0,0 char device is created as upper/dummy
// * Rename upper/dummy to cleansedOriginalPath

Steps to reproduce the issue:

  • Run containerd/cri in rootless mode

Describe the results you received:

failed to convert whiteout file "etc/ca-certificates/.wh..wh..opq": operation not permitted: unknown

Describe the results you expected:

It should either support or disable direct unpack

Output of containerd --version:
1.3.0
@AkihiroSuda AkihiroSuda mentioned this issue Oct 19, 2019
Closed
@AkihiroSuda AkihiroSuda mentioned this issue Oct 19, 2019
Merged
@AkihiroSuda
Copy link
Member Author

PR to disable direct unpack: #3763

AkihiroSuda added a commit to AkihiroSuda/containerd that referenced this issue Oct 24, 2019
@AkihiroSuda
apply: use naive applier when running in UserNS
c224edc 
`OverlayConvertWhiteout` calls `mknod c 0 0` which is not allowed when
running in a user namespace, even in Ubuntu kernel.

Although there is an alternative hacky way to create whiteouts without
calling mknod as Moby `overlay2` actually does(see containerd#3762), let's use
naive applier when running in UserNS and call it a day.

Close containerd#3762

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda added a commit to AkihiroSuda/containerd that referenced this issue Oct 25, 2019
@AkihiroSuda
apply: use naive applier when running in UserNS
f3c48da 
`OverlayConvertWhiteout` calls `mknod c 0 0` which is not allowed when
running in a user namespace, even in Ubuntu kernel.

Although there is an alternative hacky way to create whiteouts without
calling mknod as Moby `overlay2` actually does(see containerd#3762), let's use
naive applier when running in UserNS and call it a day.

Close containerd#3762

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit c224edc)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
@thaJeztah thaJeztah mentioned this issue Nov 8, 2019
Merged
dmcgowan pushed a commit to thaJeztah/containerd that referenced this issue Nov 19, 2019
@thaJeztah @dmcgowan
Prepare v1.3.1 release
d512382 
* Update the runc vendor to v1.0.0-rc9 which includes an additional mitigation for [CVE-2019-16884](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-16884).
    - More details on the runc CVE in [opencontainers/runc#2128](opencontainers/runc#2128), and the additional mitigations in [opencontainers/runc#2130](opencontainers/runc#2130).
* Add local-fs.target to service file to fix corrupt image after unexpected host reboot. Reported in [containerd#3671](containerd#3671), and fixed by [containerd#3745](containerd#3745).
* Fix large output of processes with TTY getting occasionally truncated. Reported in [containerd#3738](containerd#3738) and fixed by [containerd#3754](containerd#3754).
* Fix direct unpack when running in user namespace. Reported in [containerd#3762](containerd#3762), and fixed by [containerd#3779](containerd#3779).
* Update Golang runtime to 1.12.13, which includes security fixes to the `crypto/dsa` package made in Go 1.12.11 ([CVE-2019-17596](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17596)), and fixes to the go command, `runtime`, `syscall` and `net` packages (Go 1.12.12).
* Add Windows process shim installer [containerd#3792](containerd#3792)

* CRI fixes:
    - Fix shim delete error code to avoid unnecessary retries in the CRI plugin. Discovered in [containerd/cri#1309](containerd/cri#1309), and fixed by [containerd#3733](containerd#3733) and [containerd#3740](containerd#3740).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@errordeveloper
Copy link
Contributor

In case anyone else reads this, I faced same issue with containerd 1.3.3 running in kata pod, turned out it was to do with /var/lib being on a 9p filesystem, which doesn't support mknod. For my use-case It was sufficient for to set root = /run/containerd/root and state = /run/containerd/state in the containerd config file, as /run is a tmpfs and I also got some performance improvements on image pulls that way (I do have plenty of memory to spare, at least at the moment).

@rohinb2 rohinb2 mentioned this issue Jun 23, 2020
Open
This was referenced Jul 19, 2020
Closed
Closed
tussennet pushed a commit to tussennet/containerd that referenced this issue Sep 11, 2020
@AkihiroSuda @87654testtesttesttesttesttest
apply: use naive applier when running in UserNS
55f8fe7 
`OverlayConvertWhiteout` calls `mknod c 0 0` which is not allowed when
running in a user namespace, even in Ubuntu kernel.

Although there is an alternative hacky way to create whiteouts without
calling mknod as Moby `overlay2` actually does(see containerd#3762), let's use
naive applier when running in UserNS and call it a day.

Close containerd#3762

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
@liiri
Copy link

liiri commented Sep 19, 2021

In case anyone else reads this, I faced same issue with containerd 1.3.3 running in kata pod, turned out it was to do with /var/lib being on a 9p filesystem, which doesn't support mknod. For my use-case It was sufficient for to set root = /run/containerd/root and state = /run/containerd/state in the containerd config file, as /run is a tmpfs and I also got some performance improvements on image pulls that way (I do have plenty of memory to spare, at least at the moment).

I don't think you should use root on a tmpfs -- your state won't get saved on machine restart and will cause unexpected behavior when trying to start containerd service

This was referenced Nov 22, 2021
Closed
Closed
@mathis-m
Copy link
Contributor

it fails in user namespace with containerd github.com/containerd/containerd v1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1

error

FATA[0006] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/jupyterhub/configurable-http-proxy:4.5.3": failed to extract layer sha256:1be7bfe3945a9f69b537b3d4112245b7290837feadb68fca2214cc5f33a6386d: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount2456203030: failed to convert whiteout file "usr/local/lib/node_modules/.wh.npm": unlinkat /var/lib/containerd/tmpmounts/containerd-mount2456203030/usr/local/lib/node_modules/npm/node_modules/yallist: input/output error: unknown

@AkihiroSuda how can I get this to work within the user namespace?

@AkihiroSuda
Copy link
Member Author

@mathis-m Could you open a new issue with details (minimum repro steps, host kernel version, filesystem, etc.)

@mathis-m mathis-m mentioned this issue Nov 24, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

apply: use naive applier when running in UserNS AkihiroSuda/containerd
4 participants
@errordeveloper @AkihiroSuda @mathis-m @liiri