support or disable direct unpack when running in userns

[AkihiroSuda]

Description

Direct unpack added in #3528 (v1.3.0) doesn't work well when running in user namespace:

E1019 19:24:27.024977   21918 remote_runtime.go:200] CreateContainer in sandbox "941524a9dca443253b954cc64183131f5cc89054cd0dd70f4ab342671cd1bf1f" from runtime service failed: rpc error: code = Unknown desc = failed to create containerd container: error unpacking image: failed to extract layer sha256:d635f458a6f8a4f3dd57a597591ab8977588a5a477e0a68027d18612a248906f: failed to convert whiteout file "etc/ca-certificates/.wh..wh..opq": operation not permitted: unknown
E1019 19:24:27.025143   21918 kuberuntime_manager.go:783] container start failed: CreateContainerError: failed to create containerd container: error unpacking image: failed to extract layer sha256:d635f458a6f8a4f3dd57a597591ab8977588a5a477e0a68027d18612a248906f: failed to convert whiteout file "etc/ca-certificates/.wh..wh..opq": operation not permitted: unknown

The error happens because whiteout cannot be created with mknod in user namespace.

Moby implementation supports creating whiteout in a hacky alternative way:

https://github.com/moby/moby/blob/a8b04b17fd37ed797e34bea6534d307929c6337b/pkg/archive/archive_linux.go#L122-L175

// mknodChar0Overlay creates 0,0 char device by mounting overlayfs and unlinking.
// This function can be used for creating 0,0 char device in userns on Ubuntu.
//
// Steps:
// * Mkdir lower,upper,merged,work
// * Create lower/dummy
// * Mount overlayfs
// * Unlink merged/dummy
// * Unmount overlayfs
// * Make sure a 0,0 char device is created as upper/dummy
// * Rename upper/dummy to cleansedOriginalPath

Steps to reproduce the issue:

• Run containerd/cri in rootless mode

Describe the results you received:

failed to convert whiteout file "etc/ca-certificates/.wh..wh..opq": operation not permitted: unknown

Describe the results you expected:

It should either support or disable direct unpack

Output of containerd --version:
1.3.0

[AkihiroSuda]

PR to disable direct unpack: #3763

[errordeveloper]

In case anyone else reads this, I faced same issue with containerd 1.3.3 running in kata pod, turned out it was to do with /var/lib being on a 9p filesystem, which doesn't support mknod. For my use-case It was sufficient for to set root = /run/containerd/root and state = /run/containerd/state in the containerd config file, as /run is a tmpfs and I also got some performance improvements on image pulls that way (I do have plenty of memory to spare, at least at the moment).