Rootless podman pull: Error: kernel does not support overlay fs

[amirgon]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I'm trying to create an image storage on a shared directory, to be used with additionalimagestores.
Following Exploring additional image stores in Podman, I'm running:

$ podman --root ./image_storage pull ubuntu:20.04

Fails with:

Error: kernel does not support overlay fs: unable to create kernel-style whiteout: operation not permitted

Running sudo podman --root ./image_storage pull works, but then the created image storage cannot be used by a rootless podman.
When setting additionalimagestores to point to a readonly share with the image storage created with sudo podman, I'm getting:

$ podman images
REPOSITORY                                 TAG         IMAGE ID      CREATED      SIZE        R/O
docker.io/library/ubuntu                   20.04       3bc6e9f30f51  7 days ago   75.2 MB     true

$ podman run --rm -it ubuntu:20.04
Error: creating /etc/mtab symlink: permission denied

Rootless podman gives this error, but sudo podman run --rm -it ubuntu:20.04 works here.

As a workaround I can run everything with sudo, but it's not clear to me what is the issue with rootless podman here.

I'm running podman 3.4.2 on Ubuntu 20.04.4, but also tried containerized podman from quay.io/podman/stable.

Steps to reproduce the issue:

Create an image storage with podman --root

Describe the results you received:

Error: kernel does not support overlay fs: unable to create kernel-style whiteout: operation not permitted

Steps to reproduce the issue:

• Create an image storage with sudo podman --root
• set additionalimagestores
• podman run an image from the storage image

Describe the results you received:

Error: creating /etc/mtab symlink: permission denied

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 3.4.2

Tried also with podman run -it --rm quay.io/podman/stable

podman version 4.1.1

Output of podman info:

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 100:3.4.2-5 amd64 [installed]

podman-4.1.1-3.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Creating the image storage on Ubuntu 20.04.4 LTS

Running the image on Fedora release 36

[rhatdan]

Do you have fuse-overlay installed?

[rhatdan]

On fedora 36 this worked fine for me.

$ podman --root ./image_storage pull ubuntu:20.04
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:20.04...
Getting image source signatures
Copying blob 3b65ec22a9e9 done
Copying config 3bc6e9f30f done
Writing manifest to image destination
Storing signatures
3bc6e9f30f51d2bbf9307fc9d0bdfc30caa38cf4e3b05a714230f9a9b3381d84

[rhatdan]

You can setup rootful additionalimagestore and use it with rootless users as long as you use fuse-overlay, correct @giuseppe

[amirgon]

Hi @rhatdan

Do you have fuse-overlay installed?

Could you explain why is it needed in this case?
If rootless podman pull ubuntu:20.04 works correctly for me, what makes rootless podman --root ./image_storage pull ubuntu:20.04 fail?

[giuseppe]

You can setup rootful additionalimagestore and use it with rootless users as long as you use fuse-overlay, correct @giuseppe

yes, that should work as long as the additionalimagestore has a force_mask=755.

@amirgon what is the underlying file system where you are trying to pull (stat -f ./image_storage)?

[amirgon]

@amirgon what is the underlying file system where you are trying to pull (stat -f ./image_storage)?

It says Type: ext2/ext3

yes, that should work as long as the additionalimagestore has a force_mask=755.

But the image store in my case is read-only on the client side. Should I still set force_mask=shared in such case?

[giuseppe]

yes, you need both force_mask (before pulling an image) and run sudo chmod -R 0755 /path/to/the/store to make sure files are accessible to rootless users.

If you use fuse-overlayfs for the rootless user, then it will present files with the correct ownership, otherwise native overlay will show all files with mode 0755.

[amirgon]

yes, you need both force_mask (before pulling an image) and run sudo chmod -R 0755 /path/to/the/store to make sure files are accessible to rootless users.

It works!

I've added to storage.conf:

force_mask = "shared"
mount_program = "/usr/bin/fuse-overlayfs"

Worth mentioning this in Exploring additional image stores in Podman.

@giuseppe @rhatdan Thank you very much for your help!

[hmoffatt]

I added force_mask = "shared" in $HOME/.config/containers/storage.conf, but I don't see any difference in the resulting permissions when I use podman --root /path pull.

podman system info says:

store:
  configFile: /home/vagrant/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.force_mask: shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs_1.4.0-1_amd64
      Version: |-
        fusermount3 version: 3.10.3
        fuse-overlayfs: version 1.4
        FUSE library version 3.10.3
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev
  graphRoot: /home/vagrant/.local/share/containers/storage

But when I run podman --root /path system info the extra configuration settings are missing:

store:
  configFile: /home/vagrant/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/vagrant/img

My podman is 4.3.1.

Also it seems logical that I would want to apply special permissions when writing to a to-be-shared path, but not necessarily to my own user's local storage?