Regression in V2.2.1 with rootless containers if storage links to different filesystem

[gdoepp]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug

Description
In version v2.2.1 used in RHEL8, podman does not work anymore for rootless containers if the storage path is a symbolic link to some directory crossing file system boundaries.

Steps to reproduce the issue:

1. install podman, keep default configuration, the rootless storage path is below the user's home directory

2. link some directory from a different file system to .local/share/containers/storage

3. remove .local/share/containers/storage/libpod and run podman info

Describe the results you received:
Error: remount /home/USER/.local/share/containers/storage/overlay, flags: 0x40000: invalid argument

Same with all other subcommands.

Describe the results you expected:
no error

Additional information you deem important (e.g. issue happens only occasionally):
The cause may be a change in vendor/github.com/containers/storage/utils.go:

     @@ -206,11 +206,10 @@ func getRootlessStorageOpts(rootlessUID int, systemOpts StoreOptions) (StoreOpti

          opts.RunRoot = rootlessRuntime
  -       opts.GraphRoot = filepath.Join(dataDir, "containers", "storage")

     if systemOpts.RootlessStoragePath != "" {
  -               opts.RootlessStoragePath = systemOpts.RootlessStoragePath
  +               opts.GraphRoot = systemOpts.RootlessStoragePath
        } else {
  -              opts.RootlessStoragePath = opts.GraphRoot
  +              opts.GraphRoot = filepath.Join(dataDir, "containers", "storage")
     }

    opts.GraphRoot, err = filepath.EvalSymlinks(opts.GraphRoot) // ----- HELPS ---

     if path, err := exec.LookPath("fuse-overlayfs"); err == nil {
             opts.GraphDriverName = "overlay"                                                                                                                                                                                                

The line marked with // ---- HELPS --- makes podman work again.

Output of podman version:

Version:      2.2.1
API Version:  2
Go Version:   go1.14.12
Built:        Sun Feb 21 23:51:35 2021
OS/Arch:      linux/amd64
if link was removed

[mheon]

What previous version did this work in?

Further, v2.2.1 is a very old version upstream - is there any chance you can try with v3.0.1 (shipping with RHEL 8.4.0) or, even better, the latest upstream version (v3.1.2)?

[gdoepp]

The last working version seems to have been v2.1.1-rhel. Newer versions up to v3.1.2 show the same error in centos8.3 using ext4 file systems. I have to add something like

diff --git a/vendor/github.com/containers/storage/types/options.go b/vendor/github.com/containers/storage/types/options.go
index fb80b18c5..39edc16e0 100644
--- a/vendor/github.com/containers/storage/types/options.go
+++ b/vendor/github.com/containers/storage/types/options.go
@@ -105,6 +105,9 @@ func defaultStoreOptionsIsolated(rootless bool, rootlessUID int, storageConf str
                }
                storageOpts.RootlessStoragePath = storagePath
        }
+        storageOpts.GraphRoot,err = filepath.EvalSymlinks(storageOpts.GraphRoot)
+        storageOpts.RootlessStoragePath,err = filepath.EvalSymlinks(storageOpts.RootlessStoragePath)
+        storageOpts.RunRoot,err = filepath.EvalSymlinks(storageOpts.RunRoot)
 
        return storageOpts, nil
 }

to make podman work there. I could not reproduce the issue with a btrfs backed storage directory on fedora 33 (perhaps other differences).
The original servers where we got the error are production servers and we do not control the updates. The podman update was classified as a security update. Our workaround is of course to resolve the links in storage.conf.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@giuseppe PTAL

[giuseppe]

how do your paths look like? Where is the symlink?

I guess it should be fine to resolve symlinks, we could do it as part of the expandEnvPath function

[gdoepp]

There was a symlink from ~/.local/share/containers/storage -> /some/big/filesystem/user/container-storage. I suppose this makes sense, but I wonder whether links at the level of the overlay* subdirectories should also work. For me it seems ok if everything below storage/ must reside on the same filesystem - it should be documented somewhere, though.

[rhatdan]

It would probably be better for you to bind mount the /some/big/filesystem/user/container-storage on ~/.local/share/containers/storage for now.

[gdoepp]

For now, the easier workaround was to resolve all links manually in storage.conf. This works "rootless". Please fix the bug in a future release or document the restriction and reject an erroneous configuration in storage.conf. A configuration that is not obviously wrong and once had worked should not stop to do so without notice.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.