storage.options.mount_program not set if driver overwritten by storage database

[Vogtinator]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Podman writes (meta)information about the storage into libpod/bolt_state.db, so that the right settings to use it are available.
In the case of running podman in a rootless environment with overlay as storage driver for that particular storage, it does not set mount_program automatically and it fails to set up

This can happen if e.g. /etc/containers/storage.conf is changed to use btrfs or vfs as default driver. After that, rootless overlay storage stops working with misleading error messages.

Steps to reproduce the issue:

# Create storage using overlay (fuse-overlayfs), or have some user with that already
fabian@localhost:~/podman> STORAGE_DRIVER=overlay ./bin/podman info | grep -A10 store
store:
  configFile: /home/fabian/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
# Set the global driver to something other than "overlay"
fabian@localhost:~/podman> grep ^driver /etc/containers/storage.conf 
driver = "btrfs"

Describe the results you received:

Podman fails to start entirely:

fabian@localhost:~/podman> ./bin/podman --log-level debug info
INFO[0000] ./bin/podman filtering at log level debug    
DEBU[0000] Called info.PersistentPreRunE(./bin/podman --log-level debug info) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/fabian/.local/share/containers/storage/libpod/bolt_state.db 
ERRO[0000] User-selected graph driver "btrfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/fabian/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/fabian/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/fabian/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is not supported 
Error: kernel does not support overlay fs: unable to create kernel-style whiteout: operation not permitted

Describe the results you expected:

Podman should continue using overlay like before. It suggests that User-selected graph driver "btrfs" overwritten by graph driver "overlay" from database, but it did not set mount_program to the autodetected /usr/bin/fuse-overlayfs value.

Additional information you deem important (e.g. issue happens only occasionally):

podman/libpod/runtime.go

Line 199 in f33b64d

storeOpts, err := storage.DefaultStoreOptions(rootless.IsRootless(), rootless.GetRootlessUID())

calls storage.DefaultStoreOptions, ending up in the function which sets the default value of mount_program:

podman/vendor/github.com/containers/storage/types/options.go

Lines 217 to 228 in f33b64d

	if driver := systemOpts.GraphDriverName; isRootlessDriver(driver) {
	opts.GraphDriverName = driver
	}
	if driver := os.Getenv("STORAGE_DRIVER"); driver != "" {
	opts.GraphDriverName = driver
	}
	if opts.GraphDriverName == overlay2 {
	logrus.Warnf("Switching default driver from overlay2 to the equivalent overlay driver.")
	opts.GraphDriverName = overlayDriver
	}
	if opts.GraphDriverName == "" || opts.GraphDriverName == overlayDriver {

Here it still uses the driver from the config instead of the actually used driver from the storage database, so it does not check for fuse-overlayfs. Switching to the correct driver for the storage happens later, in

podman/libpod/runtime.go

Line 223 in f33b64d

if err := makeRuntime(ctx, runtime); err != nil {

ending up in

podman/libpod/runtime.go

Lines 1050 to 1057 in f33b64d

	if !r.storageSet.GraphDriverNameSet && dbConfig.GraphDriver != "" {
	if r.storageConfig.GraphDriverName != dbConfig.GraphDriver &&
	r.storageConfig.GraphDriverName != "" {
	logrus.Errorf("User-selected graph driver %q overwritten by graph driver %q from database - delete libpod local files to resolve",
	r.storageConfig.GraphDriverName, dbConfig.GraphDriver)
	}
	r.storageConfig.GraphDriverName = dbConfig.GraphDriver
	}

This can be worked around by STORAGE_DRIVER=overlay or setting mount_program on the commandline, e.g. --storage-opt "overlay.mount_program=/usr/bin/fuse-overlayfs.

Output of podman version:

Client:       Podman Engine
Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.17.7
Git Commit:   4a242b1327fb34e6cac6c1686afb3370901180d3
Built:        Tue Mar  8 16:21:46 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

In the broken case podman refuses to start entirely. Providing output of podman info with the workaround applies would be misleading.

Package info (e.g. output of rpm -q podman or apt list podman):

Built from git.

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes.

Additional environment details (AWS, VirtualBox, physical, etc.):

VM running a kernel without userspace overlay mounts (5.3.18-150300.59.49-default)

[nalind]

When podman overrides the storage driver from the configuration file with a previously-used value from its database, it really needs to be supplying the options it used before. This is not specific to the overlay driver.