podman does not read /usr/share/containers/storage.conf as documented

[sysrich]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

As discussed on IRC, basically podman does not read /usr/share/containers/storage.conf whereas

https://github.com/containers/storage/blob/main/docs/containers-storage.conf.5.md#files says it should

Steps to reproduce the issue:

1. Have a system that historically has used multiple storage drivers (eg btrfs and overlayfs)
2. Configure /etc/containers/storage.conf to use one storage driver (eg btrfs)
3. Run podman containers ls - confirm podman works
4. Move /etc/containers/storage.conf to /usr/share/containers/storage.conf
5. Run podman containers ls - podman errors with "Error: /var/lib/containers/storage contains several valid graphdrivers: btrfs, overlay; Please cleanup or explicitly choose storage driver (-s )"

Describe the results you expected:

podman should read the storage.conf in /usr/share/containers

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.2.3
API Version:  3.2.3
Go Version:   go1.13.15
Built:        Sat Jul 17 02:00:00 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.3
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-1.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: unknown'
  cpus: 8
  distribution:
    distribution: '"opensuse-tumbleweed"'
    version: "20210906"
  eventLogger: journald
  hostname: iwreckit.suse.de
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.14.0-1-default
  linkmode: dynamic
  memFree: 6204567552
  memTotal: 33495945216
  ociRuntime:
    name: runc
    package: runc-1.0.2-1.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2
      spec: 1.0.2-dev
      go: go1.13.15
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 34359734272
  swapTotal: 34359734272
  uptime: 29h 27m 7.17s (Approximately 1.21 days)
registries:
  search:
  - registry.opensuse.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Build Version: 'Btrfs v5.13 '
    Library Version: "102"
  imageStore:
    number: 2
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.2.3
  Built: 1626480000
  BuiltTime: Sat Jul 17 02:00:00 2021
  GitCommit: ""
  GoVersion: go1.13.15
  OsArch: linux/amd64
  Version: 3.2.3

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.3-1.1.x86_64```

**Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)**

Yes

**Additional environment details (AWS, VirtualBox, physical, etc.):**

[PavelSosin-320]

Me too! #Fedora34 Desktop podman issue - possible cause
Especially, when podman is ran from the systemd unit or via systemd-run. All possible environment variables have been imported into systemd environment.

[PavelSosin-320]

It breaks the development branch again: podman info attempts to ceate something in the run/containers/storage that is impossible for the rootless user. Resolution of the storage root for the rootless users via $HOME is impossible too because HOME env variable inside systemd unit environment is not promissed unless included into unit or environment file.
After creation of my conteiner from the Session environment using podman container create I see the desired mount point inside the container $HOME/.local/share/containers/storage/btrfs/subvolums/ but when running "container create" or "run" from the systemd unit the HOME XDG_CONFIG_HOME, and XDG_DATA_HOME env variables don't exist because they are defined by the pam as a part of the session environment during login.
In the case of systemd unit generation for running by the rootless use started by systemd user manager the environment necessary for container execution as a service has to be not used or generated statically. Sorry if I don't understand storage.conf correctly.

[rhatdan]

@PavelSosin-320 This Bug was about reading /usr/share/containers/storage.conf, which upstream Podman should now do. (Or will do as soon as containers/storage is updated.
I think you are reporting a different issue.

[PavelSosin-320]

I referenced this issue because I see that both may have the same rooy cause: storage configuration plays with interactive session's environment variables too freely. It is enaugh to have some part of session environment like HOME, CONTAINER_*, etc to disappear from the environment to get storage configuration to be skipped or become non-applicable for a rootless user. systemd-run podman info with --set-env HOME and without it produce different output and in the last case no storage configuration is detected. For example, on desktop systemd-based Linux distro location of the configuration files follows certain rules: HOME, XDG_CONFIG_HOME are set by certain environment generator and known inside interactive user session. If somethings runs as a service without systemd generator and per-user environment file nothing guaranties that all parts of environment used by application exist and all configuration files can be found. Any user can create symbolic link to the configuration file located in /usr/lshare... to share configuration but but this shared file can't recurcievly use interactive session environment. rootless_user_storage... defined in the /usr/share/containers and defined via HOME can be interpreted only if HOME itself is known and correct. Non-interactive user may not have a HOME if it was not created using homectl. $> homectl output is "No home area!"
From the systemd point-of-view it is much easer to create unit config file usinng systemctl --user edit unit-file. to add something user-specific to the systemd service unit, like environment variables and put configuration to the proper place.