`podman login` writes no content into /tmp/

[tobwen]

/kind bug

Description
While Podman correctly detects systemd and uses the default XDG_RUNTIME_DIR, some parts of podman login seems to have a small bug.

Steps to reproduce the issue:

sudo -u podman podman --log-level debug login

INFO[0000] podman filtering at log level debug
DEBU[0000] Called login.PersistentPreRunE(podman --log-level debug login)
DEBU[0000] Using conmon: "/usr/local/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1001/containers
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1001/libpod/tmp
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
DEBU[0000] Initializing event backend journald
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Using OCI runtime "/usr/local/bin/crun"
INFO[0000] Found CNI network podman (type=bridge) at /home/podman/.config/cni/net.d/87-podman.conflist
INFO[0000] Found CNI network tmp_default (type=bridge) at /home/podman/.config/cni/net.d/tmp_default.conflist
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 25
INFO[0000] podman filtering at log level debug
DEBU[0000] Called login.PersistentPreRunE(podman --log-level debug login)
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] Using conmon: "/usr/local/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1001/containers
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1001/libpod/tmp
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] overlay test mount indicated that metacopy is not being used
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/local/bin/crun"
INFO[0000] Found CNI network podman (type=bridge) at /home/podman/.config/cni/net.d/87-podman.conflist
INFO[0000] Found CNI network tmp_default (type=bridge) at /home/podman/.config/cni/net.d/tmp_default.conflist
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 25
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] registry not specified, default to the first registry "docker.io" from registries.conf
DEBU[0000] No credentials for docker.io found
Username: ############
Password:
DEBU[0022] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io
DEBU[0022] GET https://registry-1.docker.io/v2/
DEBU[0022] Ping https://registry-1.docker.io/v2/ status 401
DEBU[0022] GET https://auth.docker.io/token?account=############&service=registry.docker.io
DEBU[0022] GET https://registry-1.docker.io/v2/
DEBU[0023] Stored credentials for docker.io in credential helper containers-auth.json
Login Succeeded!
DEBU[0023] Called login.PersistentPostRunE(podman --log-level debug login)

As you can see, the directory is empty. The correct content is in XDG_RUNTIME_DIR

ls -al /tmp/containers-user-1001/containers/

total 8
drwx------ 2 podman podman 4096 Jun 25 11:47 .
drwx------ 3 podman podman 4096 Jun 25 11:47 ..

sudo -u podman printenv

As expected, XDG_RUNTIME_DIR hasn't been set without sudo --login

LANG=en_US.UTF-8
TERM=xterm-256color
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAIL=/var/mail/podman
LOGNAME=podman
USER=podman
HOME=/home/podman
SHELL=/usr/sbin/nologin
SUDO_COMMAND=/usr/bin/printenv
SUDO_USER=root
SUDO_UID=0
SUDO_GID=0

machinectl shell podman@.host /usr/bin/printenv

LANG=en_US.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/home/podman
LOGNAME=podman
USER=podman
INVOCATION_ID=ad6acc5d464d4c539ff483e0ce88db73
TERM=xterm-256color
MAIL=/var/mail/podman
XDG_SESSION_ID=23
XDG_RUNTIME_DIR=/run/user/1001
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1001/bus
XDG_SESSION_TYPE=tty
XDG_SESSION_CLASS=user

When using machinectl (or sudo --login), XDG_RUNTIME_DIR gets set and no directory is created in tmp

rm -rf /tmp/containers-user-1001/
machinectl shell podman@.host /usr/local/bin/podman --log-level debug login

Describe the results you received:
Added to the steps above for better assignment.

Describe the results you expected:
Since the default XDG_RUNTIME_DIR gets used, there shouldn't be an empty directory-structure in /tmp/

Additional information you deem important (e.g. issue happens only occasionally):
I'm aware that the variable XDG_RUNTIME_DIR is mandatory but seems like something changed in Podman a few versions ago. It seems like Podman automatically chooses the default XDG_RUNTIME_DIR when it can communicate with systemd.

It seems like at least podman login doesn't has been updated. It creates the directories in /tmp/, but without content. The credentials are written at the right place: below /run/user/1001/.

systemd's user-linger has been activated of course. I am aware that you normally need machinectl (suggested by systemd-community) or sudo --login to get a login environment (not possible if it's a system user), Podman seems to auto-detect XDG_RUNTIME_DIR now.

Output of podman version:

Version:      3.3.0-dev
API Version:  3.3.0-dev
Go Version:   go1.16.5
Git Commit:   ed511d27d7d2198624dcda41b9e06e9e582e5c20-dirty
Built:        Tue Jun 22 16:27:53 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.1
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/local/bin/conmon
    version: 'conmon version 2.0.30-dev, commit: '
  cpus: 8
  distribution:
    distribution: debian
    version: "11"
  eventLogger: journald
  hostname: debian
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.13.0-051300rc6-generic
  linkmode: dynamic
  memFree: 7625961472
  memTotal: 8336752640
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/local/bin/crun
    version: |-
      crun version 0.20.1.6-60de
      commit: 7ef74c9330033cb884507c28fd8c267861486633
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/local/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.11+dev
      commit: df67b2f1220e2ede78110c195410fc34c348202d
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 1022357504
  swapTotal: 1022357504
  uptime: 12h 24m 7.25s (Approximately 0.50 days)
registries:
  search:
  - docker.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /run/user/1001/containers
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.0-dev
  Built: 1624379273
  BuiltTime: Tue Jun 22 16:27:53 2021
  GitCommit: ed511d27d7d2198624dcda41b9e06e9e582e5c20-dirty
  GoVersion: go1.16.5
  OsArch: linux/amd64
  Version: 3.3.0-dev

Package info (e.g. output of rpm -q podman or apt list podman):

n/a - self compiled

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
FrankenDebian: Debian Bullseye RC1 on Ubuntu Mainline Kernel 5.13.0-051300rc6-generic on VirtualBox

[rhatdan]

I really do not understand what the bug is, Are you saying the podman is creating /tmp/containers-user-1001/containers/ when even when XDG_RUNTIME_DIR is defined?

[tobwen]

No, but since some builds (I cannot tell the exact commit), podman doesn't complain about a missing XDG_RUNTIME_DIR anymore.

If XDG_RUNTIME_DIR isn't set, podman uses the default location (/run/user/1001/) and writes its stuff into. But in this case, /tmp/containers-user-1001/containers/ gets created without any content and doesn't seem to be used at all.

[rhatdan]

Ok, I would guess that there is different code handling different parts of the system, and one looks to see if a writable /run/user/* exists if XDG_RUNTIME_DIR does not exist and another piece just goes write to /tmp/containers-user-1001/containers/
Can you do a podman info on the container and which paths are in there? I think containers/storage is creating the content in /tmp.

[tobwen]

podman info is in my first post, if you mean inspect on a running container, here we go:

[
    {
        "Id": "064eb79b761f04507d7dc53e5f87e76a565dff1fa04698e9efd4cd08ac165668",
        "Created": "2021-06-25T12:45:29.825231474Z",
        "Path": "docker-entrypoint.sh",
        "Args": [
            "bash"
        ],
        "State": {
            "OciVersion": "1.0.2-dev",
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 4523,
            "ConmonPid": 4520,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-06-25T12:45:29.976943574Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Healthcheck": {
                "Status": "",
                "FailingStreak": 0,
                "Log": null
            }
        },
        "Image": "69c62e13fa75d3439ba7bd5bb25d323c1d33acdf140140ea65a4a76697d0fc28",
        "ImageName": "docker.io/library/bash:latest",
        "Rootfs": "",
        "Pod": "",
        "ResolvConfPath": "/run/user/1001/containers/overlay-containers/064eb79b761f04507d7dc53e5f87e76a565dff1fa04698e9efd4cd08ac165668/userdata/resolv.conf",
        "HostnamePath": "/run/user/1001/containers/overlay-containers/064eb79b761f04507d7dc53e5f87e76a565dff1fa04698e9efd4cd08ac165668/userdata/hostname",
        "HostsPath": "/run/user/1001/containers/overlay-containers/064eb79b761f04507d7dc53e5f87e76a565dff1fa04698e9efd4cd08ac165668/userdata/hosts",
        "StaticDir": "/home/podman/.local/share/containers/storage/overlay-containers/064eb79b761f04507d7dc53e5f87e76a565dff1fa04698e9efd4cd08ac165668/userdata",
        "OCIConfigPath": "/home/podman/.local/share/containers/storage/overlay-containers/064eb79b761f04507d7dc53e5f87e76a565dff1fa04698e9efd4cd08ac165668/userdata/config.json",
        "OCIRuntime": "crun",
        "ConmonPidFile": "/run/user/1001/containers/overlay-containers/064eb79b761f04507d7dc53e5f87e76a565dff1fa04698e9efd4cd08ac165668/userdata/conmon.pid",
        "PidFile": "/run/user/1001/containers/overlay-containers/064eb79b761f04507d7dc53e5f87e76a565dff1fa04698e9efd4cd08ac165668/userdata/pidfile",
        "Name": "vigilant_wilbur",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "EffectiveCaps": [
            "CAP_AUDIT_WRITE",
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_MKNOD",
            "CAP_NET_BIND_SERVICE",
            "CAP_NET_RAW",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT"
        ],
        "BoundingCaps": [
            "CAP_AUDIT_WRITE",
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_MKNOD",
            "CAP_NET_BIND_SERVICE",
            "CAP_NET_RAW",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT"
        ],
        "ExecIDs": [],
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/home/podman/.local/share/containers/storage/overlay/690a1954900b4094e9b4ca8c79d36b816205f2f97ce37b89b812a241cedc1982/diff:/home/podman/.local/share/containers/storage/overlay/487eccbc5babaffcbbe6dd63d4fc45ebe18daabbcdcd7e115a97c72116318b43/diff:/home/podman/.local/share/containers/storage/overlay/32f366d666a541852cad754ee1cdb53a736110b550f0c2d5a46bc5ba519896b6/diff",
                "MergedDir": "/home/podman/.local/share/containers/storage/overlay/8b838806480657e3ca6a0e75e29f6b3e39c06f9a8b4858bf13146f3eed5f0de8/merged",
                "UpperDir": "/home/podman/.local/share/containers/storage/overlay/8b838806480657e3ca6a0e75e29f6b3e39c06f9a8b4858bf13146f3eed5f0de8/diff",
                "WorkDir": "/home/podman/.local/share/containers/storage/overlay/8b838806480657e3ca6a0e75e29f6b3e39c06f9a8b4858bf13146f3eed5f0de8/work"
            }
        },
        "Mounts": [],
        "Dependencies": [],
        "NetworkSettings": {
            "EndpointID": "",
            "Gateway": "",
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "",
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/run/user/1001/netns/cni-f94d0149-d8a7-e009-11d6-331bb76484ec"
        },
        "ExitCommand": [
            "/usr/local/stow/podman-20210622/bin/podman",
            "--root",
            "/home/podman/.local/share/containers/storage",
            "--runroot",
            "/run/user/1001/containers",
            "--log-level",
            "debug",
            "--cgroup-manager",
            "systemd",
            "--tmpdir",
            "/run/user/1001/libpod/tmp",
            "--runtime",
            "crun",
            "--storage-driver",
            "overlay",
            "--events-backend",
            "journald",
            "--syslog",
            "container",
            "cleanup",
            "--rm",
            "064eb79b761f04507d7dc53e5f87e76a565dff1fa04698e9efd4cd08ac165668"
        ],
        "Namespace": "",
        "IsInfra": false,
        "Config": {
            "Hostname": "064eb79b761f",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "container=podman",
                "_BASH_LATEST_PATCH=8",
                "_BASH_VERSION=5.1.8",
                "_BASH_BASELINE=5.1.8",
                "_BASH_BASELINE_PATCH=8",
                "HOME=/root",
                "HOSTNAME=064eb79b761f"
            ],
            "Cmd": [
                "bash"
            ],
            "Image": "docker.io/library/bash:latest",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": "docker-entrypoint.sh",
            "OnBuild": null,
            "Labels": null,
            "Annotations": {
                "io.container.manager": "libpod",
                "io.kubernetes.cri-o.Created": "2021-06-25T12:45:29.825231474Z",
                "io.kubernetes.cri-o.TTY": "true",
                "io.podman.annotations.autoremove": "TRUE",
                "io.podman.annotations.init": "FALSE",
                "io.podman.annotations.privileged": "FALSE",
                "io.podman.annotations.publish-all": "FALSE",
                "org.opencontainers.image.stopSignal": "15"
            },
            "StopSignal": 15,
            "CreateCommand": [
                "podman",
                "--log-level",
                "debug",
                "run",
                "-it",
                "--rm",
                "-d",
                "bash"
            ],
            "Umask": "0022",
            "Timeout": 0,
            "StopTimeout": 10
        },
        "HostConfig": {
            "Binds": [],
            "CgroupManager": "systemd",
            "CgroupMode": "private",
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "k8s-file",
                "Config": null,
                "Path": "/home/podman/.local/share/containers/storage/overlay-containers/064eb79b761f04507d7dc53e5f87e76a565dff1fa04698e9efd4cd08ac165668/userdata/ctr.log",
                "Tag": "",
                "Size": "0B"
            },
            "NetworkMode": "slirp4netns",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": true,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [],
            "CapDrop": [],
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": [],
            "IpcMode": "private",
            "Cgroup": "",
            "Cgroups": "default",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "private",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [],
            "Tmpfs": {},
            "UTSMode": "private",
            "UsernsMode": "",
            "ShmSize": 65536000,
            "Runtime": "oci",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "user.slice",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": 0,
            "OomKillDisable": false,
            "PidsLimit": 2048,
            "Ulimits": [],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "CgroupConf": null
        }
    }
]

[rhatdan]

I have to revert fix containers/storage#952

Was this happening on a non systemd system?

[mtrmac]

I have to revert fix containers/storage#952

I mean, understanding both this and containers/storage#568 is the fulcrum. It’s not yet obvious to me that the revert is definite.

I’ve been fairly vocal in #568 about not caring about an unnamed vaguely-specified environment that motivated that PR. Maybe it doesn’t even exist any more. Maybe this systemd detection code is buggy. Maybe the two issues are an irreconcilable difference in fairly rare environments; maybe it’s the case and we still have a clear reason to prioritize one or the other.

Whatever happens, let’s please make sure we have a written record of why we have decided for one design or another. In one more year or a few more, it’s pretty likely that these tradeoffs will again be questioned by whatever unfortunate soul is tasked with maintaining this fallback logic; so let’s preserve our hard-earned knowledge instead of rediscovering the tradeoffs and design constraints every year, and starting from the questionable scratch over and over.

Was this happening on a non systemd system?

Assuming this is systemd as mentioned above, please capture, from the same environment that starts Podman:

• Full metadata of /proc/1/comm , including size, ownership, permissions, SELinux context, any other security-relevant attributes (AppArmor? something else?)
• Exact contents of /proc/1/comm
• Maybe strace of Podman’s system calls related to that file could help.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.