All podman functions hang after attempting to stop container

[pavinjosdev]

Issue Description

Once containers are started, podman works normally for a few hours.
But after a few hours, if I attempt to stop one of the containers, the action hangs and then podman itself hangs on subsequent commands like ps.

The issue is solved after deleting the empty files of format netns-{UUID} in /run/user/1000/netns/ directory.
Podman once again works normally for a few hours and the problem repeats.

Steps to reproduce the issue

1. Start container
2. Wait few hours
3. Try to stop container

Describe the results you received

pavin@suse-pc:~> podman --log-level debug stop trading
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called stop.PersistentPreRunE(podman --log-level debug stop trading) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using sqlite as database backend             
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/pavin/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/pavin/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/pavin/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 37             
DEBU[0000] Starting parallel job on container 8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f 
DEBU[0000] Stopping ctr 8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f (timeout 10) 
DEBU[0000] Stopping container 8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f (PID 7109) 
DEBU[0000] Sending signal 15 to container 8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f 
DEBU[0010] Timed out stopping container 8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f with SIGTERM, resorting to SIGKILL: given PID did not die within timeout 
WARN[0010] StopSignal SIGTERM failed to stop container trading in 10 seconds, resorting to SIGKILL 
DEBU[0010] Sending signal 9 to container 8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f 
DEBU[0010] Container "8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f" state changed from "stopping" to "stopped" while waiting for it to be stopped: discontinuing stop procedure as another process interfered 
DEBU[0010] Cleaning up container 8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f 
DEBU[0010] Tearing down network namespace at /run/user/1000/netns/netns-d2685668-e90c-ac8b-99d2-20649aaa0936 for container 8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f 
DEBU[0010] Netns /run/user/1000/netns/netns-d2685668-e90c-ac8b-99d2-20649aaa0936 still busy, try removing it again in 10ms 
DEBU[0010] Netns /run/user/1000/netns/netns-d2685668-e90c-ac8b-99d2-20649aaa0936 still busy, try removing it again in 10ms 
... (repeated many tens of thousands of times)

Describe the results you expected

Podman stops container without hanging

podman info output

pavin@suse-pc:~> podman info
host:
  arch: amd64
  buildahVersion: 1.37.5
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-1.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: unknown'
  cpuUtilization:
    idlePercent: 96.34
    systemPercent: 0.95
    userPercent: 2.72
  cpus: 12
  databaseBackend: sqlite
  distribution:
    distribution: opensuse-slowroll
    version: "20241002"
  eventLogger: journald
  freeLocks: 2002
  hostname: suse-pc
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.11.5-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 460140544
  memTotal: 13972418560
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-1.1.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.1.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.17-1.1.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.17
      commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-20240906.6b38f07-2.1.x86_64
    version: |
      pasta 20240906.6b38f07-2.1
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1.1.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: unknown
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 7324561408
  swapTotal: 8589930496
  uptime: 51h 16m 3.00s (Approximately 2.12 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /home/pavin/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/pavin/.local/share/containers/storage
  graphRootAllocated: 498681774080
  graphRootUsed: 88692174848
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/pavin/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.5
  Built: 1729756620
  BuiltTime: Thu Oct 24 13:27:00 2024
  GitCommit: ""
  GoVersion: go1.23.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.5

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

[Luap99]

Can you provide a fill strace -f from the podman stop process and upload the output. Also anything special about the container, are they using a userns?

Also can you reboot and see if it happens again after that?

[pavinjosdev]

Can you provide a fill strace -f from the podman stop process and upload the output.

@Luap99 Attached strace log. podman_stop.log

Also anything special about the container, are they using a userns?

It's a vanilla container running the debian 12 toolbx image.
I also have a distrobox and jellyfin container on this host that are also experiencing the same issue.

pavin@suse-pc:~> podman inspect trading 
[
     {
          "Id": "8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f",
          "Created": "2024-08-13T08:53:06.022591283+05:30",
          "Path": "/bin/bash",
          "Args": [
               "/bin/bash"
          ],
          "State": {
               "OciVersion": "1.2.0",
               "Status": "exited",
               "Running": false,
               "Paused": false,
               "Restarting": false,
               "OOMKilled": false,
               "Dead": false,
               "Pid": 0,
               "ExitCode": 137,
               "Error": "",
               "StartedAt": "2024-11-07T19:56:35.129723307+05:30",
               "FinishedAt": "2024-11-08T01:05:32.502584379+05:30",
               "CheckpointedAt": "0001-01-01T00:00:00Z",
               "RestoredAt": "0001-01-01T00:00:00Z",
               "StoppedByUser": true
          },
          "Image": "6c0d42546081348d4c4a1799421af6406105c21bc8f4afa5987624fb0a18a92a",
          "ImageDigest": "sha256:7ba014b872d0d779a851868ef737d0f916ca292130d50f4dc956e1f721389a76",
          "ImageName": "quay.io/toolbx-images/debian-toolbox:12",
          "Rootfs": "",
          "Pod": "",
          "ResolvConfPath": "/run/user/1000/containers/overlay-containers/8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f/userdata/resolv.conf",
          "HostnamePath": "/run/user/1000/containers/overlay-containers/8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f/userdata/hostname",
          "HostsPath": "/run/user/1000/containers/overlay-containers/8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f/userdata/hosts",
          "StaticDir": "/home/pavin/.local/share/containers/storage/overlay-containers/8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f/userdata",
          "OCIConfigPath": "/home/pavin/.local/share/containers/storage/overlay-containers/8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f/userdata/config.json",
          "OCIRuntime": "crun",
          "ConmonPidFile": "/run/user/1000/containers/overlay-containers/8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f/userdata/conmon.pid",
          "PidFile": "/run/user/1000/containers/overlay-containers/8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f/userdata/pidfile",
          "Name": "trading",
          "RestartCount": 0,
          "Driver": "overlay",
          "MountLabel": "",
          "ProcessLabel": "",
          "AppArmorProfile": "",
          "EffectiveCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "BoundingCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "ExecIDs": [],
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/home/pavin/.local/share/containers/storage/overlay/8fe8abec48af325f1921b44063b3f990d8ccd6a41f38f59e6c5a3fbc8e31e763/diff:/home/pavin/.local/share/containers/storage/overlay/f6faf32734e0870d82ea890737958fe33ce9ddfed27b3b157576d2aadbab3322/diff",
                    "UpperDir": "/home/pavin/.local/share/containers/storage/overlay/b53d9dd9aa22f62f2aaa3d54bb7678d7e0e3ee4b496b61ebf11760a7a768715d/diff",
                    "WorkDir": "/home/pavin/.local/share/containers/storage/overlay/b53d9dd9aa22f62f2aaa3d54bb7678d7e0e3ee4b496b61ebf11760a7a768715d/work"
               }
          },
          "Mounts": [],
          "Dependencies": [],
          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "",
               "IPAddress": "",
               "IPPrefixLen": 0,
               "IPv6Gateway": "",
               "GlobalIPv6Address": "",
               "GlobalIPv6PrefixLen": 0,
               "MacAddress": "",
               "Bridge": "",
               "SandboxID": "",
               "HairpinMode": false,
               "LinkLocalIPv6Address": "",
               "LinkLocalIPv6PrefixLen": 0,
               "Ports": {},
               "SandboxKey": "",
               "Networks": {
                    "pasta": {
                         "EndpointID": "",
                         "Gateway": "",
                         "IPAddress": "",
                         "IPPrefixLen": 0,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "",
                         "NetworkID": "pasta",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null
                    }
               }
          },
          "Namespace": "",
          "IsInfra": false,
          "IsService": false,
          "KubeExitCodePropagation": "invalid",
          "lockNumber": 39,
          "Config": {
               "Hostname": "8ae550f0ed4c",
               "Domainname": "",
               "User": "",
               "AttachStdin": false,
               "AttachStdout": false,
               "AttachStderr": false,
               "Tty": true,
               "OpenStdin": true,
               "StdinOnce": false,
               "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "container=podman",
                    "TERM=xterm",
                    "HOME=/root",
                    "HOSTNAME=8ae550f0ed4c"
               ],
               "Cmd": null,
               "Image": "quay.io/toolbx-images/debian-toolbox:12",
               "Volumes": null,
               "WorkingDir": "/",
               "Entrypoint": [
                    "/bin/bash"
               ],
               "OnBuild": null,
               "Labels": {
                    "com.github.containers.toolbox": "true",
                    "io.buildah.version": "1.23.1",
                    "maintainer": "",
                    "name": "debian-toolbox",
                    "summary": "Base image for creating Debian toolbox containers",
                    "usage": "This image is meant to be used with the toolbox command",
                    "version": "12"
               },
               "Annotations": {
                    "io.container.manager": "libpod",
                    "org.opencontainers.image.stopSignal": "15",
                    "org.systemd.property.KillSignal": "15",
                    "org.systemd.property.TimeoutStopUSec": "uint64 10000000"
               },
               "StopSignal": "SIGTERM",
               "HealthcheckOnFailureAction": "none",
               "CreateCommand": [
                    "podman",
                    "run",
                    "-it",
                    "--name",
                    "trading",
                    "--entrypoint=/bin/bash",
                    "6c0d42546081"
               ],
               "Umask": "0022",
               "Timeout": 0,
               "StopTimeout": 10,
               "Passwd": true,
               "sdNotifyMode": "container"
          },
          "HostConfig": {
               "Binds": [],
               "CgroupManager": "systemd",
               "CgroupMode": "private",
               "ContainerIDFile": "",
               "LogConfig": {
                    "Type": "journald",
                    "Config": null,
                    "Path": "",
                    "Tag": "",
                    "Size": "0B"
               },
               "NetworkMode": "pasta",
               "PortBindings": {},
               "RestartPolicy": {
                    "Name": "no",
                    "MaximumRetryCount": 0
               },
               "AutoRemove": false,
               "Annotations": {
                    "io.container.manager": "libpod",
                    "org.opencontainers.image.stopSignal": "15",
                    "org.systemd.property.KillSignal": "15",
                    "org.systemd.property.TimeoutStopUSec": "uint64 10000000"
               },
               "VolumeDriver": "",
               "VolumesFrom": null,
               "CapAdd": [],
               "CapDrop": [],
               "Dns": [],
               "DnsOptions": [],
               "DnsSearch": [],
               "ExtraHosts": [],
               "GroupAdd": [],
               "IpcMode": "shareable",
               "Cgroup": "",
               "Cgroups": "default",
               "Links": null,
               "OomScoreAdj": 0,
               "PidMode": "private",
               "Privileged": false,
               "PublishAllPorts": false,
               "ReadonlyRootfs": false,
               "SecurityOpt": [],
               "Tmpfs": {},
               "UTSMode": "private",
               "UsernsMode": "",
               "ShmSize": 65536000,
               "Runtime": "oci",
               "ConsoleSize": [
                    0,
                    0
               ],
               "Isolation": "",
               "CpuShares": 0,
               "Memory": 0,
               "NanoCpus": 0,
               "CgroupParent": "user.slice",
               "BlkioWeight": 0,
               "BlkioWeightDevice": null,
               "BlkioDeviceReadBps": null,
               "BlkioDeviceWriteBps": null,
               "BlkioDeviceReadIOps": null,
               "BlkioDeviceWriteIOps": null,
               "CpuPeriod": 0,
               "CpuQuota": 0,
               "CpuRealtimePeriod": 0,
               "CpuRealtimeRuntime": 0,
               "CpusetCpus": "",
               "CpusetMems": "",
               "Devices": [],
               "DiskQuota": 0,
               "KernelMemory": 0,
               "MemoryReservation": 0,
               "MemorySwap": 0,
               "MemorySwappiness": 0,
               "OomKillDisable": false,
               "PidsLimit": 2048,
               "Ulimits": [
                    {
                         "Name": "RLIMIT_NOFILE",
                         "Soft": 524288,
                         "Hard": 524288
                    },
                    {
                         "Name": "RLIMIT_NPROC",
                         "Soft": 52715,
                         "Hard": 52715
                    }
               ],
               "CpuCount": 0,
               "CpuPercent": 0,
               "IOMaximumIOps": 0,
               "IOMaximumBandwidth": 0,
               "CgroupConf": null
          }
     }
]

Also can you reboot and see if it happens again after that?

Yep, happens after reboot. It has been happening for a few weeks now, but I just rebooted, waited a few hours and tried to stop with the same hanging issue.

Thanks!

[Luap99]

Is that really the full strace on a stop command that hangs? I am really confused because the "Netns xxx still busy, try removing it again in 10ms" should log a unlinkat syscall for the netns path and I don't see any there. I also do not see the supposed umount call.

There has been a hang in the early 5.2.0,.1.,2. releases #24024. But AFAIK that has been fixed in 5.2.3 https://github.com/containers/podman/releases/tag/v5.2.3 and a reboot should make sure the mounts would be set correctly so it should be impossible to happen again.

Can you show the output of podman unshare findmnt -R / when the stop commands hang?

[pavinjosdev]

Is that really the full strace on a stop command that hangs? I am really confused because the "Netns xxx still busy, try removing it again in 10ms" should log a unlinkat syscall for the netns path and I don't see any there. I also do not see the supposed umount call.

It is the second time I called stop after the first one hanged.
If you want, I can try to catch the first one.

Can you show the output of podman unshare findmnt -R / when the stop commands hang?

Attached output after running stop now when container had 7 hours uptime which caused podman to hang. podman_unshare.log

[Luap99]

│ ├─/run/user/1000                                                                                                                               tmpfs                                                                          tmpfs           rw,nosuid,nodev,relatime,size=1364492k,nr_inodes=341123,mode=700,uid=1000,gid=1000,inode64
│ │ └─/run/user/1000/netns                                                                                                                       tmpfs[/netns]                                                                  tmpfs           rw,nosuid,nodev,relatime,size=1364492k,nr_inodes=341123,mode=700,uid=1000,gid=1000,inode64
│ │   └─/run/user/1000/netns/netns-129058fc-b928-6909-cc3f-5d2399a07817                                                                          nsfs[net:[4026533221]]                                                         nsfs            rw

The mount looks fine for this container so it is not the same issue as #24024 (which makes sense given the fix is already in your version)

It is the second time I called stop after the first one hanged.
If you want, I can try to catch the first one.

Yes please, although it should not matter really if we hang there we should still see the syscalls on the second command. Can you run the strace -f -o output.log -s 300 podman --log-level debug stop ... and provide the log. That should clearly log the printed debug lines in the loop so I should be able to see the syscall right before it.

[pavinjosdev]

@Luap99 New strace output as per your instructions (first time hang) attached. output.log

[Luap99]

1269491 umount2("/run/user/1000/netns/netns-7146277a-8237-d0fb-5c74-48c48b576237", MNT_DETACH <unfinished ...>
1269482 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
1269491 <... umount2 resumed>)          = -1 EINVAL (Invalid argument)

umount2 fails with EINVAL which generally means it is no a mount point so we then just try to remove the file:

1269491 unlinkat(AT_FDCWD, "/run/user/1000/netns/netns-7146277a-8237-d0fb-5c74-48c48b576237", 0) = -1 EBUSY (Device or resource busy)

But that fails because it is a mount point....

There are a few other reasons that say when umount fails with EINVAL, target is locked; see mount_namespaces(7). It is not quite clear to me what this means in our context. Reading the description it seem to be about the mount propagation part across namespaces but we should be in the same namespace where the mount was created so I don't see how any of this would be relevant here.

What is the output of podman unshare cat /proc/self/mountinfo?

@giuseppe Do you have ideas here?

[pavinjosdev]

What is the output of podman unshare cat /proc/self/mountinfo?

pavin@suse-pc:~> podman unshare cat /proc/self/mountinfo
1066 1065 0:44 /@/.snapshots/6488/snapshot / rw,noatime master:1 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=45792,subvol=/@/.snapshots/6488/snapshot
1067 1066 0:6 / /dev rw,nosuid master:2 - devtmpfs devtmpfs rw,size=4096k,nr_inodes=1697577,mode=755,inode64
1068 1067 0:25 / /dev/shm rw,nosuid,nodev master:3 - tmpfs tmpfs rw,inode64
1069 1067 0:26 / /dev/pts rw,nosuid,noexec,relatime master:4 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
1070 1067 0:21 / /dev/mqueue rw,nosuid,nodev,noexec,relatime master:18 - mqueue mqueue rw
1071 1067 0:34 / /dev/hugepages rw,nosuid,nodev,relatime master:19 - hugetlbfs hugetlbfs rw,pagesize=2M
1072 1066 0:24 / /sys rw,nosuid,nodev,noexec,relatime master:5 - sysfs sysfs rw
1073 1072 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:6 - securityfs securityfs rw
1074 1072 0:28 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime master:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
1075 1072 0:29 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:8 - pstore pstore rw
1076 1072 0:30 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime master:9 - efivarfs efivarfs rw
1077 1072 0:31 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime master:10 - bpf bpf rw,mode=700
1078 1072 0:8 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime master:17 - debugfs debugfs rw
1079 1078 0:13 / /sys/kernel/debug/tracing rw,nosuid,nodev,noexec,relatime master:566 - tracefs tracefs rw
1080 1072 0:13 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime master:20 - tracefs tracefs rw
1081 1072 0:37 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime master:23 - configfs configfs rw
1082 1072 0:45 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime master:24 - fusectl fusectl rw
1083 1066 0:23 / /proc rw,nosuid,nodev,noexec,relatime master:11 - proc proc rw
1084 1083 0:33 / /proc/sys/fs/binfmt_misc rw,relatime master:13 - autofs systemd-1 rw,fd=33,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6714
1085 1066 0:27 / /run rw,nosuid,nodev master:12 - tmpfs tmpfs rw,size=2729000k,nr_inodes=819200,mode=755,inode64
1086 1085 0:38 / /run/credentials/systemd-cryptsetup@cr_nvme\134x2deui.002538dc210238c2\134x2dpart3.service ro,nosuid,nodev,noexec,relatime,nosymfollow master:14 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,inode64,noswap
1087 1085 0:39 / /run/credentials/systemd-cryptsetup@internal_ssd.service ro,nosuid,nodev,noexec,relatime,nosymfollow master:15 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,inode64,noswap
1088 1085 0:35 / /run/credentials/systemd-journald.service ro,nosuid,nodev,noexec,relatime,nosymfollow master:21 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,inode64,noswap
1089 1085 0:36 / /run/credentials/systemd-udev-load-credentials.service ro,nosuid,nodev,noexec,relatime,nosymfollow master:22 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,inode64,noswap
1090 1085 0:47 / /run/credentials/systemd-tmpfiles-setup-dev-early.service ro,nosuid,nodev,noexec,relatime,nosymfollow master:26 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,inode64,noswap
1119 1085 0:48 / /run/credentials/systemd-sysctl.service ro,nosuid,nodev,noexec,relatime,nosymfollow master:27 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,inode64,noswap
1192 1085 0:49 / /run/credentials/systemd-sysusers.service ro,nosuid,nodev,noexec,relatime,nosymfollow master:28 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,inode64,noswap
1311 1085 0:50 / /run/credentials/systemd-tmpfiles-setup-dev.service ro,nosuid,nodev,noexec,relatime,nosymfollow master:29 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,inode64,noswap
1312 1085 0:32 / /run/credentials/systemd-vconsole-setup.service ro,nosuid,nodev,noexec,relatime,nosymfollow master:76 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,inode64,noswap
1313 1085 0:63 / /run/credentials/systemd-tmpfiles-setup.service ro,nosuid,nodev,noexec,relatime,nosymfollow master:80 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,inode64,noswap
1314 1085 0:136 / /run/user/1000 rw,nosuid,nodev,relatime master:889 - tmpfs tmpfs rw,size=1364496k,nr_inodes=341124,mode=700,uid=1000,gid=1000,inode64
1315 1314 0:137 / /run/user/1000/gvfs rw,nosuid,nodev,relatime master:963 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
1316 1314 0:148 / /run/user/1000/doc rw,nosuid,nodev,relatime master:1277 - fuse.portal portal rw,user_id=1000,group_id=1000
1317 1085 0:147 / /run/credentials/getty@tty1.service ro,nosuid,nodev,noexec,relatime,nosymfollow master:1202 - tmpfs tmpfs rw,size=1024k,nr_inodes=1024,mode=700,inode64,noswap
1318 1066 0:46 / /tmp rw,nosuid,nodev master:25 - tmpfs tmpfs rw,size=6822500k,nr_inodes=1048576,inode64
1319 1318 0:57 / /tmp/net_cls rw,relatime master:63 - cgroup net_cls rw,net_cls
1320 1066 0:42 /@/.snapshots /.snapshots rw,noatime master:52 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=265,subvol=/@/.snapshots
1321 1066 0:51 /@/boot/grub2/i386-pc /boot/grub2/i386-pc rw,relatime master:55 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=264,subvol=/@/boot/grub2/i386-pc
1322 1066 0:52 /@/boot/grub2/x86_64-efi /boot/grub2/x86_64-efi rw,relatime master:57 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=263,subvol=/@/boot/grub2/x86_64-efi
1323 1066 0:53 /@/home /home rw,noatime master:59 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=270,subvol=/@/home
1324 1066 0:54 /@/root /root rw,noatime master:61 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=272,subvol=/@/root
1325 1066 0:55 /@/opt /opt rw,noatime master:65 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=271,subvol=/@/opt
1326 1066 0:56 /@/srv /srv rw,noatime master:67 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=273,subvol=/@/srv
1327 1066 0:59 /@/var /var rw,noatime master:69 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=274,subvol=/@/var
1328 1327 259:2 / /var/crash rw,relatime master:16 - ext4 /dev/nvme0n1p2 rw,stripe=32,data=ordered
1329 1066 0:58 /@/usr/local /usr/local rw,noatime master:71 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=275,subvol=/@/usr/local
1330 1066 259:1 / /boot/efi rw,relatime master:73 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro
1331 1066 0:62 /@ /internal_ssd rw,noatime master:78 - btrfs /dev/mapper/lvmspare-ssd0 rw,compress=zstd:3,ssd,space_cache=v2,subvolid=256,subvol=/@
1332 1066 8:48 / /usbraid rw,relatime master:830 - bcachefs /dev/sdd:/dev/sdc:/dev/sdb:/dev/sdf:/dev/sdh:/dev/sde:/dev/sdg rw,metadata_replicas=2,data_replicas=2,compression=lz4,metadata_target=ssd,foreground_target=ssd,background_target=hdd,promote_target=ssd
1333 1323 0:53 /@/home/pavin/.local/share/containers/storage/overlay /home/pavin/.local/share/containers/storage/overlay rw,noatime - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=270,subvol=/@/home
1343 1333 0:150 / /home/pavin/.local/share/containers/storage/overlay/34ae9fe1074025da11cbd92d1abbaa7d5d3bfd0527db652b997bf402305a352c/merged rw,relatime - overlay overlay rw,lowerdir=/home/pavin/.local/share/containers/storage/overlay/l/NC5T7YVNKX6XKZBOBG7EI76AZM:/home/pavin/.local/share/containers/storage/overlay/l/NC5T7YVNKX6XKZBOBG7EI76AZM/../diff1:/home/pavin/.local/share/containers/storage/overlay/l/AX3KU77J6KBM4AQOLN3XXZ7TI6:/home/pavin/.local/share/containers/storage/overlay/l/4I7BXYWT2BJ2AQJPA36OA5GSHS:/home/pavin/.local/share/containers/storage/overlay/l/REZ65XBPEJXWHFHBEYGDI2UD4V:/home/pavin/.local/share/containers/storage/overlay/l/EWK6WGBO66OX6Z6XALPVM633SQ:/home/pavin/.local/share/containers/storage/overlay/l/YL3NFZYEGQ3HTMWJ54UHDHOXQQ:/home/pavin/.local/share/containers/storage/overlay/l/WO5THEXDPT6TVLRN33T7B67LXH,upperdir=/home/pavin/.local/share/containers/storage/overlay/34ae9fe1074025da11cbd92d1abbaa7d5d3bfd0527db652b997bf402305a352c/diff,workdir=/home/pavin/.local/share/containers/storage/overlay/34ae9fe1074025da11cbd92d1abbaa7d5d3bfd0527db652b997bf402305a352c/work,userxattr
1354 1314 0:136 /netns /run/user/1000/netns rw,nosuid,nodev,relatime shared:1298 master:889 - tmpfs tmpfs rw,size=1364496k,nr_inodes=341124,mode=700,uid=1000,gid=1000,inode64
1355 1354 0:4 net:[4026533023] /run/user/1000/netns/netns-5de84b11-70f7-587f-f763-46df6c0211f0 rw shared:1299 - nsfs nsfs rw
1357 1323 0:156 / /home/pavin/.local/share/containers/storage/overlay-containers/60abf3b45492f294138875f593c1e745c6be521949346e43234588e1b05f66c4/userdata/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=64000k,uid=1000,gid=1000,inode64
1375 1333 0:157 / /home/pavin/.local/share/containers/storage/overlay/d9b1d397e70bd25b9f125694207f2d741ba6d013c80ca2919e988ca54e7fe9e4/merged rw,relatime - overlay overlay rw,lowerdir=/home/pavin/.local/share/containers/storage/overlay/l/MHFJSRU3FS4KDPFGD46STOFK4G:/home/pavin/.local/share/containers/storage/overlay/l/MHFJSRU3FS4KDPFGD46STOFK4G/../diff1:/home/pavin/.local/share/containers/storage/overlay/l/ZX3HXNOHOUUZ45Z4IH5ZCKPCJ2:/home/pavin/.local/share/containers/storage/overlay/l/XW7DZ2LGCTJSPQTAF3O563EXPW:/home/pavin/.local/share/containers/storage/overlay/l/GAH7XOI4UW3ER7LN574MEWBPCE:/home/pavin/.local/share/containers/storage/overlay/l/MX5BBNZUV2UF2ZOKZGRZ5MHE33:/home/pavin/.local/share/containers/storage/overlay/l/7ZGINY6TCGK5GGBX2S4PUW47OY:/home/pavin/.local/share/containers/storage/overlay/l/YKDQTTYGNPBT6NQUUNUCYDZUWK:/home/pavin/.local/share/containers/storage/overlay/l/NJMQUL4YYPUX2IDKATKAGOTLIA:/home/pavin/.local/share/containers/storage/overlay/l/AAXZYOM74KNTZZ4SA65HIBLBMA,upperdir=/home/pavin/.local/share/containers/storage/overlay/d9b1d397e70bd25b9f125694207f2d741ba6d013c80ca2919e988ca54e7fe9e4/diff,workdir=/home/pavin/.local/share/containers/storage/overlay/d9b1d397e70bd25b9f125694207f2d741ba6d013c80ca2919e988ca54e7fe9e4/work,userxattr
1416 1354 0:4 net:[4026533090] /run/user/1000/netns/netns-5e087757-9fba-54b9-dc77-e915deedcae1 rw shared:1624 - nsfs nsfs rw
109 1323 0:457 / /home/pavin/.local/share/containers/storage/overlay-containers/8ae550f0ed4cd9de128227e96eb0661d0af4b672cae41c7cb9becee96b56a53f/userdata/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=64000k,uid=1000,gid=1000,inode64
323 1354 0:4 net:[4026533235] /run/user/1000/netns/netns-b3e8b10c-0d19-335e-07b2-fb91c8c4378c rw shared:512 - nsfs nsfs rw
327 1333 0:468 / /home/pavin/.local/share/containers/storage/overlay/b53d9dd9aa22f62f2aaa3d54bb7678d7e0e3ee4b496b61ebf11760a7a768715d/merged rw,relatime - overlay overlay rw,lowerdir=/home/pavin/.local/share/containers/storage/overlay/l/5XPOSDBIGCP5YJHVYWOE5PFHJJ:/home/pavin/.local/share/containers/storage/overlay/l/YB3XIAIZGD4RIWASY2ZGACYKFS,upperdir=/home/pavin/.local/share/containers/storage/overlay/b53d9dd9aa22f62f2aaa3d54bb7678d7e0e3ee4b496b61ebf11760a7a768715d/diff,workdir=/home/pavin/.local/share/containers/storage/overlay/b53d9dd9aa22f62f2aaa3d54bb7678d7e0e3ee4b496b61ebf11760a7a768715d/work,userxattr
343 1066 0:291 /@/.snapshots/6515/snapshot/usr /usr rw,noatime master:52 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=45974,subvol=/@/.snapshots/6515/snapshot
1284 1066 0:291 /@/.snapshots/6515/snapshot/etc /etc rw,noatime master:52 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=45974,subvol=/@/.snapshots/6515/snapshot
2095 1066 0:291 /@/.snapshots/6515/snapshot/boot /boot rw,noatime master:52 - btrfs /dev/mapper/suse-system rw,compress=zstd:3,ssd,space_cache=v2,subvolid=45974,subvol=/@/.snapshots/6515/snapshot

[github-actions]

A friendly reminder that this issue had no activity for 30 days.