Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Podman fails to start containers due to RLIMIT_NPROC being too high #18555

Closed
andrew-kennedy opened this issue May 13, 2023 · 7 comments · Fixed by #18733
Closed

Podman fails to start containers due to RLIMIT_NPROC being too high #18555

andrew-kennedy opened this issue May 13, 2023 · 7 comments · Fixed by #18733
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@andrew-kennedy
Copy link

andrew-kennedy commented May 13, 2023 •
edited

Issue Description

I just updated from fedora 37 -> 38, and a few containers that previously worked started breaking, but only ones I created within the last month or so. Here's an example of a kubernetes yaml file for one of the breaking containers:

kube yaml for pihole 
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.5.0

# NOTE: If you generated this yaml from an unprivileged and rootless podman container on an SELinux
# enabled system, check the podman generate kube man page for steps to follow to ensure that your pod/container
# has the right permissions to access the volumes added.
---
apiVersion: v1
kind: Pod
metadata:
  annotations:
    io.podman.annotations.ulimit: nofile=524288:524288,nproc=256639:256639
    io.containers.autoupdate/pihole-container: registry
  creationTimestamp: "2023-05-12T09:22:09Z"
  labels:
    app: pihole
  name: pihole
spec:
  securityContext:
    seLinuxOptions:
      type: spc_t
  containers:
  - env:
    - name: TZ
      value: America/Los_Angeles
    - name: WEBPASSWORD
      value: testpassword
    image: docker.io/pihole/pihole:latest
    name: pihole-container
    ports:
    - containerPort: 53
      hostPort: 5553
    - containerPort: 53
      hostPort: 5553
      protocol: UDP
    volumeMounts:
    - mountPath: /etc/pihole
      name: pihole_piholeetc-pvc
    - mountPath: /etc/dnsmasq.d
      name: pihole_piholednsmasq-pvc
  volumes:
  - name: pihole_piholeetc-pvc
    persistentVolumeClaim:
      claimName: pihole_piholeetc
  - name: pihole_piholednsmasq-pvc
    persistentVolumeClaim:
      claimName: pihole_piholednsmasq

When instantiated with systemctl --user restart podman-kube@$(systemd-escape pihole.yml), this creates an image with the following podman inspect output:

`podman inspect` output for crashing pihole container 
[
     {
          "Id": "33396f41786f6d3b360a23f8d256869fca2a7054eabb21a194d356eee242bb2d",
          "Created": "2023-05-12T07:11:59.371991766-07:00",
          "Path": "/s6-init",
          "Args": [
               "/s6-init"
          ],
          "State": {
               "OciVersion": "1.1.0-rc.1",
               "Status": "created",
               "Running": false,
               "Paused": false,
               "Restarting": false,
               "OOMKilled": false,
               "Dead": false,
               "Pid": 0,
               "ExitCode": 0,
               "Error": "can only stop created or running containers. 33396f41786f6d3b360a23f8d256869fca2a7054eabb21a194d356eee242bb2d is in state created: container state improper",
               "StartedAt": "0001-01-01T00:00:00Z",
               "FinishedAt": "0001-01-01T00:00:00Z",
               "Health": {
                    "Status": "",
                    "FailingStreak": 0,
                    "Log": null
               },
               "CheckpointedAt": "0001-01-01T00:00:00Z",
               "RestoredAt": "0001-01-01T00:00:00Z"
          },
          "Image": "4d6ef5c6684a8f5ce05e684dd1b92d75dfb1bbfeec2b8cf9a475fdbe71de2f07",
          "ImageDigest": "sha256:a74dde4800f54d3c0b0839babbac9f2cc7e4b8239ab4a5bc2c25c7328ec1c019",
          "ImageName": "docker.io/pihole/pihole:latest",
          "Rootfs": "",
          "Pod": "65c551f405af97b3615b7624457b53e46f8fb143f9e50df62f3de5faeca96337",
          "ResolvConfPath": "/run/user/1000/containers/overlay-containers/9a048fef4e4d3608355576f7fac466606e87e51b03121e5a4fc5a6d0106d9fca/userdata/resolv.conf",
          "HostnamePath": "/run/user/1000/containers/overlay-containers/33396f41786f6d3b360a23f8d256869fca2a7054eabb21a194d356eee242bb2d/userdata/hostname",
          "HostsPath": "/run/user/1000/containers/overlay-containers/9a048fef4e4d3608355576f7fac466606e87e51b03121e5a4fc5a6d0106d9fca/userdata/hosts",
          "StaticDir": "/home/andrew/.local/share/containers/storage/overlay-containers/33396f41786f6d3b360a23f8d256869fca2a7054eabb21a194d356eee242bb2d/userdata",
          "OCIConfigPath": "/home/andrew/.local/share/containers/storage/overlay-containers/33396f41786f6d3b360a23f8d256869fca2a7054eabb21a194d356eee242bb2d/userdata/config.json",
          "OCIRuntime": "crun",
          "ConmonPidFile": "/run/user/1000/containers/overlay-containers/33396f41786f6d3b360a23f8d256869fca2a7054eabb21a194d356eee242bb2d/userdata/conmon.pid",
          "PidFile": "/run/user/1000/containers/overlay-containers/33396f41786f6d3b360a23f8d256869fca2a7054eabb21a194d356eee242bb2d/userdata/pidfile",
          "Name": "pihole-pihole-container",
          "RestartCount": 0,
          "Driver": "overlay",
          "MountLabel": "",
          "ProcessLabel": "",
          "AppArmorProfile": "",
          "EffectiveCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "BoundingCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "ExecIDs": [],
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/home/andrew/.local/share/containers/storage/overlay/968a58d6b88aa3cff7a30f22f44ec93e5c6f7850ef91d3f4a9b7416e5254456d/diff:/home/andrew/.local/share/containers/storage/overlay/98eb067da64358a7312f7579920b542dea29a0eb15c085df1a5114dabb0ef841/diff:/home/andrew/.local/share/containers/storage/overlay/071e8679067685e92f7afd26a80ad9aee047b0a3a6118a2323e7e413a48925d7/diff:/home/andrew/.local/share/containers/storage/overlay/9f9edfcc50143ade365164a0f0ae00dce82f21524ddcaf9e3417dc5587bb9d52/diff:/home/andrew/.local/share/containers/storage/overlay/ba823c045a0af906accf3c28e5d141272aad5b2e2337c72d7cbea7a703490c76/diff:/home/andrew/.local/share/containers/storage/overlay/dd73140b8cdb5e665ece9ed0d224a6dbc7e6f6ebf85bc4115ff89f60ffab65ae/diff:/home/andrew/.local/share/containers/storage/overlay/323cfa36ef0b1128247af9935520aebb7cc4304ce3d7bf3a105b8c4f2d369a77/diff:/home/andrew/.local/share/containers/storage/overlay/9aaf4d379f9bb5c55a77a6815fd8c0a494dad1786034a1cab758138fedd648d5/diff:/home/andrew/.local/share/containers/storage/overlay/650abce4b096b06ac8bec2046d821d66d801af34f1f1d4c5e272ad030c7873db/diff",
                    "UpperDir": "/home/andrew/.local/share/containers/storage/overlay/1f3bd047a98a9f2c5cdd71b036e53c2c304d90b6da14bf759fd0a01c38a46b5d/diff",
                    "WorkDir": "/home/andrew/.local/share/containers/storage/overlay/1f3bd047a98a9f2c5cdd71b036e53c2c304d90b6da14bf759fd0a01c38a46b5d/work"
               }
          },
          "Mounts": [
               {
                    "Type": "volume",
                    "Name": "pihole_piholeetc",
                    "Source": "/home/andrew/.local/share/containers/storage/volumes/pihole_piholeetc/_data",
                    "Destination": "/etc/pihole",
                    "Driver": "local",
                    "Mode": "",
                    "Options": [
                         "nosuid",
                         "nodev",
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "volume",
                    "Name": "pihole_piholednsmasq",
                    "Source": "/home/andrew/.local/share/containers/storage/volumes/pihole_piholednsmasq/_data",
                    "Destination": "/etc/dnsmasq.d",
                    "Driver": "local",
                    "Mode": "",
                    "Options": [
                         "nosuid",
                         "nodev",
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               }
          ],
          "Dependencies": [
               "9a048fef4e4d3608355576f7fac466606e87e51b03121e5a4fc5a6d0106d9fca"
          ],
          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "",
               "IPAddress": "",
               "IPPrefixLen": 0,
               "IPv6Gateway": "",
               "GlobalIPv6Address": "",
               "GlobalIPv6PrefixLen": 0,
               "MacAddress": "",
               "Bridge": "",
               "SandboxID": "",
               "HairpinMode": false,
               "LinkLocalIPv6Address": "",
               "LinkLocalIPv6PrefixLen": 0,
               "Ports": {
                    "53/tcp": [
                         {
                              "HostIp": "",
                              "HostPort": "5553"
                         }
                    ],
                    "53/udp": [
                         {
                              "HostIp": "",
                              "HostPort": "5553"
                         }
                    ]
               },
               "SandboxKey": "",
               "Networks": {
                    "podman": {
                         "EndpointID": "",
                         "Gateway": "",
                         "IPAddress": "",
                         "IPPrefixLen": 0,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "",
                         "NetworkID": "podman",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "pihole-container",
                              "9a048fef4e4d"
                         ]
                    }
               }
          },
          "Namespace": "",
          "IsInfra": false,
          "IsService": false,
          "Config": {
               "Hostname": "pihole",
               "Domainname": "",
               "User": "",
               "AttachStdin": false,
               "AttachStdout": false,
               "AttachStderr": false,
               "Tty": false,
               "OpenStdin": false,
               "StdinOnce": false,
               "Env": [
                    "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
                    "S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0",
                    "S6_KEEP_ENV=1",
                    "DNSMASQ_USER=pihole",
                    "phpver=php",
                    "VIRTUAL_HOST=pihole.ak.codes",
                    "WEBPASSWORD=hellothere",
                    "TERM=xterm",
                    "container=podman",
                    "PHP_ERROR_LOG=/var/log/lighttpd/error-pihole.log",
                    "IPv6=True",
                    "FTL_CMD=no-daemon",
                    "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "FTLCONF_LOCAL_IPV4=0.0.0.0",
                    "TZ=America/Los_Angeles",
                    "HOME=/root",
                    "HOSTNAME=pihole"
               ],
               "Cmd": null,
               "Image": "docker.io/pihole/pihole:latest",
               "Volumes": null,
               "WorkingDir": "/",
               "Entrypoint": "/s6-init",
               "OnBuild": null,
               "Labels": {
                    "PODMAN_SYSTEMD_UNIT": "podman-kube@-home-andrew-compose-pihole-pihole.yml.service",
                    "app": "pihole",
                    "io.containers.autoupdate": "registry",
                    "org.opencontainers.image.created": "2023-03-25T19:18:44.609Z",
                    "org.opencontainers.image.description": "Pi-hole in a docker container",
                    "org.opencontainers.image.licenses": "",
                    "org.opencontainers.image.revision": "aeb42394738aa652845c652b45438fb1edcfbbee",
                    "org.opencontainers.image.source": "https://github.com/pi-hole/docker-pi-hole",
                    "org.opencontainers.image.title": "docker-pi-hole",
                    "org.opencontainers.image.url": "https://github.com/pi-hole/docker-pi-hole",
                    "org.opencontainers.image.version": "2023.03.1"
               },
               "Annotations": {
                    "io.container.manager": "libpod",
                    "io.containers.autoupdate/pihole-container": "registry",
                    "io.kubernetes.cri-o.SandboxID": "9a048fef4e4d3608355576f7fac466606e87e51b03121e5a4fc5a6d0106d9fca",
                    "io.podman.annotations.ulimit": "nofile=524288:524288,nproc=256639:256639",
                    "org.opencontainers.image.stopSignal": "15"
               },
               "StopSignal": 15,
               "Healthcheck": {
                    "Test": [
                         "CMD-SHELL",
                         "dig +short +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
                    ],
                    "Interval": 30000000000,
                    "Timeout": 30000000000
               },
               "HealthcheckOnFailureAction": "none",
               "Umask": "0000",
               "Timeout": 0,
               "StopTimeout": 10,
               "sdNotifyMode": "ignore"
          },
          "HostConfig": {
               "Binds": [
                    "pihole_piholeetc:/etc/pihole:rw,rprivate,nosuid,nodev,rbind",
                    "pihole_piholednsmasq:/etc/dnsmasq.d:rw,rprivate,nosuid,nodev,rbind"
               ],
               "CgroupManager": "systemd",
               "CgroupMode": "private",
               "ContainerIDFile": "",
               "LogConfig": {
                    "Type": "journald",
                    "Config": null,
                    "Path": "",
                    "Tag": "",
                    "Size": "0B"
               },
               "NetworkMode": "container:9a048fef4e4d3608355576f7fac466606e87e51b03121e5a4fc5a6d0106d9fca",
               "PortBindings": {},
               "RestartPolicy": {
                    "Name": "always",
                    "MaximumRetryCount": 0
               },
               "AutoRemove": false,
               "VolumeDriver": "",
               "VolumesFrom": null,
               "CapAdd": [],
               "CapDrop": [],
               "Dns": [],
               "DnsOptions": [],
               "DnsSearch": [],
               "ExtraHosts": [],
               "GroupAdd": [],
               "IpcMode": "container:9a048fef4e4d3608355576f7fac466606e87e51b03121e5a4fc5a6d0106d9fca",
               "Cgroup": "",
               "Cgroups": "default",
               "Links": null,
               "OomScoreAdj": 0,
               "PidMode": "private",
               "Privileged": false,
               "PublishAllPorts": false,
               "ReadonlyRootfs": false,
               "SecurityOpt": [],
               "Tmpfs": {},
               "UTSMode": "container:9a048fef4e4d3608355576f7fac466606e87e51b03121e5a4fc5a6d0106d9fca",
               "UsernsMode": "",
               "ShmSize": 65536000,
               "Runtime": "oci",
               "ConsoleSize": [
                    0,
                    0
               ],
               "Isolation": "",
               "CpuShares": 0,
               "Memory": 0,
               "NanoCpus": 0,
               "CgroupParent": "user.slice/user-libpod_pod_65c551f405af97b3615b7624457b53e46f8fb143f9e50df62f3de5faeca96337.slice",
               "BlkioWeight": 0,
               "BlkioWeightDevice": null,
               "BlkioDeviceReadBps": null,
               "BlkioDeviceWriteBps": null,
               "BlkioDeviceReadIOps": null,
               "BlkioDeviceWriteIOps": null,
               "CpuPeriod": 0,
               "CpuQuota": 0,
               "CpuRealtimePeriod": 0,
               "CpuRealtimeRuntime": 0,
               "CpusetCpus": "",
               "CpusetMems": "",
               "Devices": [],
               "DiskQuota": 0,
               "KernelMemory": 0,
               "MemoryReservation": 0,
               "MemorySwap": 0,
               "MemorySwappiness": 0,
               "OomKillDisable": false,
               "PidsLimit": 2048,
               "Ulimits": [
                    {
                         "Name": "RLIMIT_NOFILE",
                         "Soft": 524288,
                         "Hard": 524288
                    },
                    {
                         "Name": "RLIMIT_NPROC",
                         "Soft": 256639,
                         "Hard": 256639
                    }
               ],
               "CpuCount": 0,
               "CpuPercent": 0,
               "IOMaximumIOps": 0,
               "IOMaximumBandwidth": 0,
               "CgroupConf": null
          }
     }
]

When started, this container fails with the error:

starting container 33396f41786f6d3b360a23f8d256869fca2a7054eabb21a194d356eee242bb2d: crun: setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI permission denied

The output of ulimit -u is 256637. All containers I've created within the past week or so seem to on this new version of fedora/podman be instantiated with RLIMIT_NPROC exactly 2 higher than my ulimit.

As an example, here's a caddy container I created earlier that has a ulimit that respects the limit of 8000 that I specify in containers.conf:

`podman inspect` output for caddy container. 
[
     {
          "Id": "17adac3b7d9e7bd1aae18e8413e2394d1973674eccc6ec261e923d950ec4a356",
          "Created": "2023-05-12T07:11:59.73330826-07:00",
          "Path": "caddy",
          "Args": [
               "run",
               "--config",
               "/etc/caddy/Caddyfile",
               "--adapter",
               "caddyfile"
          ],
          "State": {
               "OciVersion": "1.1.0-rc.1",
               "Status": "running",
               "Running": true,
               "Paused": false,
               "Restarting": false,
               "OOMKilled": false,
               "Dead": false,
               "Pid": 11979,
               "ConmonPid": 11972,
               "ExitCode": 0,
               "Error": "",
               "StartedAt": "2023-05-12T07:12:00.444776786-07:00",
               "FinishedAt": "0001-01-01T00:00:00Z",
               "Health": {
                    "Status": "",
                    "FailingStreak": 0,
                    "Log": null
               },
               "CgroupPath": "/user.slice/user-1000.slice/user@1000.service/user.slice/user-libpod_pod_704d1d3474796bb48e649ae4b40dbc18fa55332909bfe083286976a1ebf4d8f7.slice/libpod-17adac3b7d9e7bd1aae18e8413e2394d1973674eccc6ec261e923d950ec4a356.scope",
               "CheckpointedAt": "0001-01-01T00:00:00Z",
               "RestoredAt": "0001-01-01T00:00:00Z"
          },
          "Image": "86ab4e60cac2e943fc115f5a9ec4f7b91ee165fbabc01b3f00963ce6319d1101",
          "ImageDigest": "sha256:bc23ee7f830ab9d029e5469e82a3e36f8c401001c2c8a5a6d919a82668d8087b",
          "ImageName": "docker.io/library/caddy:latest",
          "Rootfs": "",
          "Pod": "704d1d3474796bb48e649ae4b40dbc18fa55332909bfe083286976a1ebf4d8f7",
          "ResolvConfPath": "/run/user/1000/containers/overlay-containers/931f99e4082df7007e6a6cee7bdd569846fbf7b064033fdbda0b77f4036825a5/userdata/resolv.conf",
          "HostnamePath": "/run/user/1000/containers/overlay-containers/17adac3b7d9e7bd1aae18e8413e2394d1973674eccc6ec261e923d950ec4a356/userdata/hostname",
          "HostsPath": "/run/user/1000/containers/overlay-containers/931f99e4082df7007e6a6cee7bdd569846fbf7b064033fdbda0b77f4036825a5/userdata/hosts",
          "StaticDir": "/home/andrew/.local/share/containers/storage/overlay-containers/17adac3b7d9e7bd1aae18e8413e2394d1973674eccc6ec261e923d950ec4a356/userdata",
          "OCIConfigPath": "/home/andrew/.local/share/containers/storage/overlay-containers/17adac3b7d9e7bd1aae18e8413e2394d1973674eccc6ec261e923d950ec4a356/userdata/config.json",
          "OCIRuntime": "crun",
          "ConmonPidFile": "/run/user/1000/containers/overlay-containers/17adac3b7d9e7bd1aae18e8413e2394d1973674eccc6ec261e923d950ec4a356/userdata/conmon.pid",
          "PidFile": "/run/user/1000/containers/overlay-containers/17adac3b7d9e7bd1aae18e8413e2394d1973674eccc6ec261e923d950ec4a356/userdata/pidfile",
          "Name": "caddy-caddy-container",
          "RestartCount": 0,
          "Driver": "overlay",
          "MountLabel": "",
          "ProcessLabel": "",
          "AppArmorProfile": "",
          "EffectiveCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "BoundingCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "ExecIDs": [],
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/home/andrew/.local/share/containers/storage/overlay/f1eeae4ceac1793ccf2be445ae6ebb11548e6875e0e6ad79af0fb79f78f981ec/diff:/home/andrew/.local/share/containers/storage/overlay/442785c853b5bdcedcb78cb4c06b2ffe455f61e4a666819380025d8b7b0ab6da/diff:/home/andrew/.local/share/containers/storage/overlay/c7e89f127f4832fa2a592e8ddbaaa29dd5d1823e85494bbe3d3ec12d6028df8d/diff:/home/andrew/.local/share/containers/storage/overlay/5bc340f6d4f5a3bc999dfbc790a0bdf0920b9103ef794645034de4260ee4e9c8/diff",
                    "MergedDir": "/home/andrew/.local/share/containers/storage/overlay/44c975901c48fce7f00b96564869d5b5722ea24d6dfb07bfefb2b9cff634dc6c/merged",
                    "UpperDir": "/home/andrew/.local/share/containers/storage/overlay/44c975901c48fce7f00b96564869d5b5722ea24d6dfb07bfefb2b9cff634dc6c/diff",
                    "WorkDir": "/home/andrew/.local/share/containers/storage/overlay/44c975901c48fce7f00b96564869d5b5722ea24d6dfb07bfefb2b9cff634dc6c/work"
               }
          },
          "Mounts": [
               {
                    "Type": "bind",
                    "Source": "/home/andrew/services/caddy/caddyfile",
                    "Destination": "/etc/caddy",
                    "Driver": "",
                    "Mode": "",
                    "Options": [
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "bind",
                    "Source": "/home/andrew/services/caddy/config",
                    "Destination": "/config",
                    "Driver": "",
                    "Mode": "",
                    "Options": [
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "bind",
                    "Source": "/home/andrew/services/caddy/data",
                    "Destination": "/data",
                    "Driver": "",
                    "Mode": "",
                    "Options": [
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "bind",
                    "Source": "/etc/localtime",
                    "Destination": "/etc/localtime",
                    "Driver": "",
                    "Mode": "",
                    "Options": [
                         "rbind"
                    ],
                    "RW": false,
                    "Propagation": "rprivate"
               }
          ],
          "Dependencies": [
               "931f99e4082df7007e6a6cee7bdd569846fbf7b064033fdbda0b77f4036825a5"
          ],
          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "10.88.0.1",
               "IPAddress": "10.88.0.24",
               "IPPrefixLen": 16,
               "IPv6Gateway": "",
               "GlobalIPv6Address": "",
               "GlobalIPv6PrefixLen": 0,
               "MacAddress": "9a:10:66:8f:0f:58",
               "Bridge": "",
               "SandboxID": "",
               "HairpinMode": false,
               "LinkLocalIPv6Address": "",
               "LinkLocalIPv6PrefixLen": 0,
               "Ports": {
                    "8080/tcp": [
                         {
                              "HostIp": "",
                              "HostPort": "8080"
                         }
                    ],
                    "8443/tcp": [
                         {
                              "HostIp": "",
                              "HostPort": "8443"
                         }
                    ]
               },
               "SandboxKey": "/run/user/1000/netns/netns-fa848a40-328a-4ef1-4e4d-41c57f6fc267",
               "Networks": {
                    "podman": {
                         "EndpointID": "",
                         "Gateway": "10.88.0.1",
                         "IPAddress": "10.88.0.24",
                         "IPPrefixLen": 16,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "9a:10:66:8f:0f:58",
                         "NetworkID": "podman",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "caddy-container",
                              "931f99e4082d"
                         ]
                    }
               }
          },
          "Namespace": "",
          "IsInfra": false,
          "IsService": false,
          "Config": {
               "Hostname": "caddy",
               "Domainname": "",
               "User": "",
               "AttachStdin": false,
               "AttachStdout": false,
               "AttachStderr": false,
               "Tty": false,
               "OpenStdin": false,
               "StdinOnce": false,
               "Env": [
                    "CADDY_VERSION=v2.6.4",
                    "XDG_CONFIG_HOME=/config",
                    "XDG_DATA_HOME=/data",
                    "TZ=America/Los_Angeles",
                    "HOST_IP=10.0.0.2",
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "TERM=xterm",
                    "container=podman",
                    "HOME=/root",
                    "HOSTNAME=caddy"
               ],
               "Cmd": [
                    "caddy",
                    "run",
                    "--config",
                    "/etc/caddy/Caddyfile",
                    "--adapter",
                    "caddyfile"
               ],
               "Image": "docker.io/library/caddy:latest",
               "Volumes": null,
               "WorkingDir": "/srv",
               "Entrypoint": "",
               "OnBuild": null,
               "Labels": {
                    "PODMAN_SYSTEMD_UNIT": "podman-kube@-home-andrew-compose-caddy-caddy.yml.service",
                    "app": "caddy",
                    "io.containers.autoupdate": "registry",
                    "org.opencontainers.image.description": "a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go",
                    "org.opencontainers.image.documentation": "https://caddyserver.com/docs",
                    "org.opencontainers.image.licenses": "Apache-2.0",
                    "org.opencontainers.image.source": "https://github.com/caddyserver/caddy-docker",
                    "org.opencontainers.image.title": "Caddy",
                    "org.opencontainers.image.url": "https://caddyserver.com",
                    "org.opencontainers.image.vendor": "Light Code Labs",
                    "org.opencontainers.image.version": "v2.6.4"
               },
               "Annotations": {
                    "bind-mount-options": "/home/andrew/services/caddy/caddyfile:Z",
                    "io.container.manager": "libpod",
                    "io.containers.autoupdate/caddy-container": "registry",
                    "io.kubernetes.cri-o.SandboxID": "931f99e4082df7007e6a6cee7bdd569846fbf7b064033fdbda0b77f4036825a5",
                    "org.opencontainers.image.stopSignal": "15"
               },
               "StopSignal": 15,
               "HealthcheckOnFailureAction": "none",
               "Umask": "0000",
               "Timeout": 0,
               "StopTimeout": 10,
               "sdNotifyMode": "ignore"
          },
          "HostConfig": {
               "Binds": [
                    "/home/andrew/services/caddy/caddyfile:/etc/caddy:rw,rprivate,rbind",
                    "/home/andrew/services/caddy/config:/config:rw,rprivate,rbind",
                    "/home/andrew/services/caddy/data:/data:rw,rprivate,rbind",
                    "/etc/localtime:/etc/localtime:ro,rprivate,rbind"
               ],
               "CgroupManager": "systemd",
               "CgroupMode": "private",
               "ContainerIDFile": "",
               "LogConfig": {
                    "Type": "journald",
                    "Config": null,
                    "Path": "",
                    "Tag": "",
                    "Size": "0B"
               },
               "NetworkMode": "container:931f99e4082df7007e6a6cee7bdd569846fbf7b064033fdbda0b77f4036825a5",
               "PortBindings": {},
               "RestartPolicy": {
                    "Name": "always",
                    "MaximumRetryCount": 0
               },
               "AutoRemove": false,
               "VolumeDriver": "",
               "VolumesFrom": null,
               "CapAdd": [],
               "CapDrop": [],
               "Dns": [],
               "DnsOptions": [],
               "DnsSearch": [],
               "ExtraHosts": [],
               "GroupAdd": [],
               "IpcMode": "container:931f99e4082df7007e6a6cee7bdd569846fbf7b064033fdbda0b77f4036825a5",
               "Cgroup": "",
               "Cgroups": "default",
               "Links": null,
               "OomScoreAdj": 0,
               "PidMode": "private",
               "Privileged": false,
               "PublishAllPorts": false,
               "ReadonlyRootfs": false,
               "SecurityOpt": [],
               "Tmpfs": {},
               "UTSMode": "container:931f99e4082df7007e6a6cee7bdd569846fbf7b064033fdbda0b77f4036825a5",
               "UsernsMode": "",
               "ShmSize": 65536000,
               "Runtime": "oci",
               "ConsoleSize": [
                    0,
                    0
               ],
               "Isolation": "",
               "CpuShares": 0,
               "Memory": 268435456,
               "NanoCpus": 0,
               "CgroupParent": "user.slice/user-libpod_pod_704d1d3474796bb48e649ae4b40dbc18fa55332909bfe083286976a1ebf4d8f7.slice",
               "BlkioWeight": 0,
               "BlkioWeightDevice": null,
               "BlkioDeviceReadBps": null,
               "BlkioDeviceWriteBps": null,
               "BlkioDeviceReadIOps": null,
               "BlkioDeviceWriteIOps": null,
               "CpuPeriod": 0,
               "CpuQuota": 0,
               "CpuRealtimePeriod": 0,
               "CpuRealtimeRuntime": 0,
               "CpusetCpus": "",
               "CpusetMems": "",
               "Devices": [],
               "DiskQuota": 0,
               "KernelMemory": 0,
               "MemoryReservation": 0,
               "MemorySwap": 0,
               "MemorySwappiness": -1,
               "OomKillDisable": false,
               "PidsLimit": 2048,
               "Ulimits": [
                    {
                         "Name": "RLIMIT_NOFILE",
                         "Soft": 524288,
                         "Hard": 524288
                    },
                    {
                         "Name": "RLIMIT_NPROC",
                         "Soft": 8000,
                         "Hard": 8000
                    }
               ],
               "CpuCount": 0,
               "CpuPercent": 0,
               "IOMaximumIOps": 0,
               "IOMaximumBandwidth": 0,
               "CgroupConf": null
          }
     }
]

This one was also instantiated using systemctl --user start podman-kube@$(systemd-escape caddy.yml).service, and it was created using the following yaml:

kube yaml for caddy container 
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.4.2

# NOTE: If you generated this yaml from an unprivileged and rootless podman container on an SELinux
# enabled system, check the podman generate kube man page for steps to follow to ensure that your pod/container
# has the right permissions to access the volumes added.
---
apiVersion: v1
kind: Pod
metadata:
  annotations:
    bind-mount-options: /home/andrew/services/caddy/caddyfile:Z
    io.containers.autoupdate/caddy-container: registry
  creationTimestamp: "2023-03-14T08:51:00Z"
  labels:
    app: caddy
  name: caddy
spec:
  securityContext:
    seLinuxOptions:
      type: spc_t
  containers:
  - env:
    - name: TZ
      value: America/Los_Angeles
    - name: HOST_IP
      value: 10.0.0.2
    image: docker.io/library/caddy:latest
    name: caddy-container
    ports:
    - containerPort: 8080
      hostPort: 8080
    - containerPort: 8443
      hostPort: 8443
    resources:
      limits:
        memory: 256Mi
    volumeMounts:
    - mountPath: /etc/caddy
      name: home-andrew-services-caddy-caddyfile-host-0
    - mountPath: /config
      name: home-andrew-services-caddy-config-host-1
    - mountPath: /data
      name: home-andrew-services-caddy-data-host-2
    - mountPath: /etc/localtime
      name: etc-localtime-host-3
      readOnly: true
  volumes:
  - hostPath:
      path: /home/andrew/services/caddy/caddyfile
      type: Directory
    name: home-andrew-services-caddy-caddyfile-host-0
  - hostPath:
      path: /home/andrew/services/caddy/config
      type: Directory
    name: home-andrew-services-caddy-config-host-1
  - hostPath:
      path: /home/andrew/services/caddy/data
      type: Directory
    name: home-andrew-services-caddy-data-host-2
  - hostPath:
      path: /etc/localtime
      type: File
    name: etc-localtime-host-3

I've already done an entire podman system reset which did not resolve any of the issues. I'm out of ideas.

Steps to reproduce the issue

noted above.

Describe the results you received

Describe the results you received

Describe the results you expected

Describe the results you expected

podman info output

output of `podman info` 
host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 99.35
    systemPercent: 0.24
    userPercent: 0.42
  cpus: 24
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: server
    version: "38"
  eventLogger: journald
  hostname: yggdrasil.ak.codes
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.2.14-300.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 11583135744
  memTotal: 67320336384
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.4-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8502636544
  swapTotal: 8589930496
  uptime: 10h 19m 43.00s (Approximately 0.42 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
  - nvcr.io
store:
  configFile: /home/andrew/.config/containers/storage.conf
  containerStore:
    number: 79
    paused: 0
    running: 68
    stopped: 11
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/andrew/.local/share/containers/storage
  graphRootAllocated: 847517483008
  graphRootUsed: 175022997504
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 27
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/andrew/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 1681486942
  BuiltTime: Fri Apr 14 08:42:22 2023
  GitCommit: ""
  GoVersion: go1.20.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes
@andrew-kennedy andrew-kennedy added the kind/bug Categorizes issue or PR as related to a bug. label May 13, 2023
@rhatdan
Copy link
Member

rhatdan commented May 14, 2023

@giuseppe PTAL

@andrew-kennedy
Copy link
Author

andrew-kennedy commented May 14, 2023 •
edited

Found out the issue! Somehow missed this while going over the generated kube yaml:

io.podman.annotations.ulimit: nofile=524288:524288,nproc=256639:256639

This line was generated only starting in podman 4.5.0 and somehow had no ill effect on fedora 37, but once I upgraded to fedora 38 I think the system ulimit -u was decreased by 2 to 256639 for some reason? So then all those containers had an NPROC that was too high. Deleting this ulimit annotation entirely fixed my issue, as old version of podman also didn't generate these annotations and those kube yaml containers work fine.

@giuseppe
Copy link
Member

has anything changed on your system related to the system memory?

I've not seen any change in Fedora 38 related to user limits.

What value do you have in /proc/sys/kernel/threads-max? The default ulimit -u is that value divided by two:

$ cat /proc/sys/kernel/threads-max
253351
$ ulimit -u
126675

The first value, if not overridden, is approximately TOTAL_MEMORY in kB/128:

$ grep MemTotal /proc/meminfo 
MemTotal:       32475804 kB
$ echo $((32475804/128))
253717

@andrew-kennedy
Copy link
Author

My threads-max value is 513272, and my ulimit -u is 256636, so the values are correct in fedora. It seems that what actually happened is podman 4.5.0 generated those kube yaml files with incorrect NPROC settings in the annotations. I would imagine there is a chance I just hadn't rebooted my system and thus relaunched or recreated all of these containers from scratch until I changed fedora versions. It might be merely a coincidence. What I think is a bug is that podman 4.5.0 would reproducibly generate kube yaml with an nproc limit set to an invalid value.

@Luap99
Copy link
Member

Luap99 commented May 15, 2023

This logic was added in 35d16ea, it seems like a bad idea to add defaults when no limits were set on the cli. This makes the yaml much less portable across systems. And at least reading the logic it tried to avoid adding them when they were set to the default but it doesn't seem to work?

@rhatdan
Copy link
Member

rhatdan commented May 15, 2023

I agree if we are adding these annotations, when they were never set in the first place by the user, that would be bad.

@bsherman bsherman mentioned this issue May 15, 2023
Closed
@Luap99 Luap99 mentioned this issue May 26, 2023
Closed
@rhatdan
Copy link
Member

rhatdan commented May 26, 2023

@umohnani8 Can you change the podman code to only set the Annotation if the user actually set the limit, not if the limit is default.

@Cydox Cydox mentioned this issue May 27, 2023
Closed
@umohnani8 umohnani8 mentioned this issue May 30, 2023
Merged
@Baggypants Baggypants mentioned this issue Jun 2, 2023
Closed
@RocketRide9 RocketRide9 mentioned this issue Jun 10, 2023
Closed
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 4, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 4, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add default ulimit test for gen kube umohnani8/libpod
4 participants
@giuseppe @rhatdan @andrew-kennedy @Luap99