conmon cdfba70f9e2dbb5a18c2 <nwarn>: runtime stderr: runc create failed

[YRXING]

Issue Description

When I use podman to create container, it blocked and an error occurred in the runc service. The flowing is the error log.
Nov 18 12:27:08 dc06-p3-t628-n013 conmon[396051]: conmon 90a1aa9dc23f95a46e5e : container 397582 exited with status 137
Nov 18 12:45:19 dc06-p3-t628-n013 conmon[386855]: conmon d05ca534cae8fa505f1b : container 396038 exited with status 137
Nov 18 14:55:52 dc06-p3-t628-n013 conmon[2154175]: conmon cdfba70f9e2dbb5a18c2 : runtime stderr: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF
Nov 18 14:55:52 dc06-p3-t628-n013 conmon[2154175]: conmon cdfba70f9e2dbb5a18c2 : Failed to create container: exit status 1
Nov 18 14:55:52 dc06-p3-t628-n013 conmon[2715470]: conmon e65dbd8ac78061834562 : runtime stderr: runc create failed: unable to start container process: error during container init: unable to store init state: %!w()
Nov 18 14:55:52 dc06-p3-t628-n013 conmon[2715470]: conmon e65dbd8ac78061834562 : Failed to create container: exit status 1
Nov 19 16:09:00 dc06-p3-t628-n013 conmon[23966]: conmon d8e7b9fba2987d386bde : runtime stderr: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF
Nov 19 16:09:00 dc06-p3-t628-n013 conmon[23966]: conmon d8e7b9fba2987d386bde : Failed to create container: exit status 1
Nov 19 17:08:37 dc06-p3-t628-n013 conmon[1271628]: conmon 0c86f25a68b7d1d0086d : runtime stderr: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF
Nov 19 17:08:37 dc06-p3-t628-n013 conmon[1271628]: conmon 0c86f25a68b7d1d0086d : Failed to create container: exit status 1
Nov 19 21:23:19 dc06-p3-t628-n013 conmon[951854]: conmon 37764cef06e9a5925286 : runtime stderr: runc create failed: unable to create new parent process: unable to setup exec fifo: open /run/runc/37764cef06e9a59252863e19915cd940b2c01e2a4e91eec9339a41178ccf9751/exec.fifo: no such file or directory
Nov 19 21:23:19 dc06-p3-t628-n013 conmon[951854]: conmon 37764cef06e9a5925286 : Failed to create container: exit status 1
Nov 20 03:33:18 dc06-p3-t628-n013 systemd[1]: carrier.service: Dependency After=carrier.service dropped
Nov 20 03:33:19 dc06-p3-t628-n013 systemd[1]: carrier.service: Dependency After=carrier.service dropped
Nov 20 15:33:20 dc06-p3-t628-n013 conmon[582965]: conmon 795834d7c475dc1da9b9 : runtime stderr: runc create failed: unable to create new parent process: unable to setup exec fifo: open /run/runc/795834d7c475dc1da9b9a8c6e79575728df6aade1d2538666e5205b05771fb7e/exec.fifo: no such file or directory

And libpod created a large number of cgroup directories at the same time.

Does anyone know why?

Steps to reproduce the issue

Steps to reproduce the issue
1.
2.
3.

Describe the results you received

Describe the results you received

Describe the results you expected

Describe the results you expected

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Honny1]

Hello, can you please provide the results of podman info?

[github-actions]

A reviewer has determined we need more information to understand the reported issue. A comment on what is missing should be provided. Be certain you:

• provide an exact reproducer where possible
• verify you have provided all relevant information - minimum is podman info
• answer any follow up questions

If no response to the needs-info is provided in 30 days, this issue may be closed by our stale bot.

For more information on reporting issues on this repository, consult our issue guide.

[YRXING]

Hello, can you please provide the results of podman info?

Actually, we didn't use podman directly, but rather called the libpod library code, whose logic for creating containers is exactly the same as podman's.
runc version:
runc version 1.4.0-rc.3+dev
spec: 1.3.0
go: go1.24.0
conmon version
conmon version 2.1.12
commit: 3bc422cd8aaec542d85d1a80f2d38e6e69046b5b

the podmand version of libpod library:
version 1.6.4

Only this one machine in our cluster has this problem, so I suspect it might be a system limitation somewhere.It's unlikely to be a problem with runc.

And the libpod created a large number of cgroup directories at the same time seems to be a series of problems caused by the abnormal termination of the conmon process, so I feel there is a bug here.

[Honny1]

Ah, then I'd like to ask for a minimal reproducer. Also, if you ran the same thing with just a different version of runc, I suspect runc is the cause. I suggest reporting it to the runc repository.

cc @jnovy just in case it is caused by conmon.

[Luap99]

We do not support direct use of libpod, use the podman REST api and our bindings.

Also version 1.6.4 is way to old