ci: remove the runner image's bundled podman - #679
Merged
Conversation
This was referenced Aug 19, 2026
kolyshkin
force-pushed
the
ci-remove-bundled-podman
branch
from
August 19, 2026 08:21
9b0820d to
8cc34aa
Compare
The test suite uses podman to prepare the test rootfs (test/setup_suite.bash) and to run containers (test/06-exec-exit-status.bats), and it gets the runner image's static bundle from /usr/local/bin, which comes first in PATH -- not the one install_packages installs into /usr/bin. That bundle is built without the systemd build tag, the same way its crun is, which remove_runtimes already deals with (see also actions/runner-images#14569, reporting that container health checks silently do nothing for this very reason). It also deadlocks. "podman container cleanup", which podman hands to conmon as the exit command, wedges with every thread in futex_wait: 31832 PPID=1 S do_wait 60s bin/conmon --api-version 1 -c d2ebb658... 31836 PPID=31832 Sl futex_do_wait 60s /usr/local/bin/podman ... --exit-command-arg cleanup conmon then waits for it in wait4() indefinitely, holding nothing itself, and every other podman on the machine blocks on the database the wedged one holds open -- "podman ps" included. Three runs of ten parallel jobs each, half of them with this bundle removed, put it beyond doubt: ten hangs with the bundle, none without. So remove it, and let the podman from install_packages be the one used. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
kolyshkin
force-pushed
the
ci-remove-bundled-podman
branch
from
August 19, 2026 08:22
8cc34aa to
662548f
Compare
|
Ephemeral COPR build failed. @containers/packit-build please check. |
jnovy
approved these changes
Aug 19, 2026
jnovy
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. Clean, well-motivated CI fix with thorough evidence (3 runs x 10 parallel jobs, ~14/50 hangs with the bundle vs 0/50 without).
Correctness verified:
- Targeted removal of only /usr/local/bin/podman is correct (runc/crun get rebuilt, podman comes from apt)
- Ordering is safe: install_packages runs before remove_runtimes
- rm -f is idempotent, future-proof if the runner image stops shipping the bundle
- podman --version serves as both verification and logging, and fails fast under set -e if no podman remains
No bugs, no security concerns, no missing edge cases.
Collaborator
Author
|
conmon job is green, merging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The test suite uses podman for real --
test/setup_suite.bashpulls the image the test rootfs comes from, andtest/06-exec-exit-status.batsruns containers withpodman --conmon-- and the podman it gets is the runner image's static bundle in/usr/local/bin, which comes first in PATH (sudo'ssecure_pathincluded). Not the oneinstall_packagesputs in/usr/bin.That bundle is built without the
systemdbuild tag, exactly like its crun, whichremove_runtimesalready deals with. See also actions/runner-images#14569, reporting that container health checks silently do nothing for the same reason.It also deadlocks
podman container cleanup-- the exit command podman hands to conmon -- wedges with every thread infutex_wait:conmon is in
wait4()on the child it was told to run and holds nothing itself -- its fds are the three standard ones, all on/dev/null. The wedged podman holds/var/lib/containers/storage/db.sqlopen, and every other podman on the machine then blocks on it: in one of these dumpspodman ps -aproduced no output at all before its own 30 second timeout fired.This is what makes
integration: exec exit codes work correctlyfail after exactly 60 seconds, and, before this branch's timeouts, what made the whole job hang until it was killed.The evidence
Three runs of ten parallel jobs each, run as a matrix with and without this bundle:
The cleanest of those runs is the last one, taken with #673's pidfile fix in
place, so that the only test failures left were this: two hangs in the bundle
arm, and 10 green out of 10 in the apt arm.
What this does
Removes
/usr/local/bin/podmaninremove_runtimes, leaving the podman frominstall_packages.Note it removes the bundled one only: unlike runc and crun, which the script installs afresh right after, podman is not reinstalled, so the
/usr/{local/,}{s,}bin/glob the runtimes use would have taken/usr/bin/podmanwith it.podman --versionis printed afterwards, so which one survived is recorded in the log.🤖 Generated with Claude Code