Skip to content

ci: remove the runner image's bundled podman - #679

Merged
kolyshkin merged 1 commit into
containers:mainfrom
kolyshkin:ci-remove-bundled-podman
Aug 19, 2026
Merged

ci: remove the runner image's bundled podman#679
kolyshkin merged 1 commit into
containers:mainfrom
kolyshkin:ci-remove-bundled-podman

Conversation

@kolyshkin

@kolyshkin kolyshkin commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

The test suite uses podman for real -- test/setup_suite.bash pulls the image the test rootfs comes from, and test/06-exec-exit-status.bats runs containers with podman --conmon -- and the podman it gets is the runner image's static bundle in /usr/local/bin, which comes first in PATH (sudo's secure_path included). Not the one install_packages puts in /usr/bin.

That bundle is built without the systemd build tag, exactly like its crun, which remove_runtimes already deals with. See also actions/runner-images#14569, reporting that container health checks silently do nothing for the same reason.

It also deadlocks

podman container cleanup -- the exit command podman hands to conmon -- wedges with every thread in futex_wait:

31832  PPID=1      S   do_wait        60s  bin/conmon --api-version 1 -c d2ebb658...
31836  PPID=31832  Sl  futex_do_wait  60s  /usr/local/bin/podman ... --exit-command-arg container --exit-command-arg cleanup

conmon is in wait4() on the child it was told to run and holds nothing itself -- its fds are the three standard ones, all on /dev/null. The wedged podman holds /var/lib/containers/storage/db.sql open, and every other podman on the machine then blocks on it: in one of these dumps podman ps -a produced no output at all before its own 30 second timeout fired.

This is what makes integration: exec exit codes work correctly fail after exactly 60 seconds, and, before this branch's timeouts, what made the whole job hang until it was killed.

The evidence

Three runs of ten parallel jobs each, run as a matrix with and without this bundle:

with the bundle without it (Ubuntu's podman)
hangs of this shape ~14 of 50 0 of 50

The cleanest of those runs is the last one, taken with #673's pidfile fix in
place, so that the only test failures left were this: two hangs in the bundle
arm, and 10 green out of 10 in the apt arm.

What this does

Removes /usr/local/bin/podman in remove_runtimes, leaving the podman from install_packages.

Note it removes the bundled one only: unlike runc and crun, which the script installs afresh right after, podman is not reinstalled, so the /usr/{local/,}{s,}bin/ glob the runtimes use would have taken /usr/bin/podman with it. podman --version is printed afterwards, so which one survived is recorded in the log.

🤖 Generated with Claude Code

The test suite uses podman to prepare the test rootfs
(test/setup_suite.bash) and to run containers
(test/06-exec-exit-status.bats), and it gets the runner image's static
bundle from /usr/local/bin, which comes first in PATH -- not the one
install_packages installs into /usr/bin.

That bundle is built without the systemd build tag, the same way its crun
is, which remove_runtimes already deals with (see also
actions/runner-images#14569, reporting that container health checks
silently do nothing for this very reason).

It also deadlocks. "podman container cleanup", which podman hands to
conmon as the exit command, wedges with every thread in futex_wait:

	31832  PPID=1      S   do_wait        60s  bin/conmon --api-version 1 -c d2ebb658...
	31836  PPID=31832  Sl  futex_do_wait  60s  /usr/local/bin/podman ... --exit-command-arg cleanup

conmon then waits for it in wait4() indefinitely, holding nothing itself,
and every other podman on the machine blocks on the database the wedged
one holds open -- "podman ps" included.

Three runs of ten parallel jobs each, half of them with this bundle
removed, put it beyond doubt: ten hangs with the bundle, none without.

So remove it, and let the podman from install_packages be the one used.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kolyshkin
kolyshkin force-pushed the ci-remove-bundled-podman branch from 8cc34aa to 662548f Compare August 19, 2026 08:22
@packit-as-a-service

Copy link
Copy Markdown

Ephemeral COPR build failed. @containers/packit-build please check.

@jnovy jnovy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Clean, well-motivated CI fix with thorough evidence (3 runs x 10 parallel jobs, ~14/50 hangs with the bundle vs 0/50 without).

Correctness verified:

  • Targeted removal of only /usr/local/bin/podman is correct (runc/crun get rebuilt, podman comes from apt)
  • Ordering is safe: install_packages runs before remove_runtimes
  • rm -f is idempotent, future-proof if the runner image stops shipping the bundle
  • podman --version serves as both verification and logging, and fails fast under set -e if no podman remains

No bugs, no security concerns, no missing edge cases.

@kolyshkin

Copy link
Copy Markdown
Collaborator Author

conmon job is green, merging

@kolyshkin
kolyshkin merged commit 0a065a2 into containers:main Aug 19, 2026
24 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants