Troubleshooting

Troubleshooting

The failures you are most likely to hit, with the actual fix.

Build: btrfs/version.h: No such file or directory

# go.podman.io/storage/drivers/btrfs
fatal error: btrfs/version.h: No such file or directory
# github.com/proglottis/gpgme
Package gpgme was not found in the pkg-config search path.

You built without the tags. Use make build, or pass -tags "containers_image_openpgp exclude_graphdriver_btrfs exclude_graphdriver_devicemapper". Full rationale in Building.

go test: -tags space-separated list contains comma

You combined the tag set with integration using a comma. The tag list can't mix spaces and commas — append with a space: -tags "$(TAGS) integration".

CI job fails in ~28s: executable file 'node' not found in $PATH

The runner runs the job inside a container, and actions/checkout/actions/cache are JavaScript actions needing a node binary the base image doesn't ship. Install nodejs in a run: step before any uses: step (run steps use the shell, not node).

CI: runs stuck in waiting, no jobs assigned

runs-on: names a label no registered runner carries. This instance's runner is labelled self-hosted.

CI: integration job fails immediately on --privileged

The podman-in-podman job needs a privileged container for nested podman (storage + play kube hostPort networking). If the runner forbids privileged jobs, that has to be allowed in the runner config — it can't be fixed in the workflow.

Apply returns 502

The registry was unreachable (or the tag doesn't exist) during the mandatory pre-pull. The message is the registry's own. No secret or pod is created — fix the image ref / registry and retry. ?skip_pull=true bypasses the pre-pull only when the image is already local.

Deploy rejected: secrets require an encryption key (-spec-key-file)

The store is always on, but -spec-key-file is optional and gates secrets only. A deploy that carries secret material (or a PUT …/secrets that tries to persist) is refused when the daemon was started without a key — there is nothing to seal the secret at rest with. Either start the daemon with -spec-key-file=/etc/podman-api/spec.key, or deploy without secrets. Templates and no-secret deploys work key-less. See Operating → The state store.

State store: permission denied / daemon won't start

-state-db is a location (default /var/lib/podman-api/state.db) and the store is always opened. The daemon creates the parent directory, but a non-root run pointed at the privileged default path gets a permission error. Point -state-db at a writable path, or run as a user that owns /var/lib/podman-api/ (the bundled installer's podman-api user does). See Provisioning-a-Podman-Host § 8.

A deleted instance left its secret behind

DELETE keeps per-instance secrets unless you pass ?prune_secrets=true (and ?prune_volumes=true for volumes). To reap an already-orphaned secret, call DELETE ...?prune_secrets=true&prune_volumes=true again — delete is an idempotent reconcile and will remove orphans even though the pod is gone. Or use the per-resource endpoints: DELETE /hosts/{host}/secrets/{name} and DELETE /hosts/{host}/volumes/{name}.

Host healthz fails / unknown host

• unknown host → no hosts/<id>.yaml with that id, or it didn't load. Check -hosts-dir.
• SSH/tunnel errors → wrong addr, wrong ssh_key, the key isn't authorised on the target, or the socket path is wrong. Verify with ssh <addr> true and check the socket path is /run/user/<uid>/podman/podman.sock. See Provisioning a Podman Host.

play kube fails: catatonit: executable file not found

The target (or your local dev box) lacks the pause binary podman uses for the pod infra container. Install it: sudo dnf install -y catatonit (AlmaLinux/Rocky). The quay.io/podman/stable CI image already includes it.

A bad keys.yaml edit — am I locked out?

No. On SIGHUP, a parse error or an empty key list is logged and the previous list stays live. Fix the file and reload again. See Operating.

Related

• Building · Provisioning a Podman Host · Deploying · Operating