Recover ghost containers on start#699
Merged
Merged
Conversation
An unclean Podman Machine shutdown can desync libpod and c/storage: a
container's DB entry survives (stuck Created) while its storage layer is
gone. `podman run --replace` then fails with:
Error: getting container from store "<id>": container not known
and the unit never starts. The host podman client can't clear it
("container not known" against the missing layer); the removal only
succeeds inside the VM's rootful scope.
Detect that signature on the start pass and purge the stuck-Created
lerd-* containers in the VM via `podman machine ssh ... podman rm -f`,
then retry the start once. Targets status=created so running services
are untouched; site data is host bind-mounted, so nothing is lost.
Mirrors the existing overlay-corruption heal; darwin-only, no-op
elsewhere (native podman shares the container scope on Linux).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #698.
Problem
An unclean Podman Machine shutdown can desync libpod and c/storage: a container's DB entry survives (stuck
Created) while its storage layer is gone, a "ghost".podman run --replacethen fails and the unit never starts:The host podman client can't clear it (
container not knownagainst the missing layer), andlerd service reinstalldoesn't help because its stop step uses the same host-sidepodman rm -f. The removal only succeeds inside the VM's rootful scope.Change
On the service start pass, detect the
getting container from store ...: container not knownsignature and purge the stuck-Createdlerd-* containers inside the VM viapodman machine ssh ... podman rm -f, then retry the start once. The purge is filtered tostatus=created, so running services are left alone; site data is host bind-mounted, so nothing is lost.This mirrors the existing overlay-corruption heal. It is darwin-only (
ghost_heal_darwin.go), with a no-op on Linux where native podman shares the container scope and a plainpodman rm -falready reaches the record.Notes
Not Podman version specific: the failure persisted across a 6 to 5 downgrade. The trigger is unclean VM shutdowns (host sleep/reboot without
lerd quit).The filter selectivity was verified live against a stuck-
Createdstand-in container: the heal targeted only that container and excluded all running services.