Runner must wait for dockerd on host reboot, not just on compose up - #12
Merged
Merged
Conversation
The healthcheck added in #7 governs docker compose up. After a host reboot the restart policy starts both containers at once and ignores depends_on conditions, so the race returned in the one situation where it is least observed. A reboot test showed the runner logging three failed daemon pings before recovering; a job claimed in that window would fail. The runner entrypoint now waits for the daemon before starting, probing the Docker API ping endpoint over HTTP because the runner image ships no docker CLI, and giving up after two minutes rather than hanging forever. Verified by rebooting the live host twice: three race errors before the change, zero after, with a CI job dispatched post-reboot succeeding both times. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #7, found by rebooting the live host — a test worth doing while the instance has no DNS and therefore no users.
depends_on: condition: service_healthyonly governsdocker compose up. After a host reboot, Docker's restart policy starts both containers simultaneously and ignoresdepends_on, so the startup race came back in precisely the situation nobody watches. The runner logged three failed daemon pings before recovering. It recovered because the poller retries — but a job claimed during that window would have failed.The entrypoint now waits for the daemon before starting. It probes the Docker API's
/_pingover HTTP rather than shelling out todocker, because the runner image ships no docker CLI (my first attempt useddocker infoand would have failed immediately), and it gives up after two minutes instead of hanging forever.Verified across two reboots of the live host:
bun run validategreen.🤖 Generated with Claude Code