fix(cluster): bootstrap waits for the k3s API instead of trusting the installer exit#208
Merged
Merged
Conversation
… installer exit On a small VPS, k3s's slow first start (image unpack, datastore init) makes the upstream installer's `systemctl start k3s` exceed systemd's readiness-notify timeout, so the installer exits non-zero even though k3s comes up fine moments later. bootstrap treated that exit as fatal and aborted with `installing k3s: exit status 1` before deploying burrowd or printing the join token — a false failure. bootstrap now makes the k3s API answering — not the installer's exit code — the success criterion. After running the installer it holds any non-zero exit and polls the k3s API for readiness with a generous, configurable budget (default 4m, --k3s-api-timeout), well past a slow first-start. If the API answers, bootstrap proceeds, logging a non-zero installer exit as a warning. It fails only if the API never answers within the budget, and that error carries the installer's exit and points at `journalctl -xeu k3s.service` and `systemctl status k3s`. The idempotent skip for an already-running k3s is unchanged. Signed-off-by: Nicholas Phillips <nsphilli@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Dogfooding
burrow cluster bootstrapon a DigitalOcean droplet surfaced a false failure. The upstream k3s installer ends withsystemctl start k3s. On a small VPS, k3s's FIRST start is slow (image unpack, sqlite/etcd init) and exceeds systemd's start-notify timeout, sosystemctl startreturns non-zero — even though k3s comes up fine ~30-60s later (the journal shows the normal "Waiting for caches to sync" / "Storage is ready for all registered resources" startup). The installer therefore exited non-zero and bootstrap treated that as fatal (installing k3s: exit status 1), aborting before it deployed burrowd or printed the join token.The fix
Make the k3s API answering — not the installer's exit code — bootstrap's success criterion.
ensureK3sInstalledholds any non-zero exit rather than returning it, then polls the k3s API for readiness with a generous, configurable budget (default 4m,--k3s-api-timeout), well past a slow first-start.journalctl -xeu k3s.serviceandsystemctl status k3sfor diagnosis.The readiness budget is threaded through the
k3sInstallerseam (WaitForAPI(ctx, budget)), so unit tests substitute a fake installer and never sleep.Tests
TestEnsureK3sInstalledProceedsOnInstallerExitWhenAPIReady/TestBootstrapProceedsWhenInstallerExitsButAPIReady— the exact regression: installer exits non-zero but the API is ready, so bootstrap reaches the deploy step and prints the join token (no abort), logging the exit as a warning.TestEnsureK3sInstalledFailsWhenAPINeverReady/TestBootstrapFailsWhenAPINeverReady— the API never answers, so bootstrap fails with the journalctl/systemctl guidance and prints no join token.Light gate (
go build,go vet,gofmt -l,go test ./..., SPDX) passes.