Skip to content

fix(cluster): bootstrap waits for the k3s API instead of trusting the installer exit#208

Merged
incognick merged 1 commit into
mainfrom
fix/bootstrap-slow-k3s-start
Jul 4, 2026
Merged

fix(cluster): bootstrap waits for the k3s API instead of trusting the installer exit#208
incognick merged 1 commit into
mainfrom
fix/bootstrap-slow-k3s-start

Conversation

@incognick

Copy link
Copy Markdown
Contributor

The bug

Dogfooding burrow cluster bootstrap on a DigitalOcean droplet surfaced a false failure. The upstream k3s installer ends with systemctl start k3s. On a small VPS, k3s's FIRST start is slow (image unpack, sqlite/etcd init) and exceeds systemd's start-notify timeout, so systemctl start returns non-zero — even though k3s comes up fine ~30-60s later (the journal shows the normal "Waiting for caches to sync" / "Storage is ready for all registered resources" startup). The installer therefore exited non-zero and bootstrap treated that as fatal (installing k3s: exit status 1), aborting before it deployed burrowd or printed the join token.

The fix

Make the k3s API answering — not the installer's exit code — bootstrap's success criterion.

  • After running the installer, ensureK3sInstalled holds any non-zero exit rather than returning it, then polls the k3s API for readiness with a generous, configurable budget (default 4m, --k3s-api-timeout), well past a slow first-start.
  • If the API answers within the budget, bootstrap proceeds. A non-zero installer exit is logged as a warning ("the k3s installer reported a non-zero exit (a slow first-start can time out systemd's readiness wait); the k3s API came up, continuing").
  • bootstrap fails only if the API never answers within the budget, and that error carries the installer's exit and points at journalctl -xeu k3s.service and systemctl status k3s for diagnosis.
  • The idempotent skip for an already-running k3s is unchanged.

The readiness budget is threaded through the k3sInstaller seam (WaitForAPI(ctx, budget)), so unit tests substitute a fake installer and never sleep.

Tests

  • TestEnsureK3sInstalledProceedsOnInstallerExitWhenAPIReady / TestBootstrapProceedsWhenInstallerExitsButAPIReady — the exact regression: installer exits non-zero but the API is ready, so bootstrap reaches the deploy step and prints the join token (no abort), logging the exit as a warning.
  • TestEnsureK3sInstalledFailsWhenAPINeverReady / TestBootstrapFailsWhenAPINeverReady — the API never answers, so bootstrap fails with the journalctl/systemctl guidance and prints no join token.
  • Installer succeeds + API ready, and already-running k3s skip, remain green.

Light gate (go build, go vet, gofmt -l, go test ./..., SPDX) passes.

… installer exit

On a small VPS, k3s's slow first start (image unpack, datastore init) makes the
upstream installer's `systemctl start k3s` exceed systemd's readiness-notify
timeout, so the installer exits non-zero even though k3s comes up fine moments
later. bootstrap treated that exit as fatal and aborted with `installing k3s:
exit status 1` before deploying burrowd or printing the join token — a false
failure.

bootstrap now makes the k3s API answering — not the installer's exit code — the
success criterion. After running the installer it holds any non-zero exit and
polls the k3s API for readiness with a generous, configurable budget (default 4m,
--k3s-api-timeout), well past a slow first-start. If the API answers, bootstrap
proceeds, logging a non-zero installer exit as a warning. It fails only if the
API never answers within the budget, and that error carries the installer's exit
and points at `journalctl -xeu k3s.service` and `systemctl status k3s`. The
idempotent skip for an already-running k3s is unchanged.

Signed-off-by: Nicholas Phillips <nsphilli@gmail.com>
@incognick incognick added the pr-ready A PR is open and ready for review label Jul 4, 2026
@incognick incognick enabled auto-merge July 4, 2026 02:14
@incognick incognick added this pull request to the merge queue Jul 4, 2026
Merged via the queue into main with commit 4cd7702 Jul 4, 2026
5 checks passed
@incognick incognick deleted the fix/bootstrap-slow-k3s-start branch July 4, 2026 02:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-ready A PR is open and ready for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant