fix(provision/docker): probe host apiserver before declaring ready

[solsson]

Summary

• pkg/provision/docker/docker.go: add waitForHostAPIServer, called between waitForKubeconfig and the "k3s ready" log. It polls kubectl get --raw=/readyz against the merged context (1s interval, 60s deadline) so a not-yet-bound port forward and a still-starting apiserver are both retried before envoygateway.Install (or any other host-side kubectl call) runs.
• Updates the waitForKubeconfig doc comment, which had asserted the opposite of reality ("k3s.yaml means apiserver is ready to accept connections").

Why

ystack acceptance on ubuntu-latest has been failing every run on the docker provider with:

INFO docker/docker.go:208  k3s ready
INFO envoygateway/install.go:119  applying envoy-gateway install manifest
error: ... dial tcp 127.0.0.1:6443: connect: connection refused

(Yolean/ystack#76; example run: actions/runs/25220092901). The race is deterministic — every fresh provision shows ~2s between "starting docker" and "k3s ready", with kubectl failing within 1s. The ystack-side 4×retry workaround can't paper over it because each retry tears the cluster down and reproduces the same race.

The qemu provisioner is unaffected: its ssh-driven k3s install script returns only after the apiserver is fully up, so "k3s ready" there is already truthful.

Test plan

• go test ./... passes
• go test -tags 'e2e,docker' -run TestDocker_ProvisionTeardown ./e2e/ passes locally; "k3s ready" now arrives ~10s after "starting docker" (was ~2s) and the subsequent envoy-gateway install runs without retry
• CI e2e (e2e,docker) job green

[solsson]

Review from maintainer agent

Verdict

LGTM. Tight, focused change against a real, deterministic race that ystack acceptance has been hitting. Worth merging.

What I liked

• Honest doc-comment fix on waitForKubeconfig (line 372–376). The old comment claimed "k3s.yaml means apiserver is ready to accept connections" — that's exactly the false invariant the bug hinges on. Updating it as part of the fix is the right thing.
• Symmetric error handling. The new waitForHostAPIServer failure path captures container logs the same way waitForKubeconfig already does (dlogs, _ := dockerexec.Logs(...)). Reading the diff feels like one author wrote both.
• /readyz over /healthz. Correct choice — /readyz gates readiness (and includes the apiserver having advanced past "listening" to "serving"), /healthz doesn't.
• ctx.Done() honored in the poll loop. A caller cancellation interrupts cleanly.
• lastErr instead of returning the most recent transient. The first 30 connection-refused errors aren't useful to surface; the final state when the deadline trips is.

Suggestions (none blocking)

1. The qemu provisioner is "unaffected" by timing, not by guarantee

The PR description: "its ssh-driven k3s install script returns only after the apiserver is fully up, so 'k3s ready' there is already truthful."

That's accurate for "apiserver up inside the VM," but the host-side port forward is still a separate readiness step. qemu happens not to race today because its full-VM boot path runs much longer than the docker container's, by which time SLIRP forwarding has been bound for a while. On a slow host or with very fast k3s startup, qemu could in principle hit the same race.

I wouldn't expand the scope of this PR, but waitForHostAPIServer is generic enough that lifting it to a provision/internal (or provision/k8sready) helper and calling it from pkg/provision/qemu/qemu.go:309 after the kubeconfig merge is a natural follow-up. Filing as a tracking issue would be enough for now.

2. A start-of-wait log line would help future debugging

The function is silent until success or 60s timeout. When someone in the future stares at:

INFO docker/docker.go:208  waiting for host apiserver readiness   <-- new
INFO docker/docker.go:224  k3s ready

it's immediately obvious which step took the time. Today on success there's no log between "merge kubeconfig" and "k3s ready," so a 30s wait looks like a 30s mystery. One c.logger.Info("waiting for host apiserver", zap.String("context", c.cfg.Context)) at the top of waitForHostAPIServer is enough.

3. Why shell out to kubectl instead of a direct HTTP probe

You're forking kubectl up to 60 times in the worst case (1s interval × 60s deadline). It's not expensive in absolute terms — this whole loop only runs on a cold provision — but a client-go REST GET against the merged context's server URL would avoid the fork entirely and be one less moving piece on the host (kubectl version drift, PATH order, etc.).

I'd lean toward keeping kubectl since the rest of the provisioner already shells out (and envoygateway.Install does the same), so consistency wins here. Just flagging it because the comment on line 380 implies the choice was deliberate without spelling out why the shell-out was preferred — a one-line "matches envoygateway.Install which also drives the host kubeconfig via kubectl" would close the loop.

4. Polling interval choice

1s is fine. If the deterministic gap is "~2s (current)" vs "~10s (after this fix)" as the PR says, an aggressive interval doesn't buy much. If you ever want to tighten the floor, 200–500ms would catch the median case faster without changing the deadline logic.

5. Unit test coverage

The PR plan calls out TestDocker_ProvisionTeardown (e2e) and CI. That's the right level for a real-network probe loop, but a tiny in-process unit test against a fake kubectl-on-PATH (write a shell script to t.TempDir(), prepend to PATH via t.Setenv, have it print fail-then-success) would lock in:

• The 60s deadline is honored (run with the fake always-failing, expect error).
• The context cancellation path returns ctx.Err() (not the 60s timeout error).

Optional, but would make a refactor that accidentally drops select { <-ctx.Done(): } fail loudly instead of silently regressing to a non-cancellable wait.

File:line cross-reference

• pkg/provision/docker/docker.go:208–222 — Provision flow now has a new readiness gate; reads cleanly.
• pkg/provision/docker/docker.go:354–378 — waitForKubeconfig doc updated.
• pkg/provision/docker/docker.go:380–410 — waitForHostAPIServer. Single-purpose, well-commented.

Nit

probe.CombinedOutput() discards the separation of stdout/stderr; fine here since we're just looking at exit code + last-error context. No change suggested.

---

Net: small, surgical, addresses the failing CI mode head-on, and doesn't expand scope. Ship it.