Skip to content
This repository was archived by the owner on May 13, 2026. It is now read-only.

fix(nightly-qa-dogfood): bootstrap node before probing auto-identity - #928

Merged
shiba4life merged 1 commit into
mainfrom
fix/qa-dogfood-readiness-probe
May 7, 2026
Merged

fix(nightly-qa-dogfood): bootstrap node before probing auto-identity#928
shiba4life merged 1 commit into
mainfrom
fix/qa-dogfood-readiness-probe

Conversation

@shiba4life

Copy link
Copy Markdown
Collaborator

Summary

The 2026-05-06 nightly QA dogfood run failed at boot:

[qa-dogfood] FAIL: [node-a] backend not ready within 60s: http://localhost:9101/api/system/auto-identity
[qa-dogfood] boot_stack node-a failed

PR #908 (refactor(identity): load_or_generate -> provision) made /api/setup/bootstrap the canonical mint path; until it has run, /api/system/auto-identity returns 503 NotProvisioned. The dogfood harness used /api/system/auto-identity as its readiness probe, so boot_stack timed out at 60s on every run.

This is the sibling fix to PR #924 (scripts/test-sample-ingestion.sh), but for scripts/qa-harness-dogfood.sh instead.

Why Option B (bootstrap before probing) and not Option A (swap to a liveness endpoint)

The harness needs an identity by the time boot_stack returns:

  • fetch_user_hash (line 331) calls /api/system/auto-identity and jq -r '.user_hash' — would crash on the 503 body.
  • The org-sync leg (line 858) explicitly probes /api/system/auto-identity to confirm node B is alive.

Both depend on bootstrap having happened. So boot_stack now does the bootstrap itself rather than offloading it to the harness body. To keep readiness side-effect-free up to the bootstrap call, the new shape is:

  1. wait_http /api/health — unauthenticated, served the moment actix binds (http_server.rs::configure_health_route).
  2. POST /api/setup/bootstrap with {"name":"qa-dogfood-<label>"} and a 3-try retry around port-bind races (mirrors PR test(nightly): bootstrap node before probing auto-identity #924's retry shape).
  3. wait_http /api/system/auto-identity — confirms the provisioned identity is queryable. fetch_user_hash (line 331) and the line-858 check now both succeed.

report.md section markers (## Set-org-hash leg, ## Org-sync leg, - **Status:** FAIL) are unchanged, so the nightly-qa-dogfood.yml awk-scan at lines 248-265 keeps working as-is.

Verification

$ bash scripts/qa-harness-dogfood.sh --per-source-count 5 --skip-org
[qa-dogfood] [node-a] backend listener ready: http://localhost:9101/api/health
[qa-dogfood] [node-a] bootstrap ok (HTTP 200)
[qa-dogfood] [node-a] backend ready: http://localhost:9101/api/system/auto-identity
[qa-dogfood] [node-a] vite ready: http://localhost:5173/
...
[qa-dogfood] verdict: PASS (6 pass / 0 fail of 6 sources; set-org-hash: PASS; org leg: SKIP)

(--skip-org is local convenience; CI runs both legs and the same boot path is exercised for node-b.)

Test plan

  • Local smoke (--per-source-count 5, --skip-org) verdict PASS, set-org-hash PASS.
  • Manual gh workflow run nightly-qa-dogfood.yml after merge — conclusion success.
  • Issue Nightly QA dogfood regression tracker #671 closed once nightly is green.

Closes #671.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Sibling fix to PR #924, but for the QA dogfood harness instead of the
sample-ingestion script. PR #908 (refactor(identity): load_or_generate
-> provision) made /api/setup/bootstrap the canonical mint path; until
it has run, /api/system/auto-identity returns 503 NotProvisioned. The
2026-05-06 nightly therefore failed at boot:

  [qa-dogfood] FAIL: [node-a] backend not ready within 60s:
                     http://localhost:9101/api/system/auto-identity

boot_stack used /api/system/auto-identity as its readiness probe, which
no longer answers 200 on a fresh node. Replace it with a two-step:

1. Wait for /api/health (unauthenticated, side-effect-free, served the
   moment actix binds — see http_server.rs configure_health_route).
2. POST /api/setup/bootstrap with {name:"qa-dogfood-<label>"} and a
   3-try retry around port-bind races (mirrors PR #924 retry shape).
3. Re-probe auto-identity to confirm the provisioned identity is
   queryable, so fetch_user_hash and the org-sync leg's auto-identity
   check at line 858 still work.

Verified locally via:
  bash scripts/qa-harness-dogfood.sh --per-source-count 5 --skip-org
Verdict: PASS (6 pass / 0 fail of 6 sources; set-org-hash: PASS).

Closes #671.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@shiba4life
shiba4life enabled auto-merge May 7, 2026 03:16
@shiba4life
shiba4life added this pull request to the merge queue May 7, 2026
Merged via the queue into main with commit ddf03af May 7, 2026
12 checks passed
@shiba4life
shiba4life deleted the fix/qa-dogfood-readiness-probe branch May 7, 2026 03:27
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nightly QA dogfood regression tracker

1 participant