feat(app): add seeded live visual-QA sweep

[lalalune]

Relates to

• Follow-up visual evidence tooling for [observability] Audit runtime.reportError / RECENT_ERRORS coverage on critical device+cloud failure paths (boot, onboarding, cloud-login, connector-connect) #14415.

Definition of Done: full standard in PR_EVIDENCE.md.

• This PR targets develop and is rebased onto origin/develop with zero conflicts.
• bun install and bun run verify were not run; focused verification is listed below.
• A reviewer can confirm the change works from the command output and generated report summary below.

Sync with develop

• Rebased onto current origin/develop; PR is mergeable after the rebase.
• bun run verify not run because this is a narrow app-script change; focused checks and a live run are listed below.

Risks

Low. This adds a local operator/audit script and package script only. It does not change shipped runtime UI behavior. Main risks are false positives/false negatives in the visual harness, addressed by unit coverage and a strict live run.

Background

What does this PR do?

Adds bun run --cwd packages/app audit:live, a lightweight visual-QA sweep that points Playwright at an already-running app dev server and captures the states that matter for the MVP:

• fresh onboarding gate on desktop and mobile
• seeded post-onboarding shell and chat routes
• designed backend-unreachable error render when no backend is available
• mobile keyboard-adjacent composer state with typed text

The sweep writes screenshots plus report.json, runs each capture through the repo's visual-qa.mjs analyzer, and lets --strict fail on the no-blue brand invariant.

Review fixes included in this update:

• Preserved the newer branch behavior that writes default output under the existing ignored test/ui-smoke/output/visual-qa-live-output/ tree, so no .gitignore change is needed.
• Kept the no-blue gate on color_fractions.blue_fraction, which is the field emitted by visual-qa.mjs.
• Made navigation fail closed on page load errors and non-OK HTTP responses.
• Made mobile composer focus/type failures fail the capture instead of silently passing.
• Added a pure URL-join helper and unit coverage for it.

What kind of change is this?

Improvements: local visual-audit tooling for the app package.

Documentation changes needed?

No project documentation change required. The script has usage notes in its file header and is exposed through packages/app/package.json.

Testing

Where should a reviewer start?

Start with packages/app/scripts/visual-qa-live.mjs, then packages/app/scripts/visual-qa-live.test.mjs for the pure decision logic.

Detailed testing steps

bunx @biomejs/biome check packages/app/scripts/visual-qa-live.mjs packages/app/scripts/visual-qa-live.test.mjs packages/app/package.json
node --check packages/app/scripts/visual-qa-live.mjs packages/app/scripts/visual-qa-live.test.mjs
bunx vitest run packages/app/scripts/visual-qa-live.test.mjs
git diff --check

Result:

Checked 3 files in 99ms. No fixes applied.
Test Files  1 passed (1)
Tests       9 passed (9)

Live run:

ELIZA_UI_PORT=42138 bun run --cwd packages/app dev
bun run --cwd packages/app audit:live -- --base http://127.0.0.1:42138 --out /tmp/eliza-pr14818-visual-qa-live-a5 --strict

Result:

■ desktop-onboarding         blue=0.000
■ desktop-shell              blue=0.003
■ desktop-chat               blue=0.000
■ mobile-onboarding          blue=0.000
■ mobile-shell               blue=0.000
■ mobile-chat                blue=0.000
■ mobile-composer-focused    blue=0.000

PASS - no-blue invariant (ceiling 0.02); 7 states -> /tmp/eliza-pr14818-visual-qa-live-a5/report.json

Evidence Gate

Any change testable on the frontend is not mergeable without a video walkthrough, before/after screenshots, and logs. If you did not attach them, say why.

• Before full-page screenshots: N/A - this PR does not change a product UI surface; it adds a harness that captures existing surfaces.

• After full-page screenshots: N/A - no product UI surface changed; generated local verification screenshots are listed below but are not committed to the repo.

• Walkthrough video: N/A - CLI audit harness, no user-facing click-through flow changed.

• Backend logs: N/A - no backend code path changed; the seeded states intentionally render the backend-unreachable UI when no API is running.

• Frontend logs: N/A - no frontend runtime code changed; expected local dev-server warnings from the harness run are summarized below.

• Real-LLM trajectory: N/A - no agent/action/provider/prompt/model behavior changed.

• Domain artifacts: N/A - generated local report and screenshots are intentionally gitignored; exact verification output is summarized below.

Evidence Details

Real LLM-call trajectory

N/A - no agent/action/provider/prompt/model behavior changed.

Backend + frontend logs

Backend logs: N/A - no backend was started for this lane. That is intentional because the seeded post-onboarding states should surface the designed Backend Unreachable UI.

Frontend/dev-server log summary from the live run:

VITE v8.0.16 ready on http://localhost:42138/
http proxy error: /api/apps, /api/plugins, /api/conversations, /api/local-inference/*
[usePluginsSkillsState] failed to load plugins (error=API server unavailable)
[useSlashCommandController] Failed to load the slash-command catalog; slash menu will be empty

These warnings match the expected no-backend verification mode and the screenshot report still passed the strict no-blue invariant.

Screenshots (before / after) + video walkthrough

Before

N/A - no product UI surface changed.

After

Generated locally by the live run:

/tmp/eliza-pr14818-visual-qa-live-a5/desktop-onboarding.png
/tmp/eliza-pr14818-visual-qa-live-a5/desktop-shell.png
/tmp/eliza-pr14818-visual-qa-live-a5/desktop-chat.png
/tmp/eliza-pr14818-visual-qa-live-a5/mobile-onboarding.png
/tmp/eliza-pr14818-visual-qa-live-a5/mobile-shell.png
/tmp/eliza-pr14818-visual-qa-live-a5/mobile-chat.png
/tmp/eliza-pr14818-visual-qa-live-a5/mobile-composer-focused.png
/tmp/eliza-pr14818-visual-qa-live-a5/report.json

Walkthrough video

N/A - CLI audit harness, no interactive product flow changed.

Audio / voice walkthrough

N/A - no voice, transcript, TTS, or STT changes.

Known gaps / failures

• Full bun run verify was not run; focused checks and one strict live run were run instead.
• Port 2138 was already in use locally, so the live run used ELIZA_UI_PORT=42138 and passed --base http://127.0.0.1:42138.
• The fresh worktree lacked local workspace dist artifacts for config-time app startup. Built @elizaos/shared, @elizaos/cloud-routing, and runtime artifacts for @elizaos/core to start Vite. The worktree remained clean afterward; generated dist/ outputs are ignored.

[lalalune]

Reviewed, rebased onto develop, and pushed commit b601b34eb2 to this branch.

Fixes made while reviewing:

• Resolved the packages/app/.gitignore conflict by preserving the existing broad test/ui-smoke/output/ ignore and adding the actual generated output directory, visual-qa-live-output/.
• Fixed the strict no-blue gate to read color_fractions.blue_fraction, which is the field emitted by scripts/lib/visual-qa.mjs. The prior branch read color_fractions.blue, so blue offenders could be missed.
• Made live navigation fail closed on page.goto errors and non-OK HTTP responses.
• Made the mobile composer-focused state fail if the composer cannot be focused/typed instead of silently passing.
• Added buildCaptureUrl and unit coverage for URL joining.

Verification run locally:

bunx @biomejs/biome check packages/app/scripts/visual-qa-live.mjs packages/app/scripts/visual-qa-live.test.mjs packages/app/package.json packages/app/.gitignore
node --check packages/app/scripts/visual-qa-live.mjs packages/app/scripts/visual-qa-live.test.mjs
bunx vitest run packages/app/scripts/visual-qa-live.test.mjs
# 1 file / 9 tests passed

git diff --check
node scripts/check-pr-evidence.mjs --body-file /tmp/pr14818-body.md
# Evidence gate passed

I also ran the real live harness against a Vite dev server. Port 2138 was occupied locally, so I started this worktree on 42138 and passed that base explicitly:

ELIZA_UI_PORT=42138 bun run --cwd packages/app dev
bun run --cwd packages/app audit:live -- --base http://127.0.0.1:42138 --out /tmp/eliza-pr14818-visual-qa-live --strict

■ desktop-onboarding         blue=0.000
■ desktop-shell              blue=0.000
■ desktop-chat               blue=0.000
■ mobile-onboarding          blue=0.000
■ mobile-shell               blue=0.008
■ mobile-chat                blue=0.000
■ mobile-composer-focused    blue=0.000

PASS - no-blue invariant (ceiling 0.02); 7 states -> /tmp/eliza-pr14818-visual-qa-live/report.json

Visual spot-checks covered the mobile composer-focused screenshot and the desktop backend-unreachable screenshot. The expected no-backend Vite/API warnings appeared during the run.

[lalalune]

Update: after my first push, the PR branch moved to a5f62fc4, so I re-based the review work on that newer head instead of clobbering it. Final pushed head is 63718f4715.

I preserved the newer branch choice to write default output under the already-ignored packages/app/test/ui-smoke/output/visual-qa-live-output/ tree, so this final diff does not need a .gitignore change. The final follow-up adds the missing fail-closed pieces on top:

• buildCaptureUrl() with unit coverage.
• page.goto() errors are rethrown with state/url context.
• non-OK HTTP responses now fail the capture.
• the script waits for networkidle opportunistically but only tolerates that wait timing out.
• the mobile composer focus/type path still fails if no composer can be interacted with.

Final local verification on 63718f4715:

bunx @biomejs/biome check packages/app/scripts/visual-qa-live.mjs packages/app/scripts/visual-qa-live.test.mjs packages/app/package.json
node --check packages/app/scripts/visual-qa-live.mjs packages/app/scripts/visual-qa-live.test.mjs
bunx vitest run packages/app/scripts/visual-qa-live.test.mjs
# 1 file / 9 tests passed

git diff --check
node scripts/check-pr-evidence.mjs --body-file /tmp/pr14818-body.md
# Evidence gate passed

Strict live rerun against this final branch:

ELIZA_UI_PORT=42138 bun run --cwd packages/app dev
bun run --cwd packages/app audit:live -- --base http://127.0.0.1:42138 --out /tmp/eliza-pr14818-visual-qa-live-a5 --strict

■ desktop-onboarding         blue=0.000
■ desktop-shell              blue=0.003
■ desktop-chat               blue=0.000
■ mobile-onboarding          blue=0.000
■ mobile-shell               blue=0.000
■ mobile-chat                blue=0.000
■ mobile-composer-focused    blue=0.000

PASS - no-blue invariant (ceiling 0.02); 7 states -> /tmp/eliza-pr14818-visual-qa-live-a5/report.json

[lalalune]

Rebased onto current origin/develop (a00b5b1987) and pushed 32ee205d3b.

Review/fix notes:

• Kept the added hardening commit for URL joining and explicit navigation failures.
• Confirmed the live QA gate uses the analyzer's real color_fractions.blue_fraction field; the earlier blue field would have made strict mode blind to blue captures.
• Output now lands under the existing ignored packages/app/test/ui-smoke/output/visual-qa-live-output/ tree.

Verification on the rebased head:

• bun run --cwd packages/app test -- scripts/visual-qa-live.test.mjs -> 1 file / 9 tests passed
• bunx @biomejs/biome check packages/app/scripts/visual-qa-live.mjs packages/app/scripts/visual-qa-live.test.mjs packages/app/package.json -> clean
• git diff --check -> clean
• Live strict sweep against this worktree's shared Vite server (http://127.0.0.1:2970): node packages/app/scripts/visual-qa-live.mjs --base http://127.0.0.1:2970 --strict -> PASS, 7 states, blue=0.000 for every state, report at packages/app/test/ui-smoke/output/visual-qa-live-output/report.json.

[lalalune]

Adversarial review — not merging as-is (fabricated default inside the strict gate + vacuous seeded captures).

1. packages/app/scripts/visual-qa-live.mjs:152 (also :155/:252): (r.color_fractions?.blue_fraction ?? 0) > blueCeiling — a capture whose analyzer output lacks color_fractions (broken/renamed visual-qa.mjs shape) silently reads as zero blue and passes --strict. This is the banned "not loaded must never read as zero" pattern from the repo error policy, and visual-qa-live.test.mjs:384-386 locks the fabrication in as intended ("treats a missing color_fractions as zero blue"). A missing metric in a gate must be an error, not a pass.
2. No render-state assertion for seeded states: visual-qa-live.mjs:56-63 duplicates the steward_session_token / eliza:first-run-complete seed contract from test/ui-smoke/helpers/test-auth.ts with a "keep in sync" comment; nothing verifies desktop-shell/mobile-shell actually rendered the post-onboarding shell instead of the sign-in gate (OCR text is captured, never asserted). On key drift the sweep screenshots the wrong surface and still PASSes.
3. mobile-composer-focused contradicts the script's own rationale: visual-qa-live.mjs:127-136 uses seed: "fresh" + route / while the header (:39-41) says a fresh context only ever sees the gate; focusAndType (:160-167) then types into whatever first input the onboarding surface exposes — not the chat composer the PR body advertises, and nothing asserts which input received the text.
4. Minor: visual-qa-live.mjs:229-231 empty .catch(() => {}) without an // error-policy:J<N> annotation.

Positives worth keeping: fail-closed navigation on load errors/non-OK HTTP, the test file does run in CI, output dir already gitignored. Fix 1-3 and this is mergeable.

[lalalune]

Blocking merge as-is. The harness needs the maintainer-blocking issues resolved before this can land: missing color_fractions should fail instead of passing, seeded states should assert the intended surface, mobile composer focus should target the composer, and the empty .catch needs explicit policy handling. The PR also needs durable PR evidence rather than only local /tmp artifact paths.

[lalalune]

Review hold: the mobile composer capture does not actually target the chat composer.

mobile-composer-focused is configured as a fresh seed at route /, and the helper clicks the first generic textarea/contenteditable/text input. That can hit onboarding/sign-in UI or fail depending on the gate, not the /chat composer. The test only checks that a focused mobile state exists.

Please seed/navigate to the chat route, select the actual composer, and attach/verify live artifacts if this is meant to satisfy app visual proof.

[lalalune]

Deep architectural review + fix-forward (3a0b376). Merged.

Why it's the right shape: this is not a second audit mechanism — audit:app post-processes its own prebuilt route walk, mvp:visual-verify post-processes audit:app output, and this tool is the missing third leg: capture against an already-running server through the same single analyzer (lib/visual-qa.mjs). One analyzer, three entry points. The seed duplicates the canonical test-auth.ts contract only because a plain node script can't import the .ts helper, and both keys were verified against the renderer's source of truth (steward_session_token, eliza:first-run-complete).

Fixes applied before merge (the earlier adversarial-review blockers):

1. aggregateVerdict now fails closed on an unmeasured blue_fraction (offender reason: "blue_fraction not measured") instead of reading analyzer-shape drift as blue=0 — the previous behavior violated the repo's "not loaded must never read as zero" doctrine, and the old test enshrined it; the test now asserts fail-closed.
2. New seed-vacuity gate (seedDriftOffenders + changeMetric): the seeded shell must differ materially from the onboarding gate per viewport, so a drifted seed contract hard-fails instead of silently re-capturing the gate 7 times and passing.
3. Composer capture prefers the canonical chat-composer-textarea testid (generic input fallback kept for gates); still fail-closed when nothing focuses.
4. The networkidle settle catch is annotated error-policy:J6 (SPAs with long-poll/WS never reach networkidle; the fixed wait is the real settle).

Live proof (against the running dev server on :2138, isolated contexts): 7/7 states captured, strict PASS, blue=0.000 everywhere; seed deltas desktop 0.0557 / mobile 0.0974 (> 0.02 floor — the seed demonstrably left the gate: seeded states render the designed Backend Unreachable three-state error, and the composer capture's OCR contains the typed text "remind me to call the pharmacy"). Unit suite 12/12; Biome clean.

[lalalune]

Verification update from this pass:

• Hardened audit:live so a bad state records a failure row (and best-effort failure screenshot) instead of throwing before report.json is written.
• Removed the generic input fallback from focused composer capture; it now requires data-testid=chat-composer-textarea.
• Added mobile-chat-composer-focused so the matrix covers the signed-in /chat composer, not only the onboarding composer.
• Local checks passed:
  • node --check packages/app/scripts/visual-qa-live.mjs && node --check packages/app/scripts/visual-qa-live.test.mjs
  • bunx @biomejs/biome check packages/app/scripts/lib/visual-qa.mjs packages/app/scripts/visual-qa-live.mjs packages/app/scripts/visual-qa-live.test.mjs packages/app/package.json
  • bunx vitest run packages/app/scripts/visual-qa-live.test.mjs packages/app/scripts/lib/visual-qa.test.ts (2 files, 27 tests)
  • git diff --check
• Live visual QA passed against http://127.0.0.1:42138 with 8 states. Report: /tmp/eliza-pr14818-visual-qa-live-final2/report.json. Summary: pass=true, offenders [], seed drift [], desktop seed delta 0.9547, mobile seed delta 0.9333, all measured blue_fraction=0.
• Manual screenshot review: mobile-chat-composer-focused.png shows the signed-in half-detent chat sheet with the typed reminder preserved in the canonical composer and send affordance visible; mobile-shell.png shows the designed backend-unreachable error state with orange accent and no blue.
• Full app visual audit passed locally: bun run --cwd packages/app audit:app (365 passed, ~11m). Report still records existing non-strict audit debt: broken=0, needs-work=11, needs-eyeball=25, hover probe failures on builtin transcripts Join meeting; no blue/console/minimalism/density failures.

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.