feat(v1.9): aqa report CLI verb (sub-task 2)#53
Merged
lopadova merged 5 commits intoMay 20, 2026
Conversation
Cables `@aqa/reporter` into a real CLI verb. README quick-start has referenced `aqa report` since v0.0.1 but the verb itself was never wired — `bunx aqa report` errored at parse time. CLI surface: aqa report [--run-id <id>] [--format md|json|both] Behaviour: - `--run-id` omitted → defaults to the latest run directory (lexical sort on run-id, which has an ISO-like prefix so newest wins) - reads `.aqa/runs/<id>/events.jsonl` + `findings.jsonl` - reconstructs a schema-conformant Run object from the first `run_started` and last `run_finished` events (no new sidecar files — reports stay replayable from the audit trail alone) - writes `report.md` (renderMarkdown) + `report.json` (renderJson) into the same run directory; the JSON shape is the stable one the admin UI already consumes - structured errors for: missing runs dir, missing run-id, malformed JSONL, schema-invalid findings - `--format md|json` opt-out for each artifact Tests: 10 new node:test cases in packages/kit/test/report-cmd.test.ts cover happy path (5: full render, latest-default, md-only, json-only, zero-findings) and error cases (5: missing dir, missing run-id, empty dir, malformed JSONL, schema-invalid finding). Synthetic run dirs match @aqa/schemas Run/Finding exactly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d579777fa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const scenariosRun = readPayloadNumber(finished, 'scenarios_run') ?? 0; | ||
| const totalsFindings = readPayloadNumber(finished, 'findings') ?? findingsCount; | ||
|
|
||
| const state: Run.Run['state'] = finished ? 'succeeded' : 'running'; |
There was a problem hiding this comment.
Do not mark every run with a run_finished event as succeeded. runRun writes run_finished even when it later returns ok: false (pack/scenario/runtime errors), so this line makes failed runs appear successful in report.json/report.md, which can hide broken release-gate runs in downstream dashboards.
Useful? React with 👍 / 👎.
Comment on lines
+137
to
+139
| // Run IDs include an ISO-like prefix; lexical sort orders by recency. | ||
| .sort(); | ||
| return dirs.at(-1); |
There was a problem hiding this comment.
The default-run selection assumes all run IDs have an ISO-like sortable prefix, but aqa run --seed produces hash-based IDs (run-<sha>). In a mixed runs directory, lexical sort can pick an older seeded run instead of the most recent run, so aqa report may generate a report for the wrong execution unless --run-id is always provided.
Useful? React with 👍 / 👎.
| } | ||
|
|
||
| function readJsonl(path: string): Array<Record<string, unknown>> { | ||
| if (!existsSync(path)) return []; |
There was a problem hiding this comment.
Returning an empty array when events.jsonl or findings.jsonl is absent causes runReport to succeed on incomplete/corrupt run directories and emit synthetic output (for example, fallback timestamps and unknown metadata). For interrupted or partially deleted runs, this silently fabricates reports instead of surfacing artifact loss to the caller.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
Adds a long-documented but previously unwired aqa report CLI verb to @aqa/kit, backed by @aqa/reporter, so users can render report.md and/or report.json from the on-disk run audit trail (events.jsonl + findings.jsonl).
Changes:
- Introduces
runReport({ root, runId?, format? })to load a run directory, reconstruct aRun, and render Markdown/JSON reports. - Wires
reportinto the CLI with--run-idand--formatflags and help text. - Adds a dedicated node:test suite for the command and updates kit deps/test script accordingly.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/kit/src/commands/report.ts | Implements runReport, including run selection, JSONL parsing, run reconstruction, and report writes. |
| packages/kit/src/cli/aqa.ts | Adds the report command router branch plus --run-id / --format support and help text. |
| packages/kit/test/report-cmd.test.ts | Adds unit tests covering happy paths and key error cases for aqa report. |
| packages/kit/package.json | Adds @aqa/reporter dependency and includes the new test file in the kit test script. |
| bun.lock | Locks the new workspace dependency for @aqa/kit. |
Comment on lines
+69
to
+72
| const runDir = join(runsRoot, runIdResolved); | ||
| if (!safeIsDir(runDir)) { | ||
| return { ok: false, error: `report: run directory not found: ${runDir}` }; | ||
| } |
Comment on lines
+191
to
+195
| const scenariosRun = readPayloadNumber(finished, 'scenarios_run') ?? 0; | ||
| const totalsFindings = readPayloadNumber(finished, 'findings') ?? findingsCount; | ||
|
|
||
| const state: Run.Run['state'] = finished ? 'succeeded' : 'running'; | ||
|
|
Comment on lines
+150
to
+152
| function readJsonl(path: string): Array<Record<string, unknown>> { | ||
| if (!existsSync(path)) return []; | ||
| const text = readFileSync(path, 'utf8'); |
Comment on lines
+108
to
+111
| if (format === 'md' || format === 'both') { | ||
| const mdPath = join(runDir, 'report.md'); | ||
| writeFileSync(mdPath, renderMarkdown({ run, findings }), 'utf8'); | ||
| written.push(mdPath); |
Comment on lines
+86
to
+92
| function writeFindings(runDir: string, count: number): void { | ||
| const lines: string[] = []; | ||
| for (let i = 0; i < count; i++) { | ||
| const finding = { | ||
| schema_version: '1', | ||
| id: `AQA-2026-${String(i + 1).padStart(4, '0')}`, | ||
| run_id: RUN_ID, |
5 actionable comments, all addressed:
1. P1 — state was unconditionally 'succeeded' when run_finished was
present. But runRun writes run_finished even on most failure paths
(pack_errors / scenario_errors / missing_scenarios / unsafe_paths /
runtime_errors / zero scenarios). Now derives state by inspecting the
payload counters: any non-zero error counter OR zero scenarios → 'failed';
no run_finished → 'running'; else 'succeeded'. Added 3 regression tests.
2. P2 — latest-run selection used lexical sort on directory name, which
silently picks the wrong run when `aqa run --seed` produces hash-based
IDs (run-<sha>). Now uses statSync.mtimeMs (most recent wins) with the
directory name as a deterministic tie-breaker. Added a regression test
that intentionally writes the lexically-earlier name LAST.
3. P1 — readJsonl returned [] for missing files. That made `aqa report`
succeed on a corrupted/incomplete run dir and emit a synthetic empty
report. Now both events.jsonl and findings.jsonl are required up-front
with explicit existence checks; readJsonl assumes existence. Added 2
regression tests.
4. Path-traversal — --run-id was forwarded to join() without validation,
so a value like '../..' could read/write files outside .aqa/runs.
Added a LongSlug-shaped regex (/^[a-z0-9-]{1,80}$/) at the CLI
boundary that rejects traversal and any non-slug input. Added 2
regression tests.
5. writeFileSync wasn't wrapped — a read-only FS or disk-full would
bubble through the CLI's top-level unhandled-error path instead of
returning a structured ReportErr. Wrapped both writes in a single
try/catch.
Test fixture bug also fixed: writeFindings now accepts an optional runId
parameter (default RUN_ID) so findings written for ALT_RUN_ID-style
directories carry the matching run_id, not a stale constant.
17 tests now pass (10 original + 7 new regression tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment on lines
+24
to
+28
| // LongSlug (per @aqa/schemas) cap. Reject anything that would let `runId` | ||
| // escape `.aqa/runs/` via traversal (`../..`, absolute paths, backslashes). | ||
| // The schema's own LongSlug also enforces ^[a-z0-9-]+$ — we mirror that | ||
| // here rather than import to keep the report command self-contained. | ||
| const RUN_ID_RE = /^[a-z0-9-]{1,80}$/; |
Comment on lines
+86
to
+89
| const runDir = join(runsRoot, runIdResolved); | ||
| if (!safeIsDir(runDir)) { | ||
| return { ok: false, error: `report: run directory not found: ${runDir}` }; | ||
| } |
Comment on lines
+227
to
+229
| if (obj && typeof obj === 'object' && !Array.isArray(obj)) { | ||
| out.push(obj as Record<string, unknown>); | ||
| } |
Comment on lines
+275
to
+276
| // Deterministic dummy: this report path doesn't recompute the config | ||
| // hash from disk. The admin UI keys on `run.id`, not `config_hash`. |
4 actionable comments, all addressed:
1. RUN_ID_RE diverged from real LongSlug. Previous /^[a-z0-9-]{1,80}$/
capped at 80 (vs schema's 256) and allowed leading/trailing dashes
plus `--` runs that real LongSlug rejects — so a malformed --run-id
could slip past this guard and fail later at Finding.parse(). Now
uses the canonical SlugPattern /^[a-z0-9](?:-?[a-z0-9])*$/ + a
separate 256-char length check so error messages distinguish "bad
characters" from "too long". Added regression tests for both.
2. safeIsDir() followed symlinks via statSync. An attacker (or a prior
run) leaving a symlink under .aqa/runs/<runId> pointing outside the
project would let report.md/report.json land anywhere the link
pointed. Now explicit lstatSync().isSymbolicLink() check refuses
symlinked run dirs. Added a regression test (skipped on Windows when
symlink perms aren't available).
3. readJsonl() silently dropped any non-empty line that parsed as valid
JSON but wasn't a plain object (null, [], "x", 42). For events.jsonl
that turned corruption into a seemingly-successful report. Now
strict: throws "expected a JSON object, got <type>" with explicit
null branch (typeof null === 'object' in JS so the naive type check
needed a guard). Added 2 regression tests (null + array).
4. Misleading comment claimed admin UI keys on run.id and not
config_hash. Admin actually renders config_hash in replay copy, so
the all-zeros placeholder will be visible. Replaced with an honest
"treat as not-computed sentinel" note so future maintainers don't
try to fix a bug that isn't there.
22 tests now pass (was 17 → +5 new regression tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment on lines
+215
to
+223
| const candidates: Array<{ name: string; mtimeMs: number }> = []; | ||
| for (const name of entries) { | ||
| const dir = join(runsRoot, name); | ||
| try { | ||
| const st = statSync(dir); | ||
| if (st.isDirectory()) { | ||
| candidates.push({ name, mtimeMs: st.mtimeMs }); | ||
| } | ||
| } catch { |
Comment on lines
+173
to
+187
| try { | ||
| if (format === 'md' || format === 'both') { | ||
| const mdPath = join(runDir, 'report.md'); | ||
| writeFileSync(mdPath, renderMarkdown({ run, findings }), 'utf8'); | ||
| written.push(mdPath); | ||
| } | ||
| if (format === 'json' || format === 'both') { | ||
| const jsonPath = join(runDir, 'report.json'); | ||
| writeFileSync( | ||
| jsonPath, | ||
| `${JSON.stringify(renderJson({ run, findings }), null, 2)}\n`, | ||
| 'utf8', | ||
| ); | ||
| written.push(jsonPath); | ||
| } |
Comment on lines
+286
to
+304
| const started = pickEvent(events, 'run_started'); | ||
| const finished = pickEvent(events, 'run_finished'); | ||
|
|
||
| const startedAt = readString(started, 'ts') ?? new Date(0).toISOString(); | ||
| const finishedAt = readString(finished, 'ts'); | ||
| const profile = readPayloadString(started, 'profile') ?? 'unknown'; | ||
| const project = readPayloadString(started, 'project') ?? 'unknown'; | ||
| const scenariosRun = readPayloadNumber(finished, 'scenarios_run') ?? 0; | ||
| const totalsFindings = readPayloadNumber(finished, 'findings') ?? findingsCount; | ||
|
|
||
| const state: Run.Run['state'] = deriveState(finished, scenariosRun); | ||
|
|
||
| const run: Run.Run = { | ||
| schema_version: '1', | ||
| id: runId, | ||
| started_at: startedAt, | ||
| ...(finishedAt ? { finished_at: finishedAt } : {}), | ||
| state, | ||
| project, |
6 tasks
3 actionable comments, all addressed: 1. latestRunId considered any directory entry, including symlinks and non-run subdirs. A `.aqa/runs/readme-stash/` with the newest mtime would shadow real runs. Now filters: name matches RUN_ID_RE + not-a-symlink (lstat) + contains events.jsonl. Added a regression test that pollutes `.aqa/runs/` with a non-run subdir and asserts the latest REAL run wins. 2. Per-file symlink protection. The earlier symlink guard only checked the run directory itself. A pre-existing `report.md` or `report.json` symlink would let writeFileSync follow the link and redirect the write outside the project. Now lstat-checks each target file before writing; refuses if it's a symlink. Added a regression test that creates a symlink to an outside file, attempts the report, and asserts both (a) refusal and (b) attacker-controlled file remains byte-identical. 3. reconstructRun could in principle produce Run-schema-incompatible output if the audit chain itself is malformed. Now validates the reconstructed object via Run.Run.safeParse before handing it to the renderers — surfaces a structured error instead of writing a report.json that the admin UI would silently reject. Also flipped the import of Run from type-only to value (needed for safeParse). 24 tests now pass (was 22 → +2 regression tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
| const profile = readPayloadString(started, 'profile') ?? 'unknown'; | ||
| const project = readPayloadString(started, 'project') ?? 'unknown'; | ||
| const scenariosRun = readPayloadNumber(finished, 'scenarios_run') ?? 0; | ||
| const totalsFindings = readPayloadNumber(finished, 'findings') ?? findingsCount; |
Comment on lines
+119
to
+147
| const eventsPath = join(runDir, 'events.jsonl'); | ||
| // Missing artifacts → fail-fast. A silent empty report on a corrupted | ||
| // run dir hides the real problem (the run never wrote its audit trail). | ||
| if (!existsSync(eventsPath)) { | ||
| return { | ||
| ok: false, | ||
| error: `report: events.jsonl is missing at ${eventsPath} — run is incomplete or corrupted`, | ||
| }; | ||
| } | ||
| let events: ReadonlyArray<Record<string, unknown>>; | ||
| try { | ||
| events = readJsonl(eventsPath); | ||
| } catch (e) { | ||
| return { | ||
| ok: false, | ||
| error: `report: cannot read events.jsonl: ${e instanceof Error ? e.message : String(e)}`, | ||
| }; | ||
| } | ||
|
|
||
| const findingsPath = join(runDir, 'findings.jsonl'); | ||
| if (!existsSync(findingsPath)) { | ||
| return { | ||
| ok: false, | ||
| error: `report: findings.jsonl is missing at ${findingsPath} — run is incomplete or corrupted`, | ||
| }; | ||
| } | ||
| let findings: readonly Finding.Finding[]; | ||
| try { | ||
| findings = readJsonl(findingsPath).map((raw, idx) => { |
Comment on lines
+323
to
+334
| // Best-effort reconstruction from the audit chain — see file header. The | ||
| // reporter only reads a small subset of Run fields, but we still build the | ||
| // full schema-conformant object so the JSON report stays valid for the | ||
| // admin UI and any external dashboard. | ||
| const { runId, runDir, events, findingsCount } = input; | ||
| const started = pickEvent(events, 'run_started'); | ||
| const finished = pickEvent(events, 'run_finished'); | ||
|
|
||
| const startedAt = readString(started, 'ts') ?? new Date(0).toISOString(); | ||
| const finishedAt = readString(finished, 'ts'); | ||
| const profile = readPayloadString(started, 'profile') ?? 'unknown'; | ||
| const project = readPayloadString(started, 'project') ?? 'unknown'; |
Comment on lines
+366
to
+372
| probes: 0, | ||
| llm_tokens_in: 0, | ||
| llm_tokens_out: 0, | ||
| llm_cost_usd: 0, | ||
| }, | ||
| artifact_dir: runDir, | ||
| }; |
lopadova
merged commit af05b79
into
task/v1.9-junior-quickstart-truthing
8 checks passed
lopadova
added a commit
that referenced
this pull request
May 21, 2026
…sub-task 5) (#56) The v1.8 README claimed `bun add -d agentic-qa-kit` worked (npm 404), `aqa install-agent-files` worked (CLI verb didn't exist), `aqa report` worked (CLI verb didn't exist), and `bun --filter @aqa/admin dev` was how juniors boot the admin (only works inside the monorepo). v1.9 closes those gaps in code (PRs #52/#53/#54/#55); this PR brings the docs back in sync. ## README - New step 2: `.npmrc` snippet for GitHub Packages auth + `GITHUB_TOKEN` export. Public packages on GH Packages still require auth — this is the single biggest junior trap and now has its own step. - Step 3: `bun add -d @padosoft/agentic-qa-kit` (replaces the npm-404 `agentic-qa-kit`). - Step 4 bundles `init` + `doctor` + `validate` so juniors verify the install before moving on. - Step 5: `install-agent-files --targets` — now matches a real verb. - Step 6: explicit `risk-map.yaml` edit with a worked example invariant. - Step 7: `aqa run --profile smoke` with explicit pointer to the per-run artifacts under `.aqa/runs/<id>/`. - Step 8: `aqa report` with format flag matrix. - Step 9: `aqa admin` — single command boots SPA + API (replaces `bun --filter @aqa/admin dev`). - Step 10: reproduce from artifacts. - Footer: `bun run e2e:ecosystem` pointer for monorepo contributors. - "How you use it" 8-step model reflects the new ordering. - Roadmap rows added for v1.8 and v1.9. - Status line bumped: `v1.7 current` → `v1.9 current`. ## docs/getting-started.md Section 1 new: GitHub Packages auth setup with PAT scope + .npmrc + env export. Section 2: bun add @padosoft/agentic-qa-kit. Sections 3–9: init/doctor/validate/install-agent-files/risk/run/report/admin in order, each with the real flag matrix. Closing "Where to go next" links the new PACK-AUTHORING doc. ## CHANGELOG Added [1.9.0] entry covering the 5 PRs + the kit↔server cycle fix + the bundle/publish-prep mechanics. Backfilled [1.8.1] / [1.8.2] / [1.8.3] which had only been recorded in PROGRESS.md. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8 tasks
lopadova
added a commit
that referenced
this pull request
May 21, 2026
…t + admin verbs + GitHub Packages publish (macro) (#57) * feat(v1.9): aqa install-agent-files CLI verb (sub-task 1) (#52) * feat(v1.9): add `aqa install-agent-files` CLI verb (sub-task 1) Cables the existing `renderForTargets()` from @aqa/adapters into a real CLI verb. The README/getting-started have referenced this command since v0.0.1 but the verb itself was never wired — `bunx aqa install-agent-files --targets ...` errored out at parse time. CLI surface: aqa install-agent-files --targets claude,codex,gemini,copilot [--project-name <slug>] [--force] [--dry-run] Behaviour: - --targets: csv or repeated `--targets <single>`; case-insensitive; deduped - unknown target → fail-fast, no files written (rejects "claude,mistral" before touching disk) - empty/whitespace-only targets list → fail-fast with usage message - project name defaults to slugified basename of cwd; --project-name overrides (also slugified to satisfy @aqa/schemas Project.name = Slug) - existing files preserved unless --force (writeFileSafe contract) - --dry-run reports planned writes without touching disk Tests: 13 new node:test cases in packages/kit/test/install-agent-files-cmd.test.ts covering happy path (4 targets), targets validation (5 cases including dedup + casing), and overwrite semantics (skip / force / dry-run). Drive-by: doctor.ts hint no longer says "(Task 4)" — the verb exists now, the suggestion is the full command a junior can paste. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v1.9): address Copilot iter 1 on install-agent-files PR #52 4 actionable comments, all addressed: 1. parseTargets array form skipped trim+empty-filter. Inputs like `['claude ', '', 'codex']` would surface a confusing `unknown target ' '` error. Unified normalization across both CSV and array forms; added a regression test. 2. Help text claimed `copilot-instructions.md` but the copilot adapter writes to `.github/copilot-instructions.md`. Fixed the path in HELP so juniors know where to look. 3. Test fixture used `mkdtempSync(join(tmpdir(), 'My Junior Project '))` — Windows rejects path components ending in a space, which would crash the test before any assertion ran. Changed the trailing space to a hyphen so the prefix remains slug-meaningful but is portable. 4. `lastPathSegment` + `slugify` were duplicated between init.ts and install-agent-files.ts. Extracted to `cli-utils.ts` and imported from both — single source of truth for project-name slugging. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v1.9): address Copilot iter 2 on install-agent-files PR #52 2 actionable comments, both addressed: 1. slugify() could return strings longer than @aqa/schemas Slug.max(64). `runInit` and `runInstallAgentFiles` both feed this value into `.aqa/project.yaml` (and `risk-map.yaml`), which the Slug schema rejects at `aqa validate` time. Cap to 64 chars after normalization and re-trim trailing dashes so a cap that lands inside a `-` run doesn't produce an illegal `-`-terminated slug. Added a regression test that feeds 70 'a's and confirms the embedded slug is ≤64. 2. KNOWN_TARGETS hardcoded `['claude','codex','gemini','copilot']` which duplicates the canonical list in @aqa/adapters.adapters. Adding a new adapter (e.g. opencode) would render but be rejected at parseTargets time. Now derived from `adapters.map(a => a.target).sort()` — single source of truth. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v1.9): aqa report CLI verb (sub-task 2) (#53) * feat(v1.9): add `aqa report` CLI verb (sub-task 2) Cables `@aqa/reporter` into a real CLI verb. README quick-start has referenced `aqa report` since v0.0.1 but the verb itself was never wired — `bunx aqa report` errored at parse time. CLI surface: aqa report [--run-id <id>] [--format md|json|both] Behaviour: - `--run-id` omitted → defaults to the latest run directory (lexical sort on run-id, which has an ISO-like prefix so newest wins) - reads `.aqa/runs/<id>/events.jsonl` + `findings.jsonl` - reconstructs a schema-conformant Run object from the first `run_started` and last `run_finished` events (no new sidecar files — reports stay replayable from the audit trail alone) - writes `report.md` (renderMarkdown) + `report.json` (renderJson) into the same run directory; the JSON shape is the stable one the admin UI already consumes - structured errors for: missing runs dir, missing run-id, malformed JSONL, schema-invalid findings - `--format md|json` opt-out for each artifact Tests: 10 new node:test cases in packages/kit/test/report-cmd.test.ts cover happy path (5: full render, latest-default, md-only, json-only, zero-findings) and error cases (5: missing dir, missing run-id, empty dir, malformed JSONL, schema-invalid finding). Synthetic run dirs match @aqa/schemas Run/Finding exactly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v1.9): address Copilot iter 1 on report PR #53 5 actionable comments, all addressed: 1. P1 — state was unconditionally 'succeeded' when run_finished was present. But runRun writes run_finished even on most failure paths (pack_errors / scenario_errors / missing_scenarios / unsafe_paths / runtime_errors / zero scenarios). Now derives state by inspecting the payload counters: any non-zero error counter OR zero scenarios → 'failed'; no run_finished → 'running'; else 'succeeded'. Added 3 regression tests. 2. P2 — latest-run selection used lexical sort on directory name, which silently picks the wrong run when `aqa run --seed` produces hash-based IDs (run-<sha>). Now uses statSync.mtimeMs (most recent wins) with the directory name as a deterministic tie-breaker. Added a regression test that intentionally writes the lexically-earlier name LAST. 3. P1 — readJsonl returned [] for missing files. That made `aqa report` succeed on a corrupted/incomplete run dir and emit a synthetic empty report. Now both events.jsonl and findings.jsonl are required up-front with explicit existence checks; readJsonl assumes existence. Added 2 regression tests. 4. Path-traversal — --run-id was forwarded to join() without validation, so a value like '../..' could read/write files outside .aqa/runs. Added a LongSlug-shaped regex (/^[a-z0-9-]{1,80}$/) at the CLI boundary that rejects traversal and any non-slug input. Added 2 regression tests. 5. writeFileSync wasn't wrapped — a read-only FS or disk-full would bubble through the CLI's top-level unhandled-error path instead of returning a structured ReportErr. Wrapped both writes in a single try/catch. Test fixture bug also fixed: writeFindings now accepts an optional runId parameter (default RUN_ID) so findings written for ALT_RUN_ID-style directories carry the matching run_id, not a stale constant. 17 tests now pass (10 original + 7 new regression tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v1.9): address Copilot iter 2 on report PR #53 4 actionable comments, all addressed: 1. RUN_ID_RE diverged from real LongSlug. Previous /^[a-z0-9-]{1,80}$/ capped at 80 (vs schema's 256) and allowed leading/trailing dashes plus `--` runs that real LongSlug rejects — so a malformed --run-id could slip past this guard and fail later at Finding.parse(). Now uses the canonical SlugPattern /^[a-z0-9](?:-?[a-z0-9])*$/ + a separate 256-char length check so error messages distinguish "bad characters" from "too long". Added regression tests for both. 2. safeIsDir() followed symlinks via statSync. An attacker (or a prior run) leaving a symlink under .aqa/runs/<runId> pointing outside the project would let report.md/report.json land anywhere the link pointed. Now explicit lstatSync().isSymbolicLink() check refuses symlinked run dirs. Added a regression test (skipped on Windows when symlink perms aren't available). 3. readJsonl() silently dropped any non-empty line that parsed as valid JSON but wasn't a plain object (null, [], "x", 42). For events.jsonl that turned corruption into a seemingly-successful report. Now strict: throws "expected a JSON object, got <type>" with explicit null branch (typeof null === 'object' in JS so the naive type check needed a guard). Added 2 regression tests (null + array). 4. Misleading comment claimed admin UI keys on run.id and not config_hash. Admin actually renders config_hash in replay copy, so the all-zeros placeholder will be visible. Replaced with an honest "treat as not-computed sentinel" note so future maintainers don't try to fix a bug that isn't there. 22 tests now pass (was 17 → +5 new regression tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v1.9): address Copilot iter 3 on report PR #53 3 actionable comments, all addressed: 1. latestRunId considered any directory entry, including symlinks and non-run subdirs. A `.aqa/runs/readme-stash/` with the newest mtime would shadow real runs. Now filters: name matches RUN_ID_RE + not-a-symlink (lstat) + contains events.jsonl. Added a regression test that pollutes `.aqa/runs/` with a non-run subdir and asserts the latest REAL run wins. 2. Per-file symlink protection. The earlier symlink guard only checked the run directory itself. A pre-existing `report.md` or `report.json` symlink would let writeFileSync follow the link and redirect the write outside the project. Now lstat-checks each target file before writing; refuses if it's a symlink. Added a regression test that creates a symlink to an outside file, attempts the report, and asserts both (a) refusal and (b) attacker-controlled file remains byte-identical. 3. reconstructRun could in principle produce Run-schema-incompatible output if the audit chain itself is malformed. Now validates the reconstructed object via Run.Run.safeParse before handing it to the renderers — surfaces a structured error instead of writing a report.json that the admin UI would silently reject. Also flipped the import of Run from type-only to value (needed for safeParse). 24 tests now pass (was 22 → +2 regression tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v1.9): aqa admin CLI verb — boots SPA + API in-process (sub-task 3) (#54) * feat(v1.9): add `aqa admin` CLI verb — boots SPA + API in-process (sub-task 3) The kit now ships its own admin: `bunx aqa admin` boots a single node:http server that serves the bundled admin SPA AND wires @aqa/server.makeApi() against an in-memory store seeded from .aqa/runs/<id>/{events,findings}.jsonl. No more "bun --filter @aqa/admin dev" — that flow only worked inside the monorepo. CLI surface: aqa admin [--port <n>] [--host <h>] Behaviour: - defaults: 127.0.0.1:5173 (matches the prior Vite dev URL); --port 0 → OS-assigned (useful for tests). - / → static index.html from bundled SPA - /assets/* → static asset files (404 on miss, no SPA fallback) - /api/healthz → kit-owned `{ ok: true }` (NOT in makeApi — lets tests smoke the boot without depending on auth/store state) - /api/* → delegated to makeApi() handler table - everything else → SPA fallback (index.html) so client-side routing works - Path traversal: forbidden (refuses to serve files outside SPA dist). - Seeds the in-memory store from local .aqa/runs/, so the admin shows real runs out of the box — empty state when no runs exist. Bundling: new packages/kit/scripts/bundle-admin.mjs copies packages/admin/dist/ into packages/kit/dist/admin/ during build. Topo build runs @aqa/admin's vite build BEFORE @aqa/kit's bundle step, so the source dist is guaranteed present. Dep cycle: @aqa/server depends on @aqa/kit (for runPackNew). Adding @aqa/server to kit's deps is a workspace cycle — bun handles it, but the runtime keeps the cycle shallow via dynamic imports of @aqa/server + @aqa/store inside runAdmin(). Type imports stay static (erased at runtime). @aqa/server's index now also re-exports ApiHandler / ApiMethod / ApiRequest / ApiResponse, which kit consumes type-only. Tests: 7 new node:test cases in test/admin-cmd.test.ts boot the server on port 0, exercise the static + API paths, and verify error / 404 / traversal handling. Uses a fake admin dist so the test isn't coupled to whether the real SPA built recently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v1.9): break kit↔server build cycle by extracting @aqa/pack-author Sub-task 3's CI exposed the underlying dep cycle: @aqa/server depends on @aqa/kit for runPackNew (POST /api/packs/scaffold), and @aqa/kit now needs @aqa/server (admin command). The topo-sort build order can't resolve a cycle — server tries to build before kit's dist exists and errors with TS2307 cannot find module @aqa/kit. Fix: extract runPackNew + PackNewErrorCode/PackNewOptions/PackNewResult into a brand-new workspace package @aqa/pack-author. Both kit and server depend on it independently; no cycle remains. Mechanics: - New packages/pack-author with package.json + tsconfig + src + test - git-moved packages/kit/src/commands/pack-new.ts → packages/pack-author/src/index.ts - New packages/kit/src/commands/pack-new.ts is a 5-line re-export so the in-kit CLI / tests import paths keep working unchanged - packages/server/src/api.ts imports runPackNew from @aqa/pack-author - packages/server/package.json: replaced @aqa/kit dep with @aqa/pack-author - packages/kit/package.json: added @aqa/pack-author alongside @aqa/server Tests: 2 new node:test cases in pack-author/test confirm the package boundary (callable + structured error). All heavy behavior coverage stays in packages/kit/test/pack-new.test.ts via the re-export shim. Build now passes in topo order: pack-author → server → kit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v1.9): address Copilot iter 2 on admin PR #54 3 actionable comments, all addressed: 1. Stale comment in packages/server/src/api.ts said "Delegates to runPackNew from @aqa/kit" but the import was updated to @aqa/pack-author. Aligned the comment with the implementation. 2. packages/pack-author/package.json declared files: ["dist", "README.md"] but README.md was missing — publish would ship an empty/missing README and violate the repo guideline (every package README required). Wrote a real README explaining the cycle-break rationale and API surface. 3. packages/pack-author/test header claimed runPackNew was the "default-export shape"; it's actually a named export. Reworded. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v1.9): address Copilot iter 3 on admin PR #54 2 actionable comments, both addressed: 1. Stale comment in admin.ts said "Lazy dynamic imports break what would otherwise be a static dependency cycle: @aqa/server depends on @aqa/kit (for runPackNew)". This was true in an earlier iteration of PR #54, but the cycle is now properly broken by extracting `runPackNew` into @aqa/pack-author. Reworded: the dynamic import is now an optimisation (defer makeApi/RunnerQueue loading to the actual `aqa admin` invocation), not a cycle workaround. 2. `--host 0.0.0.0` in help text had no security warning. `aqa admin` runs without real authentication — binding to LAN exposes the in-memory store + makeApi() to any peer on the network. Added an explicit WARNING block to the help text spelling out the risk and the safe use cases (dev VM / isolated CI runner). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(v1.9): rewrite junior quick-start to match shipped CLI surface (sub-task 5) (#56) The v1.8 README claimed `bun add -d agentic-qa-kit` worked (npm 404), `aqa install-agent-files` worked (CLI verb didn't exist), `aqa report` worked (CLI verb didn't exist), and `bun --filter @aqa/admin dev` was how juniors boot the admin (only works inside the monorepo). v1.9 closes those gaps in code (PRs #52/#53/#54/#55); this PR brings the docs back in sync. ## README - New step 2: `.npmrc` snippet for GitHub Packages auth + `GITHUB_TOKEN` export. Public packages on GH Packages still require auth — this is the single biggest junior trap and now has its own step. - Step 3: `bun add -d @padosoft/agentic-qa-kit` (replaces the npm-404 `agentic-qa-kit`). - Step 4 bundles `init` + `doctor` + `validate` so juniors verify the install before moving on. - Step 5: `install-agent-files --targets` — now matches a real verb. - Step 6: explicit `risk-map.yaml` edit with a worked example invariant. - Step 7: `aqa run --profile smoke` with explicit pointer to the per-run artifacts under `.aqa/runs/<id>/`. - Step 8: `aqa report` with format flag matrix. - Step 9: `aqa admin` — single command boots SPA + API (replaces `bun --filter @aqa/admin dev`). - Step 10: reproduce from artifacts. - Footer: `bun run e2e:ecosystem` pointer for monorepo contributors. - "How you use it" 8-step model reflects the new ordering. - Roadmap rows added for v1.8 and v1.9. - Status line bumped: `v1.7 current` → `v1.9 current`. ## docs/getting-started.md Section 1 new: GitHub Packages auth setup with PAT scope + .npmrc + env export. Section 2: bun add @padosoft/agentic-qa-kit. Sections 3–9: init/doctor/validate/install-agent-files/risk/run/report/admin in order, each with the real flag matrix. Closing "Where to go next" links the new PACK-AUTHORING doc. ## CHANGELOG Added [1.9.0] entry covering the 5 PRs + the kit↔server cycle fix + the bundle/publish-prep mechanics. Backfilled [1.8.1] / [1.8.2] / [1.8.3] which had only been recorded in PROGRESS.md. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v1.9): GitHub Packages publish pipeline (sub-task 4) (#55) * feat(v1.9): GitHub Packages publish pipeline (sub-task 4) Adds the infrastructure to publish @padosoft/agentic-qa-kit as a single bundled tarball to GitHub Packages on every v* tag. ## What lands - esbuild bundler (packages/kit/scripts/build-bundle.mjs): - Walks the kit's import graph and inlines every workspace dep (@aqa/*) plus npm deps (yaml, kleur, zod, …) into one dist/cli.cjs file. - Externalises only Node built-ins (assert / fs / http / process / crypto / …) in BOTH prefix forms (`fs` and `node:fs`) — some bundled CJS deps (yaml, ajv) call `require('process')` without the prefix and would otherwise hit esbuild's "Dynamic require" stub. - Emits CJS-in-.cjs because the kit's package.json sets `"type": "module"`. Plain `.js` would be parsed as ESM and the bundled CJS wrappers would throw on `require('process')`. - Dedupes shebangs (tsc preserves the source `#!/usr/bin/env node` and esbuild keeps it; a double shebang is a SyntaxError). - Sourcemap is emitted so crash stacks point at the original TypeScript sources. - chmod 0o755 on POSIX so the bin script is executable. - Writes dist/cli.bundle.meta.json sentinel for CI to verify the artifact actually got produced. - Verified locally: bundle is ~460 KB; `node cli.cjs --version` and `--help` work. - Publish-time package.json rewriter (packages/kit/scripts/publish-prep.mjs): - Substitutes `name` from `aqa.publishName` (= @padosoft/agentic-qa-kit). GitHub Packages requires `<scope> === <owner>`, but renaming the workspace permanently would break every internal import path. - Pins every `workspace:*` dep to the kit's current version. - Idempotent; rewrite lives only in the CI checkout, never committed. - GitHub Actions workflow (.github/workflows/publish.yml): - Tag-triggered on `v*`, permissions: contents:read + packages:write + id-token:write. - Verifies tag matches packages/kit/package.json version. - bun install → bun run build → verify bundle exists → publish-prep → npm publish --provenance --access public to npm.pkg.github.com. - Pack-author extraction (packages/pack-author/): - Same cycle fix as sub-task 3 (kit ↔ server cycle on runPackNew). Required here too so sub-task 4 builds standalone in CI; the duplicate-add merges cleanly with sub-task 3's identical files. - Moved packages/kit/src/commands/pack-new.ts → pack-author/src/index.ts (git-move preserves history). Kit keeps a 5-line re-export shim so existing CLI imports work unchanged. - @aqa/server now imports runPackNew from @aqa/pack-author (was @aqa/kit). - Added 2 boundary tests on pack-author. - Kit package.json: - version 0.0.1 → 1.9.0 (target tag). - publishConfig.registry → https://npm.pkg.github.com. - aqa.publishName declares the @padosoft scope override. - main/exports/bin/files point at the new bundled dist/cli.cjs. - Build-bundle smoke test (packages/kit/test/build-bundle.test.ts): - 4 tests: bundle present + executable + shebang + size bounds (skipped if not built); publish-prep rewrites + errors on missing publishName. - Uses a hermetic temp dir for the publish-prep tests so the real packages/kit/package.json is never touched even on assertion failure. All gates green locally: 9 packages, 219 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v1.9): address Copilot iter 1 on publish-pipeline PR #55 6 actionable comments, all addressed: 1. CRITICAL — publish-prep now STRIPS @aqa/* deps entirely (was only pinning workspace:* → version). The @aqa/* packages are inlined into dist/cli.cjs by esbuild and are NOT separately published, so leaving them in `dependencies` makes `bun add @padosoft/agentic-qa-kit` fail with "404 — @aqa/runner not found on registry". Test fixture updated to include both @aqa/* (must be stripped) and non-@aqa workspace:* (must be pinned) so the new behaviour is asserted. 2. build-bundle.mjs header comment said "ESM file at dist/cli.js"; the implementation emits CJS to dist/cli.cjs. Rewrote the docstring to match — plus added the "why CJS-in-.cjs" explanation (bundled deps call `require('process')`; the .cjs extension overrides "type": "module"). 3. test/build-bundle.test.ts header similarly out of date (`dist/cli.js`). Updated. 4. packages/pack-author/test header claimed runPackNew was "default-export shape", but it's a named export. Reworded to "NAMED export". 5. packages/pack-author missing README.md (declared in `files`). Wrote a real README explaining why the package exists (kit↔server cycle) and the API surface. 6. server/api.ts comment still said "Delegates to runPackNew from @aqa/kit" — implementation now imports from @aqa/pack-author. Updated. The `dist/admin` not-in-files note from review is intentional on this sub-task branch — the bundle-admin script is added by sub-task 3 (PR #54). Macro merge will combine both: sub-task 3 adds dist/admin to files, sub-task 4 adds dist/cli.cjs + publish prep. All gates green: 220 tests, lint clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v1.9): address Copilot iter 2 on publish-pipeline PR #55 5 more actionable comments, all addressed: 1. CRITICAL — packages/kit/LICENSE was declared in `files` but didn't exist (only the repo root has LICENSE). Publishing from packages/kit would omit the license. Copied the root Apache-2.0 LICENSE into packages/kit/LICENSE so the published tarball carries it. 2. build-bundle.mjs header still referenced `dist/admin/` and bundle-admin.mjs — both belong to sub-task 3 (PR #54). Clarified that this sub-task only ships the CLI bundler; the admin static directory + its bundler land in the macro merge alongside PR #54. 3. Stale "tarball's `bin: dist/cli.js`" in build-bundle.mjs inner comment — fixed to cli.cjs. 4. Stale shebang-dedup comment claimed esbuild's banner adds the duplicate shebang; the banner option is no longer used today. Reworded as a defensive idempotent post-process so future re-introduction of a banner doesn't silently break the bundle. 5. Test now asserts the bundle's executable bit (0o111 mask) on POSIX. Skipped on Windows where mode bits don't carry POSIX semantics. The "drop types export" concern is intentional: @aqa/kit was only consumed programmatically by @aqa/server (for runPackNew); that's now via @aqa/pack-author. No remaining consumer imports anything from @aqa/kit's API surface, so the published package can be CLI- only without breaking anyone. The PR 56 docs-claim-don't-exist comments are correct in isolation but will be resolved at macro merge time (sub-task 5 PR's docs match the post-merge macro state, not its own branch state). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(v1.9): record macro closure in PROGRESS + LESSON --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Cable
@aqa/reporterinto a real CLI verb. README has documentedbunx aqa reportsince v0.0.1 but the verb was never wired — it errored at parse time.What changed
packages/kit/src/commands/report.ts—runReport({ root, runId?, format? })packages/kit/src/cli/aqa.ts: newcase 'report'+--run-id/--formatvalue flags + HELP text@aqa/reporteradded as workspace dependency of@aqa/kittest/report-cmd.test.ts(synthetic run-dir fixtures matching @aqa/schemas exactly)Junior-readable command
Reads
.aqa/runs/<id>/events.jsonl+findings.jsonl. Writesreport.md(renderMarkdown) andreport.json(renderJson) into the same run dir. The Run object is reconstructed fromrun_started+run_finishedevents — no new sidecar file, reports stay replayable from the audit trail alone.DoD checklist
Notes for reviewer
Sub-task 2 of macro
task/v1.9-junior-quickstart-truthing. Will conflict with sub-task 1 PR #52 onpackages/kit/package.jsontest script line — resolve at macro-merge time by combining both test file entries.