Skip to content

v0.9.0

Choose a tag to compare

@tranquocthong tranquocthong released this 08 Sep 08:12
· 137 commits to main since this release

Nineteen rough edges from one real dogfooding session (an eid-gateway feature and a 26-FR/47-TC phase), grouped by what they actually cost. Group 1 is false-green / lost-evidence: a command reports success, or asserts nothing, while the thing it claims to check never happened. Group 2 is noise that wastes time without corrupting the result. Both groups share one theme this release keeps hitting: an engine that silently drops the SECOND (or third, or fourth) matching thing — a table, a repo, an FR, a stderr stream — instead of merging or naming what it dropped.

Group 1 — false-green / lost evidence

  • verify-collect never wrote VERIFICATION.md. It computed {status, passed, failed, truths} and returned them as JSON only — commands/manual-test.md step 4 says "This writes VERIFICATION.md," but nothing did. status-report, doctor's verify-integrity check, and task-baseline all read that file to decide a feature is verified; a full 68/68 pass reported status:"passed" on stdout while the feature stayed "not verified" forever and the ship gate never opened. It now actually writes the file, in the status: passed|failed + - TC-xxx: verified shape those readers already expect. Also fixed in the same command: --results only accepted a file path, but the doc's own worked example passes the JSON result line inline — now accepts either. --feature is required (no inference from the shared active-feature mirror — the same write-safety rule trace-link already enforces).
  • A dict expect: on a SQL verify:/setup:/teardown: step was never asserted. db-query.sh -t has no column headers to match a dict key by, so check_scalar() treated a dict expect as purely descriptive — which is also the MOST NATURAL way to write a multi-column assertion (it's exactly how expect.body works for HTTP). A real bug in one of those columns printed green. lint-checklist.sh now flags it before the run: split into one scalar verify: row per column, or assert via the HTTP response's json_path.
  • expect: had no contains operator. expect: "contains idx_foo" fell to the exact-string fallback — the whole (often multi-line) scalar result compared against the literal string "contains idx_foo", never equal, so an EXPLAIN-plan-has-this-index assertion always failed regardless of whether the index was actually there. Added contains / not contains (substring match) to check_scalar/cmp, matching what json_path: assertions already supported.
  • A failed teardown: step vanished into console-only output. teardown: is deliberately best-effort (never aborts the run), but a teardown step is often RECOVERY — undo a mutation, restore a row — and a failed recovery only printed a line to stdout; the caller had no way to know it happened. Its dirty leftover state then broke a LATER, unrelated test with no link back to the real cause. setup.run_steps() now returns the warning list instead of swallowing it; run-checklist.sh --json reports it as teardownWarnings, and verify-collect records it in VERIFICATION.md under its own heading — visible, not misattributed.
  • sd-skeleton harvested only the FIRST FR-prefixed table. An SRS with FRs split across several sub-tables (one per module — 26 FRs across 4 tables is a real, common shape) reported fr: 6 and silently dropped the rest. Same bug for the TC/NFR ID-prefix fallbacks. tableByIdPrefix now merges every matching table's rows instead of .find()-ing the first.
  • sd-skeleton's §12.2 error-code harvest never read the SRS's own §6.2 "Error & Notification Messages" table — only per-story Edge Cases bullets, even though the generated §12.2 fallback text has said "derive from SRS §6.2" since Pass-1 existed. A project that lists its error codes in that dedicated table (rather than as story edge-case prose) harvested 0, with no signal. It's now read and merged, with the table's own Message column used as the real user-facing message instead of a TODO.
  • trace-build built tables purely by content-matched header keywords, with no cross-check against the section NUMBER. A §12.2 (or §5.1/§13.2/§10.4/§5.2) table whose column headers drifted — translated, renamed — silently produced 0 nodes for that whole section: the SD reads complete, the trace quietly drops it. Now warns by name when the numbered section exists with a table the content-matcher didn't recognize, quoting the actual headers found.
  • trace-build never flagged an FR with zero linked TCs. An orphaned requirement — present in §5.1, uncovered by any §13.2 test — built a valid-looking trace. Now warned by FR id.
  • checklist-gen's auth detection scanned the WHOLE multi-repo hub, majority-voting across every config.repos entry — a feature scoped to one no-auth/HMAC repo in a hub of Summer/APISIX services got the hub's summer classification and 401'd every generated test. Now resolves the feature's own repo scope first (--repos > declared trace-repos > file-links inference — the same precedence verify-code already uses) and detects against that one repo when it resolves unambiguously.
  • detect-auth.sh defaulted an unrecognized custom scheme to no-auth. A gateway verifying its own HMAC/signature header (no Spring Security dep, no Authorization: Bearer) has real access control, and no-auth scaffolds unsigned tests that never exercise it. Added an HMAC/signature-pattern check (X-Signature, Mac.getInstance("Hmac..."), crypto.createHmac, …) that now classifies as unknown instead — same safe-default fix as the 0.7.4 Summer/X-Userinfo flip, one auth family later.
  • parse-prd/expand, run as a bare CLI subcommand, look like they finish in one shot — they don't. Default taskCore.aiMode is agent-native: the command prints a GenerationSpec and exits 0 without calling an LLM (zero-network by design). commands/ingest.md and commands/phase.md presented it as a single self-sufficient bash command; the agent could read exit 0 + a large JSON blob as "done," run use-tag successfully, and leave tasks.json with zero tasks for that tag — silent, textbook false-success. Both docs now spell out the handoff: parse the printed spec, generate the Task[] yourself (Phase 2), then tasks-import --file <path> (Phase 3) — {"imported": N} is the real signal, not the Phase 1 exit code.

Group 2 — noise, not wrong

  • srs-snapshot's filename-derived-slug warning only fired for a date-prefixed filename. Any other filename shape (phase-3-agentgw-client.md, drifting from a project's own naming convention) was silently adopted. Now warns whenever the slug came from the filename at all (no Feature: line, no --feature).
  • checklist-gen leaked detect-auth.sh's stderr diagnostics into a caller that merges stdout+stderr. The engine's own stdout stayed pure JSON, but execFileSync's default stdio inherits the child's stderr straight into this process's — any 2>&1 capture (or a tool wrapper that combines streams) saw prose land BEFORE the JSON and failed to parse it, even though the command had already succeeded. Now explicitly captured and discarded.
  • checklist-gen always scaffolded config.db/config.redis/cleanup:, even for a phase with no persistence at all. sd-template.md's own §7 convention is "no DB change → delete this section" — now honored: no §7 (or a §7 that says N/A/stateless/no database), no db/redis/cleanup block, with a warning explaining why.
  • checklist-gen silently degenerated to one suite per test. When every §13.2 TC row carries a distinct Flow value (common for business-rule-derived TCs — 26 rules, 28 suites), grouping-by-Flow produces 28 singleton suites, and a singleton suite's lone test is ALWAYS tagged smoke, never regression (the rule needs ≥2 tests per flow to ever emit a regression tag) — so --tag regression silently skips the entire checklist with no error. Now warned by name, with the count.
  • genSd's §10.4 State Management forced a TODO:MANUAL-REVIEW marker on every internal/hybrid phase, even one with no state machine at all. sd-template.md's own convention is "no state → delete this sub-section," the same carve-out §7 gets — but Pass-1 never offered it, so a plain batch job or pure transform carried a permanently unresolvable marker blocking approval until someone hand-deleted the section. Now a plain advisory note, not a TODO.

857 Node tests green (15 new), 68 Python checklist_lib unit tests green (3 new).