Skip to content

feat(cli): Phase 3 PR 2 — devtrail charter audit (3-step orchestration)#86

Merged
montfort merged 1 commit intomainfrom
feat/phase3-pr2-charter-audit
May 3, 2026
Merged

feat(cli): Phase 3 PR 2 — devtrail charter audit (3-step orchestration)#86
montfort merged 1 commit intomainfrom
feat/phase3-pr2-charter-audit

Conversation

@montfort
Copy link
Copy Markdown
Contributor

@montfort montfort commented May 3, 2026

Summary

Second of 6 PRs implementing Phase 3 + open frictions. The CLI command that orchestrates the dual-audit + calibrator cycle, using the prompt templates and output schema shipped in PR 1 (#85).

Architecture A1 (orchestration-only) means the CLI does not invoke LLM APIs. The operator pastes resolved prompts into their auditor of choice and saves responses to canonical paths.

Three steps, each invokable independently

$ devtrail charter audit CHARTER-01              # Step 1/3: PREPARE
  → resolves auditor prompts, writes to audit/charters/CHARTER-01/prompts/

$ devtrail charter audit CHARTER-01 --calibrate  # Step 2/3: CALIBRATE
  → validates auditor outputs, resolves calibrator prompt

$ devtrail charter audit CHARTER-01 --finalize   # Step 3/3: FINALIZE
  → validates all 3 outputs, prints external_audit YAML for telemetry

Each step is a filesystem mutation. Files persist between steps — operator can prepare, walk away, come back later, calibrate. Each step prints next-action guidance pointing to exact paths.

Per RFC #82 the resolved prompt is persisted before any external action. The schema's prompt_used field cites which prompt template was used; the calibrator can verify provenance.

Module shape

  • src/audit_schema.rs: jsonschema wrapper with oneOf-aware error formatting (mirrors telemetry_schema.rs).
  • src/commands/charter/audit.rs: 3-step run dispatch, template resolution, frontmatter parsing, external_audit YAML rendering.

Placeholder set

{{charter_id}}, {{charter_title}}, {{charter_path}}, {{charter_content}}, {{git_range}}, {{git_diff}}, {{ailog_paths}}, {{ailog_contents}}, {{audit_role}}, {{schema_path}}, {{auditor_primary_findings}}, {{auditor_secondary_findings}}. Unknown placeholders are left as literals (no surprise mutations).

Test plan

  • 400/400 tests pass (10 new unit + 7 new integration on top of 383 carried forward).
  • Schema discriminates auditor vs calibrator outputs via oneOf on audit_role (5 unit tests in audit_schema).
  • Template resolution preserves unknown placeholders (literal pass-through).
  • Full 3-step cycle test simulates the operator pasting auditor + calibrator responses; finalize prints valid external_audit YAML.
  • Calibrate without prior auditor outputs fails with clear <path> not found error.
  • Calibrate with malformed auditor output (missing required field) bails with schema validation error.
  • Finalize prints both auditor IDs in the rendered YAML for telemetry pasting.
  • No version bump — release ships at PR 6.

🤖 Generated with Claude Code

Second of 6 PRs implementing Phase 3 + open frictions. The CLI command
that orchestrates the dual-audit + calibrator cycle, using the prompt
templates and output schema shipped in PR 1 (#85).

Architecture A1 (orchestration-only) means the CLI does NOT invoke
LLM APIs. The operator pastes resolved prompts into their auditor of
choice (Copilot, Gemini, Claude, etc.) and saves responses to canonical
paths under audit/charters/<CHARTER-ID>/. The CLI's value is structure
(prompt resolution + output schema validation + telemetry-ready YAML),
not invocation.

Three steps, each invokable independently:

  $ devtrail charter audit CHARTER-01
    Step 1/3: PREPARE
    Resolves auditor-primary.prompt.md and auditor-secondary.prompt.md
    against the Charter content + git diff + originating AILOGs, writes
    to audit/charters/CHARTER-01/prompts/.

  $ devtrail charter audit CHARTER-01 --calibrate
    Step 2/3: CALIBRATE
    Validates the two auditor responses against audit-output.schema.v0.json,
    resolves the calibrator-reconciler prompt with both responses
    embedded as context.

  $ devtrail charter audit CHARTER-01 --finalize
    Step 3/3: FINALIZE
    Validates all 3 outputs (auditor-primary + auditor-secondary +
    calibrator), prints a YAML-formatted external_audit array block
    ready to paste into the Charter telemetry, and points to the
    calibrator's reconciliation summary for outcome.scope_change_notes.

Each step is a filesystem mutation. Files persist between steps —
operator can run prepare, walk away, come back days later, run
calibrate. Each step prints clear next-action guidance pointing to
the exact paths involved.

Per RFC #82 the resolved prompt is persisted BEFORE any external
action. The schema's prompt_used field cites which prompt template
was used; the calibrator can verify provenance.

Module shape:
- src/audit_schema.rs: jsonschema wrapper with oneOf-aware error
  formatting, mirroring telemetry_schema.rs and charter_schema.rs.
- src/commands/charter/audit.rs: 3-step run dispatch, template
  resolution with placeholder substitution, frontmatter parsing for
  auditor summaries, external_audit YAML rendering.

Placeholders supported in templates:
  {{charter_id}}, {{charter_title}}, {{charter_path}},
  {{charter_content}}, {{git_range}}, {{git_diff}},
  {{ailog_paths}}, {{ailog_contents}}, {{audit_role}},
  {{schema_path}}, {{auditor_primary_findings}},
  {{auditor_secondary_findings}}.

Unknown placeholders are left as literals (no surprise mutations).

Tests:
- 5 unit tests in src/audit_schema.rs (auditor vs calibrator oneOf
  discriminator, charter_id pattern, auditors_reconciled minItems).
- 5 unit tests in src/commands/charter/audit.rs (canonical_id,
  template substitution, frontmatter parsing, AuditorSummary).
- 7 integration tests in cli/tests/charter_audit_test.rs covering
  all three steps + error paths (devtrail-not-installed, unknown
  charter, calibrate-without-auditor-outputs, schema validation
  failure, full cycle, mutually-exclusive flags).

400/400 tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@montfort montfort merged commit 412bf31 into main May 3, 2026
@montfort montfort deleted the feat/phase3-pr2-charter-audit branch May 3, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant