feat(cli): Phase 3 PR 2 — devtrail charter audit (3-step orchestration)#86
Merged
feat(cli): Phase 3 PR 2 — devtrail charter audit (3-step orchestration)#86
Conversation
Second of 6 PRs implementing Phase 3 + open frictions. The CLI command that orchestrates the dual-audit + calibrator cycle, using the prompt templates and output schema shipped in PR 1 (#85). Architecture A1 (orchestration-only) means the CLI does NOT invoke LLM APIs. The operator pastes resolved prompts into their auditor of choice (Copilot, Gemini, Claude, etc.) and saves responses to canonical paths under audit/charters/<CHARTER-ID>/. The CLI's value is structure (prompt resolution + output schema validation + telemetry-ready YAML), not invocation. Three steps, each invokable independently: $ devtrail charter audit CHARTER-01 Step 1/3: PREPARE Resolves auditor-primary.prompt.md and auditor-secondary.prompt.md against the Charter content + git diff + originating AILOGs, writes to audit/charters/CHARTER-01/prompts/. $ devtrail charter audit CHARTER-01 --calibrate Step 2/3: CALIBRATE Validates the two auditor responses against audit-output.schema.v0.json, resolves the calibrator-reconciler prompt with both responses embedded as context. $ devtrail charter audit CHARTER-01 --finalize Step 3/3: FINALIZE Validates all 3 outputs (auditor-primary + auditor-secondary + calibrator), prints a YAML-formatted external_audit array block ready to paste into the Charter telemetry, and points to the calibrator's reconciliation summary for outcome.scope_change_notes. Each step is a filesystem mutation. Files persist between steps — operator can run prepare, walk away, come back days later, run calibrate. Each step prints clear next-action guidance pointing to the exact paths involved. Per RFC #82 the resolved prompt is persisted BEFORE any external action. The schema's prompt_used field cites which prompt template was used; the calibrator can verify provenance. Module shape: - src/audit_schema.rs: jsonschema wrapper with oneOf-aware error formatting, mirroring telemetry_schema.rs and charter_schema.rs. - src/commands/charter/audit.rs: 3-step run dispatch, template resolution with placeholder substitution, frontmatter parsing for auditor summaries, external_audit YAML rendering. Placeholders supported in templates: {{charter_id}}, {{charter_title}}, {{charter_path}}, {{charter_content}}, {{git_range}}, {{git_diff}}, {{ailog_paths}}, {{ailog_contents}}, {{audit_role}}, {{schema_path}}, {{auditor_primary_findings}}, {{auditor_secondary_findings}}. Unknown placeholders are left as literals (no surprise mutations). Tests: - 5 unit tests in src/audit_schema.rs (auditor vs calibrator oneOf discriminator, charter_id pattern, auditors_reconciled minItems). - 5 unit tests in src/commands/charter/audit.rs (canonical_id, template substitution, frontmatter parsing, AuditorSummary). - 7 integration tests in cli/tests/charter_audit_test.rs covering all three steps + error paths (devtrail-not-installed, unknown charter, calibrate-without-auditor-outputs, schema validation failure, full cycle, mutually-exclusive flags). 400/400 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second of 6 PRs implementing Phase 3 + open frictions. The CLI command that orchestrates the dual-audit + calibrator cycle, using the prompt templates and output schema shipped in PR 1 (#85).
Architecture A1 (orchestration-only) means the CLI does not invoke LLM APIs. The operator pastes resolved prompts into their auditor of choice and saves responses to canonical paths.
Three steps, each invokable independently
Each step is a filesystem mutation. Files persist between steps — operator can prepare, walk away, come back later, calibrate. Each step prints next-action guidance pointing to exact paths.
Per RFC #82 the resolved prompt is persisted before any external action. The schema's
prompt_usedfield cites which prompt template was used; the calibrator can verify provenance.Module shape
src/audit_schema.rs: jsonschema wrapper with oneOf-aware error formatting (mirrorstelemetry_schema.rs).src/commands/charter/audit.rs: 3-step run dispatch, template resolution, frontmatter parsing, external_audit YAML rendering.Placeholder set
{{charter_id}},{{charter_title}},{{charter_path}},{{charter_content}},{{git_range}},{{git_diff}},{{ailog_paths}},{{ailog_contents}},{{audit_role}},{{schema_path}},{{auditor_primary_findings}},{{auditor_secondary_findings}}. Unknown placeholders are left as literals (no surprise mutations).Test plan
oneOfonaudit_role(5 unit tests inaudit_schema).external_auditYAML.<path> not founderror.🤖 Generated with Claude Code