feat(planning): audit-answers skill — agent-validated interview finish (#1043)

[kyle-sexton]

Closes #1043.

What

A new planning skill: /planning:audit-answers — an optional agent-validated
finish to /planning:interview. Instead of hand-answering every round, the
orchestrator accepts each open question's recommended answer, then fresh-context
validators adversarially re-examine each accepted answer and only the doubtful
ones return to the human.

• Accept-to-completion builds a provisional, in-session filled ledger —
  nothing is written to the tracked ledger/Brief on bare invocation. A
  mechanical never-auto floor (USER-RESERVED deferred questions + the
  interview's auto-guard class) is held out of the auto-accept and always routes
  to the human; no CONFIRMED verdict can collapse one.
• 1–3 fresh-context (non-fork) validators receive the filled ledger + Brief
  with the recommendation's rationale withheld (audit the decision, not the
  pitch) and return a per-answer verdict — CONFIRMED / CHALLENGED /
  RECLASSIFIED-TO-HUMAN — plus shaky dependency-chain flags.
• Triaged confirm: CONFIRMED collapse to one line; CHALLENGED + RECLASSIFIED
  become real questions in the interview round format. Persistence happens
  only after the human confirmation round — the human-confirmation is the
  explicit override the audit verb contract sanctions.
• Validation, never derivation — subagent-invented answers are out of scope
  (fresh-context independence is real only for checking an answer, not producing
  one).

Boundary with devils-advocate (cite, don't duplicate)

The adversarial evidence discipline is /planning:devils-advocate's and is
cited, not re-implemented. The dispatch + per-answer verdict contract are
purpose-built because the input and output genuinely differ: devils-advocate
extracts assumptions from one plan artifact → severity findings + revised-plan
recs; this skill validates a pre-enumerated per-answer ledger → per-answer
verdicts fed back as interview questions. Same-plugin, so the interview seam
sentence is a direct reference (seam-phrasing is cross-plugin only).

Where

• skills/audit-answers/ — new skill (SKILL.md + evals + gotchas).
• skills/interview/SKILL.md — one Composition-table seam row (the removal
  contract: delete the skill dir + this one row).
• README.md, CHANGELOG.md, plugin.json (keyword), marketplace.json (tag)
  — registration. Minor bump 0.25.0.

Naming

Provisional pick audit-answers was confirmed by a blind, verb-table-aware
fresh-context naming pass (three lenses): it fits the fixed-verb table (audit =
read-only findings; mutation only behind an explicit override — the
human-confirmation round is that override), needs no PLUGIN-PHILOSOPHY exception,
and lands top-2 across lenses. Shortlist carried to Kyle: audit-answers
(recommended), audit-recommendations (table-fit, sharper), cross-examine
(vivid, but a coined verb needing a documented table exception). Renameable
pre-merge if Kyle prefers a shortlist alternative.

Verification

• markdownlint-cli2 clean; skill-quality:check audit-answers → PASS (0
  errors, 0 warnings); SKILL.md 97 lines. Interview still PASS (pre-existing
  WARNs only). plugin.json/marketplace.json/evals.json valid; version
  parity 0.25.0; both planning shell tests pass.

Related

• planning/interview: round-format legibility levers (session-hop anchor, jargon ledger, artifact escape hatch, AUQ sharpening) #1042 / feat(planning): interview round-format legibility levers (#1042) #1048 — the interview round-format levers PR this branch is stacked
  on (base of this PR). Not closed by this PR; retarget to main after feat(planning): interview round-format legibility levers (#1042) #1048
  merges.

🤖 Generated with Claude Code

[claude]

Claude finished @kyle-sexton's task in 3m 7s —— View job

---

PR Review: feat(planning): audit-answers skill

Tasks

• Gather context and understand the request
• Read changed files
• Review SKILL.md for correctness and completeness
• Review evals.json for quality
• Review plugin.json / marketplace.json registration
• Check CLAUDE.md design rule compliance
• Post comprehensive review feedback

---

Summary

This is a well-reasoned skill addition. The core design decisions are sound: accept-first-then-validate (not derive), the never-auto floor for USER-RESERVED/auto-guard, rationale withheld from validators, one-dissent-is-signal merge rule, and deferred persistence until the mandatory human confirmation round. The devils-advocate evidence discipline is correctly reused by citation rather than duplicated. Registration (plugin.json, marketplace.json, README, CHANGELOG) is clean and internally consistent at version 0.25.0.

Two issues worth addressing before merge, plus a few observations.

---

Issues

1. Missing gotcha: session loss between Steps 4 and 5

SKILL.md line 39 states that provisional accepted answers are not persisted until the human confirmation round in Step 5. That's the right call — but the gotchas section doesn't document what happens if the session ends after Step 4 (triage is visible) but before the human confirms. In that case, the provisional state is silently lost: no ledger, no Brief update, nothing written. The user must rerun the skill from scratch. For a workflow whose whole premise is avoiding redundant effort, a surprise rerun is a meaningful failure mode.

Suggested gotcha (or addition to the existing one at line 97):

The human confirmation round (Step 5) is the only persistence gate. If the session ends between seeing the triage and confirming, the provisional answers are lost — rerun the skill. Step 4's CONFIRMED collapse is not a checkpoint.

Fix this →

---

2. Cross-vendor advisor path is underspecified and potentially ambiguous

SKILL.md lines 48–49:

prefer a cross-vendor advisor for a validator when one is installed and set up — e.g. the OpenAI Codex plugin, when its documented surface can take this input, invoked per its own docs — with the fresh-context same-vendor subagent as the stated fallback, never a route to a command that may not resolve.

The guardrails here are good ("never a route to a command that may not resolve") but the positive path is underdetermined: the skill gives no heuristic for how to discover whether "its documented surface can take this input." Without that, Claude has to guess — which is the exact failure mode the fallback clause is trying to prevent. The example plugin name is also a moving target.

The fresh-context same-vendor subagent already provides the correlated-blind-spot protection the cross-vendor path is motivated by (different context window, no shared reasoning chain). Given that this skill's primary value is the accept-then-check loop rather than cross-vendor diversity, the simpler fix is to either (a) make the same-vendor fresh-context path the sole stated path and drop the speculative preference, or (b) tighten the cross-vendor guidance to require explicit user configuration of the advisor plugin rather than auto-discovery.

Fix this →

---

Observations (non-blocking)

Evals are all behavioral (no file fixtures) — consistent with the plugin's established pattern. The interview skill's evals also use "files": [] throughout. For a skill whose precondition is an existing ledger file, a future improvement would be a fixture-backed eval for the "no-ledger stops" case (eval 3), but this is out of scope for this PR and not a deviation from existing practice.

Composition table entry in interview/SKILL.md. The new row uses the full /planning:audit-answers prefix; the adjacent devils-advocate row uses the bare /devils-advocate. This is a pre-existing inconsistency in the table, not introduced here — but if the table is ever cleaned up, the prefix form is the correct one.

The "validation, never derivation" principle is well-stated and the rationale (quadratic frontier cost, false patina) is in the right place (Purpose section, early). The redundant reinforcement in Gotcha 1 is appropriate for a property this load-bearing.

Version bump to 0.25.0 (minor) for a new skill addition is correct per semver.

---

Overall this is in good shape. The two issues above are both small targeted edits. Happy to re-review after they're addressed.

[claude]

Claude finished @kyle-sexton's task in 2m 32s —— View job

---

Security Review — PR #1051

• Gather PR diff and changed files
• Read skills/audit-answers/SKILL.md
• Read skills/audit-answers/evals/evals.json
• Read modified plugin.json, marketplace.json, CHANGELOG.md, README.md, interview/SKILL.md
• Analyze for security issues
• Post findings

---

Findings

IMPORTANT · Confidence: HIGH — Unguarded cross-vendor data egress without per-invocation consent

File: plugins/planning/skills/audit-answers/SKILL.md, line 48

The Step 2 paragraph instructs the orchestrator to prefer routing to a third-party AI system by default when one is installed:

"Where the verdict is high-stakes and correlated blind spots are the risk, prefer a cross-vendor advisor for a validator when one is installed and set up — e.g. the OpenAI Codex plugin, when its documented surface can take this input, invoked per its own docs…"

The data sent to that third-party system is the filled interview ledger (accepted answers) and the PLAN.md Brief (line 50) — which may contain confidential business decisions, proprietary constraints, acceptance criteria, and architecture choices.

Three specific problems:

1. Implicit, installation-level consent is not per-invocation consent. "When one is installed and set up" means the user agreed to install a plugin, not that they consented to each transfer of sensitive project context to an external AI vendor during a planning session. A user who installed an OpenAI plugin for a different workflow has no reason to expect it will receive their company's PLAN.md Brief.

2. The preferred path routes externally by default on high-stakes decisions. The skill instructs Claude to prefer the cross-vendor route for exactly the answers where the data is most sensitive ("high-stakes"). The same-vendor fresh-context subagent is the fallback, not the default.

3. This violates the repo's design rules. CLAUDE.md is explicit: "Deny by default on unjustified egress or trust delegation." No user-facing opt-in or userConfig key gates this routing; it fires automatically whenever a compatible plugin is detected.

Recommendation: Remove the cross-vendor routing suggestion from this skill entirely, or gate it behind an explicit userConfig boolean (e.g. allow_cross_vendor_validation: false by default) with a documentation note that enabling it will send planning data to the configured third-party plugin. The same-vendor fresh-context subagent path already achieves the independence goal with no data egress.

Fix this →

---

No other security issues found

The remaining changes (marketplace.json tag, plugin.json version bump, CHANGELOG.md, README.md, interview/SKILL.md composition row, evals.json) are documentation and metadata. No command injection surfaces, no credential handling, no GitHub Actions workflow changes, no path traversal risks, no ../ reach-outs. The shell: bash pre-computed context runs only git branch --show-current, which is read-only and safe.

[claude]

Claude finished @kyle-sexton's task in 1m 58s —— View job

---

Re-review: New Push to PR #1051

Tasks

• Load tools
• Check new commit diff
• Read updated SKILL.md
• Assess whether prior review findings were resolved
• Post updated review

---

What the new commit does

The a6ea9bf commit reframes the skill from an "auto-accept finish" to neutral validation of any filled ledger — hand-answered or auto-accepted. It:

• Rewrites the description, Purpose, and Step 1 to treat accept-to-completion as a conditional path (when open branches remain) rather than the skill's identity
• Renames Step 1 to "Ensure a filled ledger, holding the never-auto floor"
• Adds eval perf(markdown-formatter): unwired producer builds telemetry payload before the opt-in guard (~90ms/edit on Windows) #6 covering the hand-answered-ledger path

This is a correct and meaningful improvement. The validates-a-hand-answered-ledger-too eval is a good addition.

---

Status of prior review findings

Code Review — Finding 1: Missing session-loss gotcha ❌ Not addressed

The Gotchas section still does not document what happens if the session ends after Step 4 (triage visible) but before the human confirms in Step 5. The provisional answers are in-session only — silent rerun is the failure mode, and it's not called out anywhere.

Current last gotcha at SKILL.md line 97 ends the file without this case. Suggested addition:

The human confirmation round (Step 5) is the only persistence gate. If the session ends between seeing the triage and confirming, the provisional answers are lost — rerun the skill. Step 4's CONFIRMED collapse is not a checkpoint.

Fix this →

---

Code Review — Finding 2: Cross-vendor path underspecified ❌ Not addressed

SKILL.md line 48 is unchanged:

"prefer a cross-vendor advisor for a validator when one is installed and set up — e.g. the OpenAI Codex plugin, when its documented surface can take this input, invoked per its own docs"

There is still no heuristic for how to discover whether "its documented surface can take this input." Without one, Claude has to guess — which is the failure mode the fallback clause is there to prevent.

---

Security Review — IMPORTANT: Unguarded cross-vendor data egress ❌ Not addressed

The same line 48 still instructs the orchestrator to prefer routing the filled ledger (containing potentially confidential business decisions, constraints, architecture choices) to a third-party plugin by default when one is detected — without per-invocation user consent. CLAUDE.md is explicit: "Deny by default on unjustified egress or trust delegation."

Options ranked by preference:

1. Remove the cross-vendor preference entirely — the fresh-context same-vendor path already achieves correlated-blind-spot protection
2. Gate behind userConfig — add an allow_cross_vendor_validation key (default false) with a doc note that enabling it sends planning data to the configured third-party plugin

Fix this →

---

Summary

The a6ea9bf commit is a clean, correct improvement — the reframe from "auto-accept finish" to "validate any filled ledger" is the right call and the new eval is a good addition. However all three issues flagged in the prior reviews remain open. The security finding (Finding 2 / egress) is the most important to resolve before merge; Finding 1 (session-loss gotcha) is a small targeted edit.

[claude]

Claude finished @kyle-sexton's task in 1m 41s —— View job

---

Security Review — PR #1051 (re-run after synchronize)

Tasks

• Gather context and understand the request
• Get PR diff to see what changed
• Read changed files
• Analyze for security issues
• Post findings

---

Findings

IMPORTANT · Confidence: HIGH — Cross-vendor data egress still unaddressed

File: plugins/planning/skills/audit-answers/SKILL.md, line 48

The synchronize commit (a6ea9bf) addressed the "validates any filled ledger" framing — description, Purpose, Step 1, and an added eval — but did not touch line 48. The cross-vendor routing instruction flagged in the prior security review is unchanged:

"Where the verdict is high-stakes and correlated blind spots are the risk, prefer a cross-vendor advisor for a validator when one is installed and set up — e.g. the OpenAI Codex plugin, when its documented surface can take this input, invoked per its own docs…"

The data routed to that third-party system is the filled interview ledger plus the PLAN.md Brief (line 50) — potential confidential business decisions, acceptance criteria, proprietary constraints, and architecture choices.

The three specific problems from the first pass remain:

1. No per-invocation consent. "When one is installed and set up" is installation-level consent, not a per-use decision. A user who installed an OpenAI plugin for another workflow has no reason to expect it will receive their project's PLAN.md Brief during a planning session.

2. Sensitive data takes the preferred external route on high-stakes decisions. The instruction deliberately prefers cross-vendor routing exactly when the data carries the most weight.

3. Violates repo design rules. CLAUDE.md requires: "Deny by default on unjustified egress or trust delegation." No userConfig key gates this; it fires on plugin presence alone.

Recommendation: Remove the cross-vendor routing preference, or gate it behind an explicit userConfig boolean (allow_cross_vendor_validation, default false) with a note that enabling it will send planning data to the configured third-party plugin. The same-vendor fresh-context subagent already provides the independence the cross-vendor path is motivated by.

Fix this →

---

No other security issues found

The synchronize commit's changes are documentation-only: description reframing, Step 1 clarification (auto-accept is now a conditional path for an unfinished ledger, not the skill's identity), and one new eval for the hand-answered path. None of these introduce new security surfaces. The shell command (git branch --show-current) remains read-only and safe; no new credential handling, path traversal, or injection surfaces were introduced.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a6ea9bf3f5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[claude]

Claude finished @kyle-sexton's task in 2m 13s —— View job

---

Re-review: New Push to PR #1051

Tasks

• Load tools
• Check new commit diff (81b56dd)
• Read updated SKILL.md
• Assess prior review findings (session-loss gotcha, cross-vendor egress, Codex P2)
• Post comprehensive re-review

---

What the new commit does

81b56dd restores two trigger phrases to the description field's Use-when list that were dropped in the a6ea9bf neutrality reword:

• 'have subagents second-guess the recommendations'
• 'auto-answer then verify the ledger'

This is a correct, targeted fix — these phrases are meaningful auto-invocation triggers that distinguish the accept-then-check flow from a bare "audit answers" invocation. The change is minimal and introduces nothing new.

---

Status of prior review findings

Code Review — Finding 1: Missing session-loss gotcha ❌ Not addressed

The Gotchas section still ends at line 97 without documenting what happens if the session ends after Step 4 (triage visible) but before the human confirms in Step 5. The provisional accepted answers are in-session only — the user must rerun from scratch with no indication this is coming.

Suggested addition to Gotchas (after line 97):

The human confirmation round (Step 5) is the only persistence gate. If the session ends between seeing the triage and confirming, the provisional answers are lost — rerun the skill. Step 4's CONFIRMED collapse is not a checkpoint.

Fix this →

---

Security / Code Review — Finding 2+3: Cross-vendor data egress ❌ Not addressed

Line 48 is unchanged across all three commits:

"prefer a cross-vendor advisor for a validator when one is installed and set up — e.g. the OpenAI Codex plugin, when its documented surface can take this input, invoked per its own docs"

Both the underspecification problem (no heuristic for "its documented surface can take this input") and the egress problem (sends confidential ledger + Brief to a third-party plugin on installation-level consent alone, no per-invocation gate, no userConfig key) remain open. CLAUDE.md is explicit: "Deny by default on unjustified egress or trust delegation."

The same-vendor fresh-context subagent already achieves correlated-blind-spot protection. The cross-vendor preference adds egress risk without adding a meaningfully different independence guarantee — the fresh-context property (different context window, no shared reasoning chain) is what matters, not the vendor.

Options, ranked:

1. Remove the cross-vendor preference — make fresh-context same-vendor the sole stated path. Simplest and correct.
2. Gate behind userConfig — allow_cross_vendor_validation: false by default, with a doc note that enabling it sends planning data to the configured third-party plugin.

Fix this →

---

Codex P2 — General interview artifacts (assessment)

The Codex finding is valid. The Preconditions section (lines 30–31) requires both:

• interview-checklist.md in the topic's memory slice, and
• the ## Brief in PLAN.md

But interview/SKILL.md is explicit (lines 147, 184, 198): "A general session never creates or edits a PLAN.md Brief" — a general-decision interview produces only a shared-understanding summary to the memory slice. Invoking audit-answers after a valid general interview would hit the missing-ledger STOP (no Brief) with no explanation that this skill is engineering-only.

This is a real documentation gap, but scoping it correctly — "engineering interviews only" — is the minimal fix, not restructuring the skill to accept both artifact shapes (the general summary has no Deferred-question arbiter tags, so the never-auto-floor logic doesn't translate). A one-line scope note in the Preconditions and a trigger-phrase qualifier in the description would close it cleanly.

Suggested Preconditions note (after line 31):

This skill is scoped to engineering interviews — a general-decision interview produces no PLAN.md Brief, so there is nothing for the validators to consume. For a general session, audit the shared-understanding summary manually or ask /planning:interview to re-examine a specific decision.

And in the description field's Not-for clause: change "or asking the human the questions the first time" to "asking the human the questions the first time ('/planning:interview'), or validating a general (non-engineering) interview (which produces no Brief)".

Fix this →

---

Summary

The 81b56dd commit is correct — restoring two trigger phrases that auto-invocation depends on is the right call after the neutrality reword dropped them. Three issues remain open from prior reviews. The security egress finding (line 48) is the most important to resolve before merge; the session-loss gotcha and the engineering-only scoping note are small, targeted edits.

[claude]

Claude finished @kyle-sexton's task in 1m 34s —— View job

---

Security Review — PR #1051 (re-run after synchronize)

Tasks

• Load tools
• Read SKILL.md and other changed files
• Assess whether prior security finding was resolved in latest commit
• Post updated review

---

What the latest commit does

81b56dd restores two trigger phrases (have subagents second-guess the recommendations and auto-answer then verify the ledger) to the description frontmatter's Use-when list — phrases that were dropped in the prior neutrality reword. This is a one-line change to the description field only. It does not touch line 48 or any other part of the file body.

---

Findings

IMPORTANT · Confidence: HIGH — Cross-vendor data egress still unaddressed (4th pass)

File: plugins/planning/skills/audit-answers/SKILL.md, line 48

Line 48 is unchanged from the PR's initial state:

"Where the verdict is high-stakes and correlated blind spots are the risk, prefer a cross-vendor advisor for a validator when one is installed and set up — e.g. the OpenAI Codex plugin, when its documented surface can take this input, invoked per its own docs — with the fresh-context same-vendor subagent as the stated fallback, never a route to a command that may not resolve."

The data sent to that third-party system is the filled interview ledger (accepted answers) and the PLAN.md Brief (line 50) — which may contain confidential business decisions, proprietary constraints, acceptance criteria, and architecture choices. This has been flagged in three consecutive rounds of security review (covering commits 04eb729, a6ea9bf) and remains open.

The three specific problems are unchanged:

1. No per-invocation consent. "When one is installed and set up" is installation-level consent, not a per-use authorization. A user who installed an OpenAI plugin for a different workflow has not consented to that plugin receiving their project's PLAN.md Brief during a planning session.

2. The preferred path routes externally on the most sensitive decisions. The instruction explicitly prefers cross-vendor routing for high-stakes verdicts — exactly when the data carries the most weight.

3. Violates repo design rules. CLAUDE.md requires: "Deny by default on unjustified egress or trust delegation." No userConfig key gates this routing; it activates on plugin presence alone.

Options, ranked by preference:

1. Remove the cross-vendor preference entirely — the fresh-context same-vendor subagent already achieves the correlated-blind-spot independence the cross-vendor path is motivated by (different context window, no shared reasoning chain), with no data egress.
2. Gate behind explicit userConfig — add an allow_cross_vendor_validation key defaulting to false, with a note that enabling it sends the filled ledger and Brief to the configured third-party plugin.

Fix this →

---

No other security issues found

The 81b56dd commit is a one-line description-field change (trigger phrase restoration) with no new security surface. All other changed files (marketplace.json, plugin.json, CHANGELOG.md, README.md, interview/SKILL.md, evals.json) remain as reviewed in prior rounds — documentation and metadata with no injection surfaces, credential handling, path traversal, or workflow changes. The git branch --show-current shell command is read-only and safe.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 81b56ddb16

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[claude]

Claude finished @kyle-sexton's task in 2m 52s —— View job

---

Re-review: Merge commit 1d5a35b — PR #1051

Tasks

• Gather context and understand the request
• Read current SKILL.md (post-merge state)
• Check prior open findings against current state
• Assess any new issues from the merge
• Post comprehensive re-review

---

What the merge commit does

1d5a35b reconciles origin/main (the squashed P15a round-format levers from PR #1042) into the branch. SKILL.md is unchanged. All prior open findings carry forward.

---

Status of open findings

#	Finding	Source	Status
1	Session-loss gotcha	Code review	❌ Open
2	Cross-vendor egress (security)	Security review	❌ Open
3	Cross-vendor path underspecified	Code review	❌ Open
4	General-session scope gap	Codex P2	❌ Open
5	Brief-only (no-checklist) scope gap	Codex P2	❌ Open
6	Large-ledger shard merge semantics	Codex P2	❌ Open

---

Detailed status

Finding 1 — Session-loss gotcha ❌

SKILL.md line 97 ends the Gotchas section with "do not treat the collapsed CONFIRMED block as the finish." It still does not state what happens if the session ends between Step 4 and Step 5: provisional answers are in-session only, the user restarts from scratch with no warning.

Suggested addition (after line 97):

The human confirmation round (Step 5) is the only persistence gate. If the session ends between seeing the triage and confirming, the provisional answers are lost — rerun the skill. Step 4's CONFIRMED collapse is not a checkpoint.

Fix this →

---

Finding 2+3 — Cross-vendor egress + underspecified path ❌ (5th pass)

SKILL.md line 48 is unchanged through all five passes:

"prefer a cross-vendor advisor for a validator when one is installed and set up — e.g. the OpenAI Codex plugin, when its documented surface can take this input, invoked per its own docs"

The two problems remain co-located:

• Security (egress): "When one is installed" is installation-level consent, not per-use authorization. The data sent — filled ledger + PLAN.md Brief — may contain confidential business decisions, proprietary constraints, acceptance criteria. CLAUDE.md is unambiguous: "Deny by default on unjustified egress or trust delegation." No userConfig key gates this; it fires on plugin presence alone.
• Correctness (underspecified): The phrase "when its documented surface can take this input" provides no heuristic for discovery. Without one, Claude must guess — which is the failure mode the fallback clause is there to prevent.

The same-vendor fresh-context subagent already achieves the independence goal (different context window, no shared reasoning chain) with no egress. The cross-vendor path adds risk without a qualitatively different guarantee.

Options ranked:

1. Remove the cross-vendor preference — make fresh-context same-vendor the sole path.
2. Gate behind userConfig — allow_cross_vendor_validation: false by default, with a note that enabling it sends planning data to the configured third-party plugin.

Fix this →

---

Finding 4 — General-session scope gap ❌ (Codex P2)

interview/SKILL.md lines 147 and 184 are explicit: a general (non-engineering) session never creates a PLAN.md Brief. The audit-answers preconditions (lines 30–31) require one, so invoking after a valid general interview hits the missing-ledger STOP with no explanation that this skill is engineering-only.

Smallest fix: one scope sentence in Preconditions and a Not-for qualifier in the description.

Fix this →

---

Finding 5 — Brief-only (no-checklist) scope gap ❌ (Codex P2)

interview/SKILL.md line 36: the checklist is only emitted for sessions with ≥2 open questions OR explicit me mode. A completed engineering interview in lock mode or auto with clear intent produces a Brief but no checklist. Audit-answers requires the checklist as a precondition (line 30), so it stops with "nothing to validate" even when the Brief exists.

The practical question is whether a Brief-synthesized-without-Q&A deserves audit-answers treatment — if intent was crystal-clear, the ledger of open decisions is empty and there's nothing to validate per-answer. But the preconditions should be explicit about this boundary rather than leaving the user to infer it from a generic STOP. A "Brief exists but no checklist" path should either explain the case and stop cleanly, or accept it and note that only the Deferred questions in the Brief are in scope.

Fix this →

---

Finding 6 — Large-ledger shard merge semantics ❌ (Codex P2)

SKILL.md line 48 says "up to three to split coverage of a large one," implying non-overlapping answer subsets per validator. But Step 4 line 66 requires merging verdicts using "CONFIRMED by all" as the collapse condition. For any answer that only one validator examined, "CONFIRMED by all" is vacuously true for the other validators — they simply didn't see it. This silently undermines the cross-check guarantee: sharded answers collapse as if independently verified, but they weren't.

Fix: either require every validator to examine every answer (and "split focus/depth" rather than "split coverage"), or add explicit shard-merge semantics that distinguish "validated by one" from "validated by all."

Fix this →

---

Summary

The merge commit is a clean base update — no content changes to SKILL.md. All six open findings from prior review rounds carry forward unaddressed. Finding 2 (cross-vendor egress) is the highest-priority block — it's a CLAUDE.md design rule violation. Findings 1, 4, 5, 6 are targeted documentation edits. Finding 3 (underspecified discovery) is resolved by whichever option fixes Finding 2. Findings 4–6 are each small, independent fixes.

[claude]

Claude finished @kyle-sexton's task in 1m 27s —— View job

---

Security Review — PR #1051 (post-merge synchronize)

Tasks

• Load tools
• Read PR diff and changed files
• Analyze for security issues
• Post findings

---

Findings

IMPORTANT · Confidence: HIGH — Unguarded cross-vendor data egress (5th pass; still open)

File: plugins/planning/skills/audit-answers/SKILL.md, line 48

This line has been flagged in every prior security review pass (commits 04eb729, a6ea9bf, 81b56dd) and is unchanged in the latest push:

"Where the verdict is high-stakes and correlated blind spots are the risk, prefer a cross-vendor advisor for a validator when one is installed and set up — e.g. the OpenAI Codex plugin, when its documented surface can take this input, invoked per its own docs — with the fresh-context same-vendor subagent as the stated fallback, never a route to a command that may not resolve."

The data sent to the third-party system on this path is named on line 50: the filled interview ledger (accepted answers) and the PLAN.md Brief — which typically contains confidential architecture choices, proprietary constraints, acceptance criteria, and business decisions.

The three specific problems are unchanged:

1. No per-invocation consent. "When one is installed and set up" binds to a plugin install event, not a per-use authorization. A user who installed a third-party AI plugin for a different purpose has not consented to that plugin receiving their project's PLAN.md Brief mid-session.

2. The preferred route carries the most sensitive data. The instruction explicitly reserves the cross-vendor path for high-stakes verdicts — exactly the cases where the data is most sensitive.

3. Violates the repo's explicit egress rule. CLAUDE.md states: "Deny by default on unjustified egress or trust delegation." No userConfig key guards this path; it fires on plugin presence alone.

Recommended fix — option 1 (preferred): Remove the cross-vendor preference entirely and make fresh-context same-vendor the sole stated path. Fresh-context (non-fork) subagents already deliver the correlated-blind-spot protection the cross-vendor path is motivated by (different context window, no shared reasoning chain), with zero data egress.

Recommended fix — option 2: Gate behind an explicit userConfig boolean:

userConfig:
  allow_cross_vendor_validation:
    type: boolean
    default: false
    description: >
      When true, the audit-answers skill may route the filled interview ledger and
      PLAN.md Brief to a configured third-party AI plugin for validation. Enabling
      this sends potentially confidential project data to an external vendor.

Fix this →

---

No other security issues found

The latest push (1d5a35b) is a merge of origin/main into the feature branch — no new content was introduced. The prior content commits are documented in earlier review passes. All other changed files (marketplace.json tag, plugin.json version bump, CHANGELOG.md, README.md, interview/SKILL.md composition row, evals.json) remain documentation and metadata with no injection surfaces, credential handling, path traversal, or workflow changes. The pre-computed shell command (git branch --show-current 2>/dev/null || echo "unknown") is read-only and safe; $ARGUMENTS is consumed by the model for slug derivation, not passed to a shell.

[kyle-sexton]

🔒 babysit-prs lane claiming this PR for on-branch fix work this cycle (amendment-round: 16, safe tier). Will fix clear branch-owned findings and push; will not resolve threads or merge (safe tier).

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1d5a35bca6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[claude]

Claude finished @kyle-sexton's task in 2m 13s —— View job

---

Re-review: Commit b74fa96 — PR #1051

Tasks

• Gather context and understand the request
• Read current SKILL.md (post-commit b74fa96)
• Diff this commit against prior state (1d5a35b)
• Assess all prior open findings
• Post comprehensive re-review

---

What b74fa96 does

A substantive fix commit titled "address Codex review on audit-answers." It rewrites the Preconditions section and makes targeted improvements to Steps 1, 2, 4, and 5, plus updates eval 2.

Changes made:

• Preconditions fully generalized — no longer requires both interview-checklist.md + PLAN.md Brief. Now explicitly handles all four interview artifact shapes: engineering Brief, Brief-only (auto/lock, no checklist), decision-tree ledger, and general session's shared-understanding summary. Only stops when no interview output at all exists.
• Step 2 — sharding eliminated — "up to three to split coverage of a large one" is replaced with "each reviewing the whole answer set — the count scales independent redundancy (more validators = more independent cross-checks on every answer), never a sharding of the work." The why is made explicit: "splitting coverage would make 'confirmed by all' indistinguishable from 'unseen by some'."
• USER-RESERVED carry-forward explicit in both Step 1 and Step 4 — Step 1 now says "it is not auto-accepted, not validated, and not turned into an audit question here; it passes through untouched, arbiter tag intact." Step 4 lists USER-RESERVED as a separate carry-forward bucket distinct from CHALLENGED/RECLASSIFIED.
• Step 5 persistence generalized — now writes back to "the Brief or the general summary, and the ledger when the interview kept one" rather than the old "tracked ledger and Brief."
• Eval 2 updated — expected output and expectations now correctly describe USER-RESERVED as a carry-forward item, not an audit question that gets resolved.

This is a high-quality fix batch. All four Codex P2 findings it targets are resolved correctly.

---

Status of all prior findings

Codex P2 — Finding: General-session scope gap ✅ RESOLVED

The Preconditions section at SKILL.md line 28–34 now explicitly lists all artifact shapes including "a general (non-engineering) session's shared-understanding summary in the memory slice." A summary-only input is called out as valid, not a STOP.

Codex P2 — Finding: Brief-only (no-checklist) scope gap ✅ RESOLVED

Line 34: "A Brief-only interview (an auto/lock session with no checklist) and a summary-only general interview are both valid inputs, not a reason to stop." This is exactly the right language.

Codex P2 — Finding: Shard merge semantics ✅ RESOLVED

Step 2 (line 49): "each reviewing the whole answer set — the count scales independent redundancy … never a sharding of the work … splitting coverage would make 'confirmed by all' indistinguishable from 'unseen by some' and is not done." Correct and explicit.

Codex P2 — Finding: USER-RESERVED audit-question escalation ✅ RESOLVED

Step 1 (line 42) and Step 4 (line 71) both correctly distinguish USER-RESERVED (carry-forward, never an audit question) from auto-guard choices (route to human as a question). The gotcha on line 97 also explicitly names this.

---

Code Review — Finding 1: Missing session-loss gotcha ❌ Still open

The Gotchas section ends at line 99 with "Do not treat the collapsed CONFIRMED block as the finish — the human confirmation round is the stop condition." That sentence is correct but doesn't tell the user what happens if the session ends between Step 4 and Step 5: the provisional answers are in-session only and would be silently lost, requiring a full rerun. This is the one non-obvious failure mode worth calling out.

Suggested addition after line 99:

The human confirmation round (Step 5) is the only persistence gate — provisional answers are in-session only. If the session ends between seeing the triage (Step 4) and confirming, the provisional answers are lost; rerun the skill.

Fix this →

---

Security / Code Review — Findings 2+3: Cross-vendor data egress ❌ Still open

SKILL.md line 49 is unchanged through all commits:

"Where the verdict is high-stakes and correlated blind spots are the risk, prefer a cross-vendor advisor for a validator when one is installed and set up — e.g. the OpenAI Codex plugin, when its documented surface can take this input, invoked per its own docs — with the fresh-context same-vendor subagent as the stated fallback, never a route to a command that may not resolve."

The data sent on this path is named at line 51: the provisional answer set and the interview's persisted context (Brief or general summary) — which may contain confidential business decisions, acceptance criteria, and proprietary constraints.

Three problems remain unchanged:

1. No per-invocation consent. "When one is installed and set up" is installation-level consent, not per-use authorization.
2. Preferred path on most sensitive decisions. The instruction explicitly prefers the cross-vendor route for high-stakes verdicts.
3. Violates CLAUDE.md design rule. "Deny by default on unjustified egress or trust delegation." No userConfig key gates this routing; it fires on plugin presence alone.

Additionally: "when its documented surface can take this input" still provides no heuristic for discovery. A model receiving this instruction must guess — which is exactly the failure mode the fallback clause is trying to prevent.

Note: the b74fa96 commit now makes this path potentially send general session summaries too (not just engineering Briefs), slightly broadening the egress surface with the new precondition generalization.

Options, ranked:

1. Remove the cross-vendor preference — make fresh-context same-vendor the sole stated path. Step 2 now correctly explains why each validator must use a fresh context window; that property is vendor-independent. The independence guarantee is the same.
2. Gate behind userConfig — allow_cross_vendor_validation: false by default, with a note that enabling it sends interview output to the configured third-party plugin.

Fix this →

---

Minor observation: Eval 3 expectation text is stale

evals.json line 35 still reads:

"Output recognizes the precondition: a filled interview ledger (interview-checklist.md decision-tree ledger + PLAN.md Brief) must exist"

With the Preconditions rewrite, the skill now accepts Brief-only and general summary inputs — neither requires the interview-checklist.md ledger. The expectation text should be updated to match: something like "Output recognizes the precondition: a completed /planning:interview output (Brief, decision-tree ledger, or general summary) must exist."

This is non-blocking but worth updating for eval accuracy.

---

Summary

b74fa96 cleanly addresses all four Codex P2 findings — the precondition generalization, no-sharding semantics, USER-RESERVED carry-forward, and artifact-aware persistence are all correct and well-stated. The rewrite is a meaningful quality improvement.

Two issues from prior reviews remain open:

1. Security (egress, line 49) — highest priority block; violates CLAUDE.md deny-by-default rule; unchanged across all commits.
2. Session-loss gotcha — one sentence to add, low effort.

One minor: eval 3's expectation text references the old dual-artifact precondition.

[claude]

Claude finished @kyle-sexton's task in 1m 45s —— View job

---

Security Review — PR #1051 (HEAD b74fa96)

Tasks

• Load tools
• Read current SKILL.md and changed files at HEAD (b74fa96)
• Assess all prior open security findings against current state
• Report findings

---

What changed in the latest commit (b74fa96)

The commit addresses four Codex P2 findings:

• Preconditions now accept all interview output shapes (Brief, Brief-only, checklist ledger, general summary) — not just the Brief + checklist pair.
• Validators each review the whole answer set; the 1–3 count scales independent redundancy, never shards coverage. This fixes the "CONFIRMED by all" indistinguishable from "unseen by some" logic flaw.
• USER-RESERVED deferred questions no longer become audit questions — they carry forward untouched to /planning:plan with their arbiter tag intact.
• Persistence writes back to whichever artifact the interview used, leaving USER-RESERVED in place.

These are all correct fixes. None introduces new security surface.

---

Findings

IMPORTANT · Confidence: HIGH — Unguarded cross-vendor data egress (6th pass; still open)

File: plugins/planning/skills/audit-answers/SKILL.md, line 49

The b74fa96 commit did not touch line 49. The instruction is unchanged across all six review passes (commits 04eb729, a6ea9bf, 81b56dd, merge 1d5a35b, b74fa96):

"Where the verdict is high-stakes and correlated blind spots are the risk, prefer a cross-vendor advisor for a validator when one is installed and set up — e.g. the OpenAI Codex plugin, when its documented surface can take this input, invoked per its own docs — with the fresh-context same-vendor subagent as the stated fallback, never a route to a command that may not resolve."

The data sent on this path is named on line 51: the whole provisional answer set and the interview's persisted context (Brief or general summary) — which typically contains confidential architecture decisions, proprietary constraints, acceptance criteria, and business decisions.

Three specific problems remain:

1. No per-invocation consent. "When one is installed and set up" is installation-level consent, not a per-use authorization. A user who installed a third-party AI plugin for a completely different workflow has not consented to it receiving their project's PLAN.md Brief mid-session.

2. The preferred route carries the most sensitive data on the most sensitive decisions. The instruction explicitly reserves the cross-vendor path for high-stakes verdicts — exactly when the data is most sensitive.

3. Violates the repo's explicit egress rule. CLAUDE.md states: "Deny by default on unjustified egress or trust delegation." No userConfig key guards this path; it fires on plugin presence alone.

Options, ranked:

1. Remove the cross-vendor preference entirely — make fresh-context same-vendor the sole stated path. The same-vendor fresh-context subagent already delivers the correlated-blind-spot protection this path is motivated by (different context window, no shared reasoning chain), with zero data egress. This is the minimal correct fix.

2. Gate behind explicit userConfig — add allow_cross_vendor_validation (default false) with a note that enabling it sends the filled answer set and Brief/summary to the configured third-party plugin.

Fix this →

---

Resolved by b74fa96

Prior finding	Status
Codex P2: General-session scope gap (no Brief produced)	✅ Fixed — preconditions now accept all output shapes
Codex P2: Brief-only (no checklist) scope gap	✅ Fixed — explicitly handled
Codex P2: USER-RESERVED deferrals turned into audit questions	✅ Fixed — carry-forward untouched, not audited
Codex P2: Large-ledger shard merge semantics (vacuous CONFIRMED)	✅ Fixed — whole-answer-set per validator, sharding prohibited

---

No other security issues found

The b74fa96 changes are documentation — precondition restructuring, Step 1 and Step 4/5 prose updates, and an eval fix. No new injection surfaces, no credential handling, no path traversal, no workflow changes. The shell command (git branch --show-current 2>/dev/null || echo "unknown") is read-only and safe; $ARGUMENTS is consumed by the model for slug derivation, not passed to a shell.

The cross-vendor egress issue (line 49) is the sole remaining block. It is a design-rule violation and the only finding warranting a fix before merge.

[kyle-sexton]

All four P2 findings addressed in b74fa96: (1) precondition now accepts any interview output — engineering Brief, Brief-only (no checklist) sessions, the ledger, or a general session's shared-understanding summary — stopping only when nothing is persisted; (2) validators each review the whole answer set (the 1-3 count scales independent redundancy, no sharding), so 'confirmed by all' stays meaningful; (3) USER-RESERVED deferrals stay deferred to /planning:plan with their arbiter tag, never turned into audit questions; (4) persistence writes back to whichever artifact the interview used. Thanks.

[kyle-sexton]

🔒 babysit-prs safe-tier classification — head 1d5a35b. Every open finding here is a design decision on this skill, not a clear mechanical branch-owned fix, so this pass classifies + surfaces them and changes no code and resolves no threads (safe tier). Claims validated against the code on the PR head.

#	Source	Finding	Classification	Evidence	Action
1	claude[bot] code review	Session-loss gotcha (SKILL.md L97) — no note that provisional answers are lost if the session ends between Step 4 and Step 5	VALID	L39 marks provisional answers "NOT persisted … yet"; L73 "Persistence happens here and only here" (Step 5). A crash before Step 5 loses them — undocumented.	Surfaced to author (additive doc note); not auto-applied
2	claude[bot] security · IMPORTANT (5th pass)	Cross-vendor egress (SKILL.md L48) — cross-vendor advisor preferred "when one is installed", gating on plugin presence not per-invocation consent; sends filled ledger + PLAN.md Brief to a third-party vendor	VALID	L48 fires on plugin presence alone; L50 names the egressed data. CLAUDE.md: "Deny by default on unjustified egress or trust delegation." No userConfig gates it.	DESIGN DECISION for author. RECOMMENDED: option 1 — remove the cross-vendor preference (fresh-context same-vendor already delivers producer≠critic independence with zero egress); alt option 2 — gate behind a userConfig boolean defaulting false. Not auto-decided (safe tier)
3	claude[bot] code review	Cross-vendor path underspecified (L48) — "when its documented surface can take this input" gives no discovery heuristic	VALID	Same line; no resolution heuristic for the surface.	Resolves together with #2 (whichever option is chosen)
4	codex P2 (L31)	General-session scope gap — Preconditions require a PLAN.md Brief, but general (non-engineering) interviews never produce one	VALID	interview/SKILL.md L148 & L189: "A general session never creates or edits a PLAN.md Brief." audit-answers L31 requires it → STOP with no engineering-only explanation.	DESIGN DECISION for author — narrow scope to engineering interviews (Preconditions + Not-for) vs consume the general shared-understanding summary. Not auto-decided
5	codex P2 (L30)	Brief-only scope gap — interview-checklist.md required, but lock/auto sessions can produce a Brief without a checklist	VALID	interview/SKILL.md L36: checklist emitted only for "≥2 open questions OR explicit me mode." audit-answers L30 requires it → STOP "nothing to validate."	DESIGN DECISION for author — explain-and-stop-clean vs narrow to Deferred-questions-only when no checklist. Not auto-decided
6	codex P2 (L48)	Large-ledger shard merge semantics — "split coverage" contradicts Step 4 "CONFIRMED by all"	VALID	L48 "split coverage of a large one" (non-overlapping subsets) vs L68 "CONFIRMED by all" collapse — vacuously true for validators that never saw an answer, silently voiding the cross-check.	DESIGN DECISION for author — every validator sees all answers ("split focus/depth") vs explicit shard-merge semantics. Not auto-decided
7	codex P2 (L69)	USER-RESERVED deferrals turned into audit questions	UNCERTAIN	The skill routes USER-RESERVED to a numbered human question (L69) and does NOT auto-resolve it (L78 "Does not auto-resolve a reserved decision"), so "resolves it during audit-answers" is imprecise. The timing concern (re-ask now vs preserve as carry-forward to /planning:plan's approval gate) is a real design nuance.	DESIGN DECISION for author — confirm intended timing. Not auto-decided

CI is green except the in-progress review / review check; mergeStateStatus is BLOCKED (the four unresolved codex threads + these open findings). The blocking item is #2 (cross-vendor egress) — a CLAUDE.md egress-rule decision the author must make before this is mergeable.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b74fa96e75

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Keep USER-RESERVED deferrals out of the summary

Fresh evidence: the skill body now says USER-RESERVED Deferred questions stay carry-forward and are not turned into audit questions, but this description still tells the model/user that user-reserved decisions “come back as real questions.” Because frontmatter is the trigger/summary Claude sees before applying the detailed steps, invocations with a USER-RESERVED deferred pricing/scope decision can still route that plan-gate item into this audit confirmation round instead of preserving its arbiter for /planning:plan.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Align the no-output eval with valid Brief-only inputs

Fresh evidence: after the skill's Preconditions accept a Brief-only interview and a summary-only general interview, this eval still asserts that a filled interview-checklist.md plus PLAN.md must exist. Model-graded evals using this case can reward the obsolete “no ledger ⇒ stop” behavior and penalize the intended behavior for valid completed interviews that only persisted a Brief or summary; adjust the scenario to mean no persisted interview output at all.

Useful? React with 👍 / 👎.