plan: add reasoning_depth A/B experiment

[Copilot]

Instruments the plan workflow with a three-variant reasoning_depth experiment to measure whether a lighter one-pass prompt (shallow) or an explicit reflection step (deep) improves sub-issue quality and/or reduces cost versus the current behavior (baseline).

Frontmatter

• Adds experiments: reasoning_depth block with variants [shallow, baseline, deep], weights [34, 33, 33], 137 minimum samples per variant, and guardrails on workflow_success_rate (≥0.95) and empty_plan_rate (≤0.02).

Prompt body

Adds a ## Planning Approach conditional block immediately before ## Begin Planning:

{{#if experiments.reasoning_depth == 'shallow'}}
## Planning Approach
Use a concise, single-pass approach. Scan the issue for 3-5 concrete deliverables
and immediately draft sub-issues. Skip elaborate reflection…
{{#elseif experiments.reasoning_depth == 'deep'}}
## Planning Approach
Before drafting any sub-issue, complete an explicit reflection step:
1. Enumerate all distinct work items…
2. Sequence check…
3. Independence check…
4. Acceptance-criteria check…
{{#else}}
## Planning Approach
Use the standard planning approach…
{{/if}}

Lock file

Regenerated plan.lock.yml via gh aw compile plan — zero errors.

[github-actions]

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

[github-actions]

✅ PR Code Quality Reviewer completed the code quality review.

[github-actions]

✅ Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR does not have the 'implementation' label and has 0 new lines of code in business logic directories (threshold: 100).

[github-actions]

❌ Test Quality Sentinel failed during test quality analysis.

[Copilot]

Pull request overview

This PR adds an A/B/n experiment (reasoning_depth) to the plan workflow to compare three planning instruction variants (shallow/baseline/deep), and updates the prompt to conditionally inject variant-specific “Planning Approach” guidance before the planning begins.

Changes:

• Adds experiments.reasoning_depth frontmatter with variants, weights, minimum samples, and guardrail metrics.
• Adds a conditional ## Planning Approach block that changes planning instructions based on the selected experiment variant.
• Regenerates plan.lock.yml to wire experiment selection/state persistence into the compiled GitHub Actions workflow.

Show a summary per file

File	Description
.github/workflows/plan.md	Defines the reasoning_depth experiment and conditionally injects variant-specific planning instructions into the prompt.
.github/workflows/plan.lock.yml	Compiled workflow update that introduces experiment pick/state steps and updates runtime/action/container pins.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 2/2 changed files
• Comments generated: 3
• Review effort level: Low

[github-actions]

🤖 PR Triage

Field	Value
Category	feature
Risk	🟡 Medium
Score	30 / 100
Action	⏸️ defer

Score breakdown: Impact 20 · Urgency 5 · Quality 5

Draft/WIP. Adds A/B experiment campaign for reasoning depth to plan.md + recompiled lock file. CI unstable. Low priority until experiment design is finalized and PR is ready for review.

Run §28591321991

Generated by 🔧 PR Triage Agent · 76 AIC · ⌖ 9.74 AIC · ⊞ 1.6K · ◷

[github-actions]

REQUEST_CHANGES: three high-severity issues block this experiment from working correctly.

Blocking issues (3 HIGH)

1. Unresolvable primary metric — has no collection mechanism; the experiment cannot be evaluated against its own hypothesis.
2. Supply chain risk — is the only action in the manifest pinned by mutable tag rather than commit SHA; all other actions use immutable SHAs.
3. Container image regression — firewall images downgraded from 0.27.20 to 0.27.11 and from v1.5.0 to v1.4.0 with no documented reason; this could re-introduce fixed bugs on every invocation.

Non-blocking concerns (3 MEDIUM)

4. **No ** — experiment runs indefinitely, requires manual cleanup PR.
5. ** v5.0.5 vs v6.1.0** — one step selectively uses an older version with no explanation.
6. Artifact retention 30 days, no end_date — run data older than 30 days will be deleted before the sample set is complete.

🔎 Code quality review by PR Code Quality Reviewer · 93.9 AIC · ⌖ 10.2 AIC · ⊞ 1.6K
_{Comment /review to run again}

[github-actions]

** is pinned by mutable tag, not a commit SHA** — every other action in this manifest has an immutable SHA; this one has , silently re-pointable.

💡 Details

The manifest header shows:

Every other action entry (e.g. , ) carries a real 40-hex commit SHA. A tag can be force-pushed without any PR or audit trail. If is force-pushed to a malicious commit, all jobs using this action will silently execute untrusted code with permissions.

Resolve by pinning to the commit SHA the tag currently points to:

[github-actions]

** is declared as the primary metric but has no defined collection path in this PR** — the experiment cannot be analysed.

💡 Details

The hypothesis states:

H1: deep improves quality score by >=8%

But neither nor the new pipeline emits a value. Without instrumentation, the guardrail check against this metric is a no-op and the experiment will only ever produce secondary metrics (, , ).

Before merging, either:

• Add a scoring step that emits to the experiment artifact, or
• Replace the primary metric with one already collected (e.g. ) and document the proxy rationale in the field.

[github-actions]

No — this experiment will run indefinitely past its intended window.

💡 Details

The workflow classifies an experiment as while ; without it never terminates automatically. only gates analysis readiness, not experiment shutdown.

A permanently active experiment continuously splits production traffic among all three variants, making the prompt permanently larger and requiring a manual follow-up PR to remove the conditionals.

Add an explicit :

[github-actions]

** selectively downgraded from v6.1.0 to v5.0.5 for this one step only** — unexplained inconsistency across the same lock file.

💡 Details

All other cache/restore usages in this file use (v6.1.0), but the daily-AIC-cache restore step was changed to (v5.0.5). The manifest header now carries two separate SHAs for the same action.

This inconsistency has no explanation in the PR description and is suspicious: if v5 was needed here for compatibility, that should be documented. If it is an error, it silently pins one cache step to an older version that may lack bugfixes.

Either upgrade to v6.1.0 consistently, or add a comment explaining the required downgrade.

[github-actions]

Experiment artifact retention (30 days) is shorter than the experiment could run — early-run data will be deleted before analysis completes.

💡 Details

With required and no , this experiment could easily run for more than 30 days if the command is not invoked frequently. GitHub Actions artifacts are automatically deleted after the window; any run artifacts older than 30 days will be gone by the time you are ready to analyse the full sample set.

Recommendations:

1. Set an (see the separate comment on ) so you can size the retention window to match.
2. Or increase to at least 60-90 days as a buffer.
3. Or ensure the job writes cumulative state to the branch on every run, so the branch (not the artifact) is the source of truth for analysis.

[github-actions]

All firewall container images and MCP server were downgraded without explanation — 0.27.20 → 0.27.11, v1.5.0 → v1.4.0.

💡 Details

The previous lock file used firewall image and . This PR downgrades both to earlier versions. Lock files are compiled outputs, and the new lock was compiled with compiler v0.81.6 (added field), which implies these older image tags are what v0.81.6 emits.

If this is intentional (the new compiler pins to an older runtime for compatibility), that should be documented. If it is unintentional, it may re-introduce bugs or CVEs fixed in the higher versions, and the workflow will run against an older MCP server API surface.

Verify that the downgraded image versions are intentional and compatible with the new experiment features added in this PR.

[github-actions]

Skills-Based Review 🧠

Applied /grill-with-docs and /tdd — commenting with 5 findings, no blocking issues but a few items worth addressing before the experiment concludes.

📋 Key Themes & Highlights

Key Themes

• Undefined primary metric (line 48): plan_quality_score has no collection mechanism described in this PR — the experiment's H1 threshold can't be evaluated without it.
• No experiment lifetime bound (line 57): Missing end_date means the experiment runs indefinitely if /plan volume is low or results are not acted on.
• Fragile positional weight mapping (line 58): weight: [34, 33, 33] maps silently to [shallow, baseline, deep] — a future reorder would silently mis-assign weights.
• Implicit baseline branch (line 174): The {{#else}} silently catches any unknown variant as baseline; naming it explicitly makes the contract clearer.
• Minor: shallow lower bound (line 162): "3-5 deliverables" anchors a lower bound that the other variants don't have; intentional or not, it should be documented.

Positive Highlights

• ✅ Well-structured experiment spec with clear hypothesis, guardrail metrics, and secondary metrics
• ✅ Variant text is precise and actionable — the deep reflection steps are particularly well-specified
• ✅ Experiment artifact upload and state-restore steps are properly guarded with if: always()
• ✅ Guardrail thresholds (workflow_success_rate >= 0.95, empty_plan_rate <= 0.02) are sensible safety nets

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · 77.9 AIC · ⌖ 9 AIC · ⊞ 6.6K
_{Comment /matt to run again}

[github-actions]

[/grill-with-docs] plan_quality_score is listed as the primary metric, but there is no definition of how it is measured or collected anywhere in this PR.

This metric drives the experiment conclusion (H1 threshold is >=8%), yet without a collection mechanism the experiment cannot be evaluated against it.

💡 Suggestion

Either:

1. Add a note in the frontmatter or a linked doc explaining where plan_quality_score is produced and how it is persisted (e.g., a separate evaluation workflow, a manual labeling step), or
2. Replace it with a metric that is already automatically collected such as issue_creation_count or output_token_estimate until a quality-scoring pipeline exists.

@copilot please address this.

[github-actions]

[/grill-with-docs] No end_date or maximum-duration guardrail is defined for this experiment.

Without a stop condition, the experiment could run indefinitely if min_samples is never reached (e.g., the /plan workflow is rarely triggered), or if the results are not actioned after completion.

💡 Suggestion

Add an end_date field to bound the experiment lifetime:

    min_samples: 137
    end_date: "2026-09-01"  # auto-conclude after this date

This also makes it clear when to revisit and decide whether to promote a winner or abort.

@copilot please address this.

[github-actions]

[/grill-with-docs] The weight array [34, 33, 33] maps positionally to variants: [shallow, baseline, deep], but this mapping is implicit and fragile — reordering either array would silently change which variant gets which weight.

A reader or future editor has no in-line signal that 34 corresponds to shallow.

💡 Suggestion

If the schema supports named weights, prefer that form:

    weight:
      shallow: 34
      baseline: 33
      deep: 33

If only array form is supported, add a comment immediately after the weights:

    weight: [34, 33, 33]  # shallow, baseline, deep

@copilot please address this.

[github-actions]

[/grill-with-docs] The shallow variant prompt says "Scan the issue for 3-5 concrete deliverables" but the workflow-level safe-outputs hard-cap is also max: 5. The shallow description anchors the user to 3-5, but the baseline and deep variants only say "at most 5" (from ## Your Mission).

This introduces a soft inconsistency: shallow has a tighter self-imposed lower bound (3) while the other variants have no stated lower bound.

💡 Suggestion

If a minimum of 3 sub-issues is intentional for shallow, document why (to prevent trivially small plans). If not intentional, align to "up to 5" to match the other variants:

Scan the issue for concrete deliverables (up to 5) and immediately draft sub-issues.

@copilot please address this.

[github-actions]

[/tdd] The {{#else}} branch represents the baseline variant, but the condition is unnamed — if a new variant is added in the future and the #elseif chain is not updated, that new variant silently falls through to baseline behavior.

💡 Suggestion

Make the fallthrough explicit by checking for baseline:

{{#elseif experiments.reasoning_depth == 'baseline'}}
## Planning Approach

Use the standard planning approach...
{{#else}}
{{!-- Unknown variant: treat as baseline, but log a warning if possible --}}
## Planning Approach

Use the standard planning approach...
{{/if}}

This makes the experiment's variant contract machine-readable and prevents silent behavioral drift when variants are added or renamed.

@copilot please address this.

[github-actions]

🎉 This pull request is included in a new release.

Release: v0.82.2