Scout: Statistical Evaluation Framework (from eval-robuste)

[bacoco]

Statistical Evaluation Framework

Source: Alexmacapple/alex-claude-skill
Score: 4.3/5.0 (Impact: 4, Novelty: 5, Applicability: 4, Effort: 4)

Three techniques from eval-robuste

1. Prompt Hash Chaining for Baseline Integrity
Hash the full evaluation chain (SKILL.md + zone defs + agent prompts + git SHA). Store in results. Reject comparisons when hash changes — prevents confusing prompt changes with code improvements.

2. Deterministic Stats Delegation
ALL math (mean, stdev, CI, verdict) delegated to a Python script. LLM never computes numbers. Any math by the LLM is a protocol violation. Add score_audit.py to ShipGuard.

3. Intrinsic Stability Verdict (NOISE/REGRESSION/IMPROVEMENT)
Run audit N times, compute stdev of finding counts. STABLE if stdev < threshold. With baseline: NOISE/REGRESSION/IMPROVEMENT based on |delta| > sigma * ref_stdev. Prevents chasing phantom regressions from LLM stochasticity.

What ShipGuard should do

Build a sg-eval skill or extend sg-improve with:

• prompt_hash.py — hash evaluation chain, store in results, reject invalid comparisons
• score_audit.py — deterministic aggregation (no LLM math)
• N-run stability detection with NOISE/REGRESSION/IMPROVEMENT verdicts

Affected skill: sg-improve
Mutation type: add_constraint
Scouted: 2026-04-16

[traycerai]

Warning

Ticket Assist will be discontinued on May 1, 2026. After that date, the feature will no longer be available. We suggest installing the Traycer IDE Extension for continued AI-powered, spec-driven development. Thank you for choosing Traycer!

Plan

Observations

The ShipGuard plugin is structured as a set of SKILL.md prompt files under plugins/shipguard/skills/. The sg-improve skill (file:plugins/shipguard/skills/sg-improve/SKILL.md) already has a rich 6-phase loop with structured YAML learnings, snapshot/rollback, and GitHub issue filing. The sg-code-audit skill (file:plugins/shipguard/skills/sg-code-audit/SKILL.md) reads .shipguard/learnings.yaml at startup. There are no Python scripts in the repo yet — all logic lives in LLM-interpreted SKILL.md prompts.

Approach

The ticket asks for three additions: prompt hash chaining, deterministic stats delegation via score_audit.py, and N-run stability verdicts (NOISE/REGRESSION/IMPROVEMENT). Since the codebase is entirely prompt-driven SKILL.md files, the right approach is:

1. Add prompt_hash.py and score_audit.py as new Python helper scripts (the LLM shells out to them — consistent with how sg-code-audit already shells out to python ast, tsc, go build, etc.).
2. Extend sg-improve/SKILL.md with a new Phase 0.5 — Prompt Hash section and a new Phase 2b — Stability Verdict section.
3. Optionally surface this as a new /sg-eval invocation alias inside sg-improve.

sequenceDiagram
    participant User
    participant sg-improve
    participant prompt_hash.py
    participant score_audit.py
    participant learnings.yaml

    User->>sg-improve: /sg-improve (or /sg-eval)
    sg-improve->>prompt_hash.py: hash(SKILL.md files + zone defs + git SHA)
    prompt_hash.py-->>sg-improve: chain_hash
    sg-improve->>sg-improve: compare chain_hash vs stored hash in learnings.yaml
    alt hash mismatch
        sg-improve-->>User: WARN — prompt chain changed, baseline comparison rejected
    end
    sg-improve->>sg-improve: Phase 1–2: collect audit-results.json (N runs)
    sg-improve->>score_audit.py: pass raw finding counts per run
    score_audit.py-->>sg-improve: mean, stdev, CI, verdict (STABLE/NOISE/REGRESSION/IMPROVEMENT)
    sg-improve->>learnings.yaml: store chain_hash + stability verdict + stats
    sg-improve-->>User: Summary with verdict

Loading

---

Implementation Steps

1. Create plugins/shipguard/scripts/prompt_hash.py

This script is shelled out to by sg-improve — the LLM never computes the hash itself.

• Accept a list of file paths as CLI arguments (or read from stdin as a newline-separated list).
• Read each file's content in sorted, deterministic order: the SKILL.md files for sg-code-audit, sg-improve, sg-scout; any zone definition files passed in; and the current git rev-parse HEAD SHA (obtained via subprocess).
• Concatenate all content with a stable delimiter, then produce a sha256 hex digest using Python's built-in hashlib.
• Print a single JSON line: {"chain_hash": "<hex>", "inputs": ["<file1>", ...], "git_sha": "<sha>"}.
• Exit code 0 on success, 1 on any missing file.

2. Create plugins/shipguard/scripts/score_audit.py

This script receives raw per-run finding counts and produces all statistics — the LLM must never compute mean/stdev/CI itself.

Input (via --input flag pointing to a JSON file, or stdin):

{
  "runs": [
    {"run_id": 1, "findings": {"critical": 3, "high": 12, "medium": 20, "low": 8}},
    {"run_id": 2, "findings": {"critical": 4, "high": 11, "medium": 22, "low": 9}}
  ],
  "baseline": {
    "mean_total": 45.0,
    "stdev_total": 2.1
  }
}

Logic (using only Python stdlib — statistics module):

• Compute total_findings per run = sum of all severity counts.
• Compute mean, stdev (population or sample, document which), and a 95% confidence interval using the t-distribution approximation from statistics.NormalDist (available since Python 3.8).
• Stability verdict (no baseline):
  • STABLE if stdev < NOISE_THRESHOLD (default 2.0, configurable via --noise-threshold).
  • UNSTABLE otherwise.
• Comparison verdict (baseline provided):
  • delta = mean - baseline.mean_total
  • NOISE if |delta| <= sigma_multiplier * baseline.stdev_total (default sigma_multiplier=1.0).
  • REGRESSION if delta > sigma_multiplier * baseline.stdev_total.
  • IMPROVEMENT if delta < -(sigma_multiplier * baseline.stdev_total).
• Output (stdout, JSON):

{
  "n_runs": 5,
  "mean_total": 44.2,
  "stdev_total": 1.8,
  "ci_95": [42.4, 46.0],
  "stability": "STABLE",
  "verdict": "NOISE",
  "delta": -0.8,
  "protocol": "score_audit_v1"
}

• Exit code 0 always (errors go to stderr, verdict "ERROR").

3. Extend plugins/shipguard/skills/sg-improve/SKILL.md

3a. Add Phase 0.5 — Prompt Hash Chaining (insert after Phase 0 Snapshot)

Add a new section that instructs the LLM to:

1. Identify the canonical input files for the hash: plugins/shipguard/skills/sg-code-audit/SKILL.md, plugins/shipguard/skills/sg-improve/SKILL.md, plugins/shipguard/skills/sg-scout/SKILL.md, and any zone definition files found in .code-audit-results/ or visual-tests/_results/.
2. Shell out:

   python plugins/shipguard/scripts/prompt_hash.py <file1> <file2> ...
   

3. Read the returned chain_hash.
4. Check .shipguard/learnings.yaml for a stored prompt_chain_hash key.
5. If stored hash differs from current hash: print a prominent warning — ⚠️ Prompt chain changed since last baseline. Cross-run comparison is invalid. Storing new hash; baseline stats reset. — then update the stored hash and clear eval_baseline from learnings.yaml.
6. If hashes match: proceed normally; baseline comparison is valid.
7. Store the new chain_hash in learnings.yaml under a top-level prompt_chain_hash key alongside a prompt_chain_hash_updated timestamp.

3b. Add Phase 2b — N-Run Stability Detection (insert after Phase 2 Extract Friction Signals)

Add a new section that instructs the LLM to:

1. Check whether audit-results.json contains a run_history array (see step 3c below) with ≥ 2 entries. If fewer than 2 runs exist, skip stability analysis and note: "Need ≥ 2 runs for stability verdict."
2. Build the input JSON for score_audit.py from the run_history entries (each entry has per-severity finding counts).
3. If learnings.yaml contains an eval_baseline block (mean + stdev from a prior stable period), include it in the input JSON.
4. Shell out:

   python plugins/shipguard/scripts/score_audit.py --input /tmp/sg_eval_input.json
   

5. Parse the JSON output. The LLM must not recompute or second-guess any number in the output.
6. Store the result in learnings.yaml under eval_stats (latest run's stats) and promote to eval_baseline when stability == "STABLE" and n_runs >= N_MIN (default 3).
7. Surface the verdict prominently in the Phase 6 summary:
   • ✅ STABLE — stdev {x}, verdict: IMPROVEMENT (delta: -{d} findings vs baseline)
   • ⚠️ UNSTABLE — stdev {x} exceeds threshold. Do not compare this run to baseline.
   • 🔴 REGRESSION — mean findings increased by {d} beyond 1σ of baseline.

3c. Extend audit-results.json schema to support run_history

In the sg-code-audit SKILL.md (Phase 6 — Aggregation), add an instruction to append the current run's finding counts to a run_history array in audit-results.json:

run_history:
  - run_id: <monotonic int>
    date: "<ISO 8601>"
    mode: "<mode>"
    chain_hash: "<hash from Phase 0.5 of sg-improve, if available>"
    findings:
      critical: <n>
      high: <n>
      medium: <n>
      low: <n>
      total: <n>

Keep the last 10 entries (prune oldest on write). This gives sg-improve the multi-run data it needs without requiring the user to manually re-run audits.

3d. Add /sg-eval as an invocation alias

In the sg-improve SKILL.md frontmatter description and Invocations table, add:

Command	Behavior
/sg-eval	Run only Phase 0.5 (hash check) + Phase 2b (stability verdict) — skip local learnings write and GitHub issue
/sg-eval --set-baseline	Mark current eval_stats as the new eval_baseline in learnings.yaml

3e. Extend learnings.yaml schema (v3)

Add the following top-level keys to the schema documented in sg-improve/SKILL.md:

schema_version: 3

prompt_chain_hash: "<sha256 hex>"
prompt_chain_hash_updated: "2026-04-16T00:00:00Z"

eval_baseline:
  mean_total: 44.2
  stdev_total: 1.8
  n_runs: 5
  established: "2026-04-16T00:00:00Z"
  chain_hash: "<must match prompt_chain_hash for comparison to be valid>"

eval_stats:
  # Output of the most recent score_audit.py run
  n_runs: 2
  mean_total: 41.0
  stdev_total: 1.2
  ci_95: [39.8, 42.2]
  stability: "STABLE"
  verdict: "IMPROVEMENT"
  delta: -3.2
  computed: "2026-04-16T00:00:00Z"

4. Update plugins/shipguard/skills/sg-improve/SKILL.md Final Checklist

Add the following checklist items:

• [ ] Prompt chain hashed via prompt_hash.py (Phase 0.5)
• [ ] Hash compared to stored hash — mismatch triggers baseline reset
• [ ] run_history from audit-results.json passed to score_audit.py (Phase 2b)
• [ ] LLM did NOT compute any statistic — all numbers come from score_audit.py output
• [ ] Stability verdict (STABLE/UNSTABLE) and comparison verdict (NOISE/REGRESSION/IMPROVEMENT) stored in learnings.yaml
• [ ] Verdict surfaced in Phase 6 summary

5. Update README.md

In the Self-Improving Engine section, add a row to the /sg-improve invocations table:

| /sg-eval | Hash-check prompt chain + compute stability verdict (NOISE/REGRESSION/IMPROVEMENT) |
| /sg-eval --set-baseline | Promote current stats to baseline for future comparisons |

And add a brief paragraph under the reinforcement loop explaining that score_audit.py handles all math and the LLM only interprets verdicts.

Import In IDE

🤖 Prompt for AI Agents

## Observations

The ShipGuard plugin is structured as a set of SKILL.md prompt files under `plugins/shipguard/skills/`. The `sg-improve` skill (`file:plugins/shipguard/skills/sg-improve/SKILL.md`) already has a rich 6-phase loop with structured YAML learnings, snapshot/rollback, and GitHub issue filing. The `sg-code-audit` skill (`file:plugins/shipguard/skills/sg-code-audit/SKILL.md`) reads `.shipguard/learnings.yaml` at startup. There are no Python scripts in the repo yet — all logic lives in LLM-interpreted SKILL.md prompts.

## Approach

The ticket asks for three additions: prompt hash chaining, deterministic stats delegation via `score_audit.py`, and N-run stability verdicts (NOISE/REGRESSION/IMPROVEMENT). Since the codebase is entirely prompt-driven SKILL.md files, the right approach is:
1. Add `prompt_hash.py` and `score_audit.py` as new Python helper scripts (the LLM shells out to them — consistent with how `sg-code-audit` already shells out to `python ast`, `tsc`, `go build`, etc.).
2. Extend `sg-improve/SKILL.md` with a new **Phase 0.5 — Prompt Hash** section and a new **Phase 2b — Stability Verdict** section.
3. Optionally surface this as a new `/sg-eval` invocation alias inside `sg-improve`.

```mermaid
sequenceDiagram
    participant User
    participant sg-improve
    participant prompt_hash.py
    participant score_audit.py
    participant learnings.yaml

    User->>sg-improve: /sg-improve (or /sg-eval)
    sg-improve->>prompt_hash.py: hash(SKILL.md files + zone defs + git SHA)
    prompt_hash.py-->>sg-improve: chain_hash
    sg-improve->>sg-improve: compare chain_hash vs stored hash in learnings.yaml
    alt hash mismatch
        sg-improve-->>User: WARN — prompt chain changed, baseline comparison rejected
    end
    sg-improve->>sg-improve: Phase 1–2: collect audit-results.json (N runs)
    sg-improve->>score_audit.py: pass raw finding counts per run
    score_audit.py-->>sg-improve: mean, stdev, CI, verdict (STABLE/NOISE/REGRESSION/IMPROVEMENT)
    sg-improve->>learnings.yaml: store chain_hash + stability verdict + stats
    sg-improve-->>User: Summary with verdict
```

---

## Implementation Steps

### 1. Create `plugins/shipguard/scripts/prompt_hash.py`

This script is shelled out to by `sg-improve` — the LLM never computes the hash itself.

- Accept a list of file paths as CLI arguments (or read from stdin as a newline-separated list).
- Read each file's content in sorted, deterministic order: the SKILL.md files for `sg-code-audit`, `sg-improve`, `sg-scout`; any zone definition files passed in; and the current `git rev-parse HEAD` SHA (obtained via `subprocess`).
- Concatenate all content with a stable delimiter, then produce a `sha256` hex digest using Python's built-in `hashlib`.
- Print a single JSON line: `{"chain_hash": "<hex>", "inputs": ["<file1>", ...], "git_sha": "<sha>"}`.
- Exit code 0 on success, 1 on any missing file.

### 2. Create `plugins/shipguard/scripts/score_audit.py`

This script receives raw per-run finding counts and produces all statistics — the LLM must never compute mean/stdev/CI itself.

**Input** (via `--input` flag pointing to a JSON file, or stdin):
```json
{
  "runs": [
    {"run_id": 1, "findings": {"critical": 3, "high": 12, "medium": 20, "low": 8}},
    {"run_id": 2, "findings": {"critical": 4, "high": 11, "medium": 22, "low": 9}}
  ],
  "baseline": {
    "mean_total": 45.0,
    "stdev_total": 2.1
  }
}
```

**Logic** (using only Python stdlib — `statistics` module):
- Compute `total_findings` per run = sum of all severity counts.
- Compute `mean`, `stdev` (population or sample, document which), and a 95% confidence interval using the t-distribution approximation from `statistics.NormalDist` (available since Python 3.8).
- **Stability verdict** (no baseline):
  - `STABLE` if `stdev < NOISE_THRESHOLD` (default `2.0`, configurable via `--noise-threshold`).
  - `UNSTABLE` otherwise.
- **Comparison verdict** (baseline provided):
  - `delta = mean - baseline.mean_total`
  - `NOISE` if `|delta| <= sigma_multiplier * baseline.stdev_total` (default `sigma_multiplier=1.0`).
  - `REGRESSION` if `delta > sigma_multiplier * baseline.stdev_total`.
  - `IMPROVEMENT` if `delta < -(sigma_multiplier * baseline.stdev_total)`.
- **Output** (stdout, JSON):
```json
{
  "n_runs": 5,
  "mean_total": 44.2,
  "stdev_total": 1.8,
  "ci_95": [42.4, 46.0],
  "stability": "STABLE",
  "verdict": "NOISE",
  "delta": -0.8,
  "protocol": "score_audit_v1"
}
```
- Exit code 0 always (errors go to stderr, verdict `"ERROR"`).

### 3. Extend `plugins/shipguard/skills/sg-improve/SKILL.md`

#### 3a. Add Phase 0.5 — Prompt Hash Chaining (insert after Phase 0 Snapshot)

Add a new section that instructs the LLM to:

1. Identify the canonical input files for the hash: `plugins/shipguard/skills/sg-code-audit/SKILL.md`, `plugins/shipguard/skills/sg-improve/SKILL.md`, `plugins/shipguard/skills/sg-scout/SKILL.md`, and any zone definition files found in `.code-audit-results/` or `visual-tests/_results/`.
2. Shell out:
   ```
   python plugins/shipguard/scripts/prompt_hash.py <file1> <file2> ...
   ```
3. Read the returned `chain_hash`.
4. Check `.shipguard/learnings.yaml` for a stored `prompt_chain_hash` key.
5. **If stored hash differs from current hash:** print a prominent warning — `⚠️ Prompt chain changed since last baseline. Cross-run comparison is invalid. Storing new hash; baseline stats reset.` — then update the stored hash and clear `eval_baseline` from `learnings.yaml`.
6. **If hashes match:** proceed normally; baseline comparison is valid.
7. Store the new `chain_hash` in `learnings.yaml` under a top-level `prompt_chain_hash` key alongside a `prompt_chain_hash_updated` timestamp.

#### 3b. Add Phase 2b — N-Run Stability Detection (insert after Phase 2 Extract Friction Signals)

Add a new section that instructs the LLM to:

1. Check whether `audit-results.json` contains a `run_history` array (see step 3c below) with ≥ 2 entries. If fewer than 2 runs exist, skip stability analysis and note: "Need ≥ 2 runs for stability verdict."
2. Build the input JSON for `score_audit.py` from the `run_history` entries (each entry has per-severity finding counts).
3. If `learnings.yaml` contains an `eval_baseline` block (mean + stdev from a prior stable period), include it in the input JSON.
4. Shell out:
   ```
   python plugins/shipguard/scripts/score_audit.py --input /tmp/sg_eval_input.json
   ```
5. Parse the JSON output. **The LLM must not recompute or second-guess any number in the output.**
6. Store the result in `learnings.yaml` under `eval_stats` (latest run's stats) and promote to `eval_baseline` when `stability == "STABLE"` and `n_runs >= N_MIN` (default 3).
7. Surface the verdict prominently in the Phase 6 summary:
   - `✅ STABLE — stdev {x}, verdict: IMPROVEMENT (delta: -{d} findings vs baseline)`
   - `⚠️ UNSTABLE — stdev {x} exceeds threshold. Do not compare this run to baseline.`
   - `🔴 REGRESSION — mean findings increased by {d} beyond 1σ of baseline.`

#### 3c. Extend `audit-results.json` schema to support `run_history`

In the `sg-code-audit` SKILL.md (Phase 6 — Aggregation), add an instruction to append the current run's finding counts to a `run_history` array in `audit-results.json`:

```
run_history:
  - run_id: <monotonic int>
    date: "<ISO 8601>"
    mode: "<mode>"
    chain_hash: "<hash from Phase 0.5 of sg-improve, if available>"
    findings:
      critical: <n>
      high: <n>
      medium: <n>
      low: <n>
      total: <n>
```

Keep the last **10 entries** (prune oldest on write). This gives `sg-improve` the multi-run data it needs without requiring the user to manually re-run audits.

#### 3d. Add `/sg-eval` as an invocation alias

In the `sg-improve` SKILL.md frontmatter `description` and `Invocations` table, add:

| Command | Behavior |
|---------|----------|
| `/sg-eval` | Run only Phase 0.5 (hash check) + Phase 2b (stability verdict) — skip local learnings write and GitHub issue |
| `/sg-eval --set-baseline` | Mark current `eval_stats` as the new `eval_baseline` in `learnings.yaml` |

#### 3e. Extend `learnings.yaml` schema (v3)

Add the following top-level keys to the schema documented in `sg-improve/SKILL.md`:

```yaml
schema_version: 3

prompt_chain_hash: "<sha256 hex>"
prompt_chain_hash_updated: "2026-04-16T00:00:00Z"

eval_baseline:
  mean_total: 44.2
  stdev_total: 1.8
  n_runs: 5
  established: "2026-04-16T00:00:00Z"
  chain_hash: "<must match prompt_chain_hash for comparison to be valid>"

eval_stats:
  # Output of the most recent score_audit.py run
  n_runs: 2
  mean_total: 41.0
  stdev_total: 1.2
  ci_95: [39.8, 42.2]
  stability: "STABLE"
  verdict: "IMPROVEMENT"
  delta: -3.2
  computed: "2026-04-16T00:00:00Z"
```

### 4. Update `plugins/shipguard/skills/sg-improve/SKILL.md` Final Checklist

Add the following checklist items:

- `[ ]` Prompt chain hashed via `prompt_hash.py` (Phase 0.5)
- `[ ]` Hash compared to stored hash — mismatch triggers baseline reset
- `[ ]` `run_history` from `audit-results.json` passed to `score_audit.py` (Phase 2b)
- `[ ]` LLM did NOT compute any statistic — all numbers come from `score_audit.py` output
- `[ ]` Stability verdict (STABLE/UNSTABLE) and comparison verdict (NOISE/REGRESSION/IMPROVEMENT) stored in `learnings.yaml`
- `[ ]` Verdict surfaced in Phase 6 summary

### 5. Update `README.md`

In the **Self-Improving Engine** section, add a row to the `/sg-improve` invocations table:

| `/sg-eval` | Hash-check prompt chain + compute stability verdict (NOISE/REGRESSION/IMPROVEMENT) |
| `/sg-eval --set-baseline` | Promote current stats to baseline for future comparisons |

And add a brief paragraph under the reinforcement loop explaining that `score_audit.py` handles all math and the LLM only interprets verdicts.

---

Execution Information

Branch: main
Commit: 725d47a

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.