v0.3.0 — M3 generalization

M3 — generalization. vanguarstew now scores across multiple repos with an aggregated cross-repo composite, driven by a leakage-safe, versioned repo-set config. Held-out generalization scoring is the remaining M3 piece; the fully agentic loop is M4.

[0.3.0] - 2026-07-03

Added

• Generalization (M3): multi-repo replay — run_multi_replay / scripts/run_eval.py --repos
  runs several repos and averages each repo's own composite_mean into one cross-repo number
  (per-repo results retained; too-small repos skipped), so the agent is scored on breadth rather
  than a single tuned repo (#51).
• Generalization (M3): leakage-safe repo-set config + loader (benchmark/repo_set.py,
  benchmark/repo_sets/) — the replay repo list is a checked-in, strictly-validated JSON config
  (recent/obscure tier, held_out, freeze-window hints) instead of a hardcoded array, so the
  curated selection is reviewable and versioned (#55).
• Composite: file-weighted module recall now feeds the composite score — the objective anchor
  weights modules by how much of the revealed maintainer effort landed in each, so the blended
  score better reflects where the work actually concentrated (#91).
• Judge integrity: the pairwise judge now defends against LLM position bias with dual-order
  consistency — it asks both presentation orders and awards a win only if it survives the swap,
  otherwise a tie. A position-biased judge can no longer earn a spurious win, and per-task
  variance drops. Default on; opt out via run_replay(dual_order_judge=False) /
  --single-order-judge; the replay result reports judge_dual_order (#87).
• Planner queue reconciliation (agent/planner.py): a deterministic pass makes the plan honor
  the open-PR queue even when the LLM disregards it — an item that restates an open PR's work
  is down-weighted to a triage review item and flagged with restates_pr, redundant items
  targeting the same PR are collapsed, and if the plan ignores the queue entirely a review
  item for the top PR is prepended. Keeps the output coherent and de-duplicated regardless of
  model quality (#68).
• Development backend: tools/codex_llm.py, an optional agent.llm.LLM-compatible LLM backed
  by the local codex CLI (ChatGPT / OAuth, e.g. gpt-5.5), for running the benchmark and
  maintenance tooling without an API key. Dev/ops only — it is deliberately kept out of the
  scored agent.solve path, which still uses only validator-supplied inference per the
  managed-inference contract (agent/llm.py).
• Objective scoring: commit-kind recall (benchmark/score.py) — objective_score now
  reports kind_recall, actual_kinds, and matched_kinds, grading whether a plan
  anticipated the kind of maintainer work (feat/fix/docs/refactor/…/release) that the
  revealed window actually did, parsed deterministically from Conventional-Commit subjects
  (#41).
• M2: open-issue backlog recall in the objective anchor — when frozen open_issues are
  knowable at T, score whether the plan anticipated issues the revealed window actually
  addressed (title ↔ commit-subject overlap); git-only runs with an empty backlog degrade
  gracefully (#44).
• Maintainer-assist mode (agent/review.py, scripts/review_pr.py): the same agent the
  benchmark scores, applied to a live PR — it reads the PR and outputs a maintainer review
  (recommended action, best-fit mult:* value tier, scope/tests checks, concerns, advice).
  This is the "how it helps a maintainer" side: real triage/review assistance, not just scoring.
• Composite score: the pairwise judge (trajectory + decision process) and the objective anchor
  (module recall, release/bump correctness) are now blended into a single per-task and mean
  score in [0, 1], with tunable weights (--w-judge / --w-objective, default 0.6 / 0.4).
• Objective anchor: semver-aware release-bump scoring — when a genuine release appears in the
  revealed window, objective_score derives the actual bump level (major/minor/patch) from
  the semver delta between the frozen base version and the released version, and reports
  bump_actual / bump_match against the agent's predicted version_bump (tags with or
  without a leading v, and missing-patch/pre-release forms, all parse). The released version
  is read only from genuine release subjects, so a dependency bump can't skew the bump level.

Fixed

• Judge robustness (follow-up to #54): the offline substance heuristic keyed only on
  title/theme presence, so a plan stuffed with generic filler titles (misc, updates,
  various, …) could still out-rank a shorter, concrete one. Substance is now a weighted score
  — filler/blank items count for nothing, and each structured action field (kind, files,
  per-item rationale) beyond a real title adds weight — so length/filler never beats
  substance (#70).
• Judge robustness: the offline pairwise stand-in ranked submissions by raw plan length,
  so a plan padded with empty-of-substance items could beat a shorter, substantive one. It now
  ranks by the count of items that actually name something (non-empty title/theme), so
  length alone can't win over substance (#54).
• Objective anchor: release_signaled/release_predicted no longer fire on an incidental
  version mentioned mid-subject (e.g. chore(deps): bump lodash to v4.17.21, fix crash in v1.2.0 parser). Release detection now requires explicit release wording or a version-tag
  subject, so dependency bumps no longer inflate the release-prediction signal (#57).
• Leakage: frozen milestone state is now computed as-of-T from closed_at instead of copying
  the milestone's present-day state, so a milestone that existed at T but was closed after T
  is no longer leaked into the context as completed (#77).