v0.8.0 — Autonomous, receipt-bound maintenance

Vanguarstew now operates an autonomous, fail-closed maintenance pipeline within its published authority boundaries.

What is live

• Autonomous review, labeling, merging, and closing for eligible incoming contributions.
• New agent/ contributions execute in digest-pinned, non-root, read-only, no-network isolation.
• Model credentials remain outside candidate-controlled code.
• Baseline and candidate benchmark artifacts are checked before scoring.
• A fixed validator recomputes the final conservative decision inside a Polaris Intel TDX workload with egress=none.
• The result is accepted only after exact receipt-binding verification.
• Any isolation, evidence, billing, transport, Polaris, or receipt failure fails closed without a score-backed action.
• Benchmark and contribution-integrity changes remain manually controlled by design.

Trust boundary

This release provides execution-integrity evidence for the receipt-bound decision. Hosted model inference does not run inside the TEE, and the current verification level is polaris-verified, not independently hardware-verified. It does not claim workload confidentiality or GPU provenance.

Full reports and receipts remain private; public review output follows the repository's aggregate disclosure policy.

Delivered since v0.7.0

• Polaris receipt client and fail-closed verification.
• Deterministic replay and mounted-input binding.
• Receipt-bound benchmark decision recomputation.
• Isolated candidate execution and credential-safe model brokering.
• Public benchmark-change admission controls.
• Restrictive-environment fixes and successful end-to-end activation validation.

The release target passed the complete CI workflow on Python 3.10, 3.11, and 3.12.

Full history: v0.7.0...v0.8.0.

v0.6.0 — M6: live on gittensor

vanguarstew is now live and earning on the gittensor subnet. This release marks M6 complete and sets the North Star for what comes next.

Milestones

• M6 — gittensor integration ✅ — registered as a scored repository on the gittensor subnet via the trusted label pipeline; the measured perf:* bands map 1:1 to on-chain label multipliers.
• North Star + M7/M8 — the roadmap now defines the path to a public, verifiable maintainer-foresight metric and demonstration.

Contribution scoring & leaderboard

• Measured perf:xs–perf:xl bands (replacing the mult:* ladder) with the Pareto-floor regression block — a PR earns its label from a real benchmark delta, and a regression on either scored axis is a hard merge block.
• Public leaderboard feed with fixed-anchor since_anchor cumulative progress and an independent base-bar anchor score.

Agent

• Planner/decider improvements for objective-anchor scoring, commit-kind recall, and release-cadence gating.

Benchmark integrity

• A systematic hardening sweep: the benchmark gates and CLI loaders now fail closed rather than crash or silently pass on non-finite numbers, oversized integer literals, malformed per_repo rows, unreadable/directory artifact paths, and zero-task generalization partitions.

---

Full history: v0.5.0...v0.6.0.

v0.5.0 — M4 hardening + M5 anti-gaming scoring

402 commits since v0.3.0. Two milestones landed:

M4 — Hardening & release readiness

• Agent hardening: every field the LLM emits is guarded against non-string types (#297, #313, #317).
• Benchmark scoring: module-recall farming fixed (#289), backlog threshold reachable for single-word titles (#308), composite-score wiring (#341).
• Leakage lockout: tag-creation-date filter for frozen releases (#332), release-tag scrubbing (#330), forward-reference masking in the git-only fallback (#312).
• Tooling: compare_eval CLI for diffing replay artifacts (#306), --fail-under score floor for CI gating (#318, #367).
• M3 acceptance run completed clean: generalization_gap = 0.097, zero crashes across 5 repos.

M5 — Measured, anti-gaming contribution scoring

• scripts/score_pr_delta.py: diffs two run_eval artifacts (baseline vs. a PR's agent) and applies a Pareto floor — a PR's value label now requires a measured, non-regressing benchmark delta, not a maintainer's read of the diff (#1295).
• Merge-block + ceiling label: a measured regression is a hard merge block for agent/ PRs; a large, clean win on every axis unlocks a new ceiling label, mult:breakthrough (×3.0), above mult:core-correctness (×2.0) (#1302).
• REVIEW.md documents the full evidence requirement and tier ladder for agent/ PRs.
• Public CI smoke check (agent-benchmark-smoke.yml) on every agent/-touching PR — crash/shape only, explicitly not the scoring evidence itself.

Full test suite green (3675 passed). See ROADMAP.md for the complete milestone breakdown.

v0.3.0 — M3 generalization

M3 — generalization. vanguarstew now scores across multiple repos with an aggregated cross-repo composite, driven by a leakage-safe, versioned repo-set config. Held-out generalization scoring is the remaining M3 piece; the fully agentic loop is M4.

[0.3.0] - 2026-07-03

Added

• Generalization (M3): multi-repo replay — run_multi_replay / scripts/run_eval.py --repos
  runs several repos and averages each repo's own composite_mean into one cross-repo number
  (per-repo results retained; too-small repos skipped), so the agent is scored on breadth rather
  than a single tuned repo (#51).
• Generalization (M3): leakage-safe repo-set config + loader (benchmark/repo_set.py,
  benchmark/repo_sets/) — the replay repo list is a checked-in, strictly-validated JSON config
  (recent/obscure tier, held_out, freeze-window hints) instead of a hardcoded array, so the
  curated selection is reviewable and versioned (#55).
• Composite: file-weighted module recall now feeds the composite score — the objective anchor
  weights modules by how much of the revealed maintainer effort landed in each, so the blended
  score better reflects where the work actually concentrated (#91).
• Judge integrity: the pairwise judge now defends against LLM position bias with dual-order
  consistency — it asks both presentation orders and awards a win only if it survives the swap,
  otherwise a tie. A position-biased judge can no longer earn a spurious win, and per-task
  variance drops. Default on; opt out via run_replay(dual_order_judge=False) /
  --single-order-judge; the replay result reports judge_dual_order (#87).
• Planner queue reconciliation (agent/planner.py): a deterministic pass makes the plan honor
  the open-PR queue even when the LLM disregards it — an item that restates an open PR's work
  is down-weighted to a triage review item and flagged with restates_pr, redundant items
  targeting the same PR are collapsed, and if the plan ignores the queue entirely a review
  item for the top PR is prepended. Keeps the output coherent and de-duplicated regardless of
  model quality (#68).
• Development backend: tools/codex_llm.py, an optional agent.llm.LLM-compatible LLM backed
  by the local codex CLI (ChatGPT / OAuth, e.g. gpt-5.5), for running the benchmark and
  maintenance tooling without an API key. Dev/ops only — it is deliberately kept out of the
  scored agent.solve path, which still uses only validator-supplied inference per the
  managed-inference contract (agent/llm.py).
• Objective scoring: commit-kind recall (benchmark/score.py) — objective_score now
  reports kind_recall, actual_kinds, and matched_kinds, grading whether a plan
  anticipated the kind of maintainer work (feat/fix/docs/refactor/…/release) that the
  revealed window actually did, parsed deterministically from Conventional-Commit subjects
  (#41).
• M2: open-issue backlog recall in the objective anchor — when frozen open_issues are
  knowable at T, score whether the plan anticipated issues the revealed window actually
  addressed (title ↔ commit-subject overlap); git-only runs with an empty backlog degrade
  gracefully (#44).
• Maintainer-assist mode (agent/review.py, scripts/review_pr.py): the same agent the
  benchmark scores, applied to a live PR — it reads the PR and outputs a maintainer review
  (recommended action, best-fit mult:* value tier, scope/tests checks, concerns, advice).
  This is the "how it helps a maintainer" side: real triage/review assistance, not just scoring.
• Composite score: the pairwise judge (trajectory + decision process) and the objective anchor
  (module recall, release/bump correctness) are now blended into a single per-task and mean
  score in [0, 1], with tunable weights (--w-judge / --w-objective, default 0.6 / 0.4).
• Objective anchor: semver-aware release-bump scoring — when a genuine release appears in the
  revealed window, objective_score derives the actual bump level (major/minor/patch) from
  the semver delta between the frozen base version and the released version, and reports
  bump_actual / bump_match against the agent's predicted version_bump (tags with or
  without a leading v, and missing-patch/pre-release forms, all parse). The released version
  is read only from genuine release subjects, so a dependency bump can't skew the bump level.

Fixed

• Judge robustness (follow-up to #54): the offline substance heuristic keyed only on
  title/theme presence, so a plan stuffed with generic filler titles (misc, updates,
  various, …) could still out-rank a shorter, concrete one. Substance is now a weighted score
  — filler/blank items count for nothing, and each structured action field (kind, files,
  per-item rationale) beyond a real title adds weight — so length/filler never beats
  substance (#70).
• Judge robustness: the offline pairwise stand-in ranked submissions by raw plan length,
  so a plan padded with empty-of-substance items could beat a shorter, substantive one. It now
  ranks by the count of items that actually name something (non-empty title/theme), so
  length alone can't win over substance (#54).
• Objective anchor: release_signaled/release_predicted no longer fire on an incidental
  version mentioned mid-subject (e.g. chore(deps): bump lodash to v4.17.21, fix crash in v1.2.0 parser). Release detection now requires explicit release wording or a version-tag
  subject, so dependency bumps no longer inflate the release-prediction signal (#57).
• Leakage: frozen milestone state is now computed as-of-T from closed_at instead of copying
  the milestone's present-day state, so a milestone that existed at T but was closed after T
  is no longer leaked into the context as completed (#77).

v0.2.0

Second release of vanguarstew — the SN74 repo-maintainer agent and its GitHub-history replay benchmark. Everything below landed since v0.1.0.

Scoring & judging

• The pairwise judge now weighs the decision process (inferred maintainer philosophy + reasoning), not just trajectory match.
• Robust judge parsing, tolerant of truncated/verbose model output (found via live verification against a real model).
• Selectable reference baselines for the judge — a deterministic heuristic opponent alongside empty, via --baseline (thanks @real-venus).
• Maintainer-philosophy inference now uses few-shot examples for steadier, evidence-based output (thanks @real-venus).

Benchmark & leakage

• Leakage hardening: forward-reference scrubbing (issue/PR back-refs, GitHub links, SHAs — including release names) and recent-window + deterministic rotation for freeze-point selection.
• GitHub issue/PR context now paginates back to the freeze time T, with a truncation flag.

Project & CI

• Trustable contribution pipeline: a published review/scoring rubric (REVIEW.md), a PR-integrity check, CODEOWNERS routing, and a CI coverage floor.
• Auto-label workflow (organizational area/type labels) and CI split into focused lint / validate / test jobs.

Thanks to first-time contributors @real-venus and @galuis116.

v0.1.0

First tagged release of vanguarstew — a general repository-maintainer agent and its GitHub-history replay benchmark.

Highlights:

• Maintainer agent with a fixed solve() entrypoint (philosophy -> plan -> decide -> implement) and an OpenAI-compatible managed-inference client.
• Time-travel replay benchmark: freeze a repo at a point in time, generate tasks from history, and score plans with a pairwise judge.
• GitHub-API context enrichment reconstructing the maintainer's working surface knowable at time T.
• Deterministic objective scoring anchor (module recall + release match) alongside the judge.
• Open-source scaffolding: MIT license, contributing guide, CI across Python 3.10-3.12, issue/PR templates.