Skip to content

0.2.0 — measured supervision

Choose a tag to compare

View all tags
@CallMeHFK CallMeHFK released this 22 Sep 06:48
· 33 commits to main since this release
v0.2.0
4b0362e

0.2.0 — measured supervision

0.1.0 shipped a supervisor that could notice drift. 0.2.0 ships one that can
prove what it noticed: a deterministic counterfactual benchmark, structured
outcome signals, risk-controlled admission, and a configuration surface you can
read without opening a file.

What is in here

Evidence-driven admission (Tier 0). Every tool result is classified into a
three-state outcome (FAILED / OK / UNKNOWN) instead of being pattern-matched
for the word "error", so a lost response stops being silently counted as success.
Line-anchored error syntax replaces the marker list; non-atomic tool failures are
detected.

Change-point drift detection. A CUSUM detector over the outcome stream, with a
Monte-Carlo-calibrated alarm threshold and a self-normalizing baseline taken from
the pre-alarm reference period only. It folds over tool results rather than all
events, so a session that goes quiet can still alarm.

Conformal risk control. nudge_threshold / block_threshold are admission
gates fitted from observed outcomes with a finite-sample (n+1)/n correction, not
a confidence number the judge self-reports. Priors apply until evidence
accumulates, and a key with too little evidence is dropped rather than pinned to a
fallback.

Binding-drift detection. Right tool, wrong entity (read detectors.py then
edit detectors.py.bak) — a class a loop detector cannot see because the
arguments genuinely differ, and an off-spec check cannot see because the path may
be in scope. Similarity is computed over the filename only; a pair differing in
test/source kind is not treated as confusable.

Anti-saturation. A per-detector nudge cooldown, a natural-checkpoint wake gate
that no longer spends the judge on clean boundaries, and wall-clock (not
event-count) context-rot detection, so streamed reasoning deltas are not mistaken
for spinning.

Learned guidance. A cross-session rulebook promotes guidance an agent
demonstrably followed into pre-drift context, ranked by adherence × evidence.

Counterfactual benchmark (shepherd eval). Scripted healthy sessions plus a
fault injector that breaks exactly one thing at a known step, with first-detection
attribution, per-detector precision/recall, detection delay, false-alarm rate and
cost. Deterministic, offline, runs in CI, and gates on F1.

Operations. shepherd start-bg|stop|status|risk|config|doctor|eval|stats, a
shipped systemd user unit that self-heals on SIGTERM, readiness-gated daemon
start, ${VAR} expansion in the config file, and config merge / config prune
for files written against an older release.

Measured, offline (seed 7, 8 sessions × 10 fault kinds)
overall recall / F1 0.78 / 0.88
detection delay (mean) 5.0 steps
false alarms per clean session 0.00
per-detector precision 1.00 for loop, regression, drift, contextrot, binding
cost 63.7 events and 7.1 judge wakes per session

Full machine-readable report attached (eval-v0.2.0.json). Reproduce with
shepherd eval --seed 7 --sessions 8 --steps 60.

What is not working, before you rely on it

  • Those numbers are synthetic. They come from a scripted counterfactual
    benchmark, not from production traffic. The live evidence in this release is
    narrower: one real mid-loop injection, observed end to end in a QwenPaw session
    (detected → parked → drained at the gate → delivered into the agent context as a
    user turn, and acted on).
  • offspec_edit has recall 0.00 and is marked unanswerable under the default
    config. Contract-scope nudging is off by default (scope_nudge_unobserved=false)
    because measuring it cost one false alarm per healthy session; the trade was made
    deliberately and the benchmark records it as a gap rather than a pass.
  • Admission thresholds ship as priors (0.60 / 0.85) with zero outcome labels on
    the releasing machine.     Getting [calibrated] requires a live-judge run
    (shepherd eval --judge --fit-risk …), which needs a reachable judge and
    patience — the full-corpus run exceeded 10 minutes previously.
  • block is never emitted unless an agent opts in (block_enabled). A
    supervisor that silently terminates agents is not the default.
  • Claude Code and Codex adapters are install-verified, not traffic-verified on
    a live session in this cycle; QwenPaw is the only adapter with an observed
    in-loop intervention. Codex additionally needs the hook trusted once in its TUI.
  • Tier 1 is a cost: 715k judge tokens across the 80 benchmark sessions. If you
    cannot pay for a judge, run Tier 0 only — the detectors need no network.

Upgrading

pip install --upgrade "git+https://github.com/CallMeHFK/agent-shepherd.git@v0.2.0"   # or: git pull && pip install -e .
shepherd config merge                  # add settings an older config.yaml predates
shepherd config prune                  # drop settings it no longer reads
shepherd doctor                        # what is configured, what is missing, what to do next
shepherd install qwenpaw               # re-copy the plugin bundle (restart QwenPaw after)
shepherd install claude && shepherd install codex

shepherd status prints code loaded from: … — check it first. Two installed
checkouts of this project resolve by working directory, and it has cost a debugging
session more than once.

Plugin bundles archived during install now land in ~/.qwenpaw/plugin-archive/,
outside the plugin scan path: QwenPaw loads every subdirectory of plugins/ and
each copy carries the same manifest id, so a backup left there is a second plugin,
not a backup.

Importing the plugin without a checkout

agent-shepherd.zip is the QwenPaw plugin bundle, named after the plugin id so
releases/latest/download/agent-shepherd.zip stays valid for every future
release — the version the host shows comes from the plugin.json inside, not
from the filename. It is the same tree shepherd install qwenpaw writes, and it
carries its own README.md. Hand the URL to the installer, or upload the file in
QwenPaw's plugin page:

qwenpaw plugin install \
  https://github.com/CallMeHFK/agent-shepherd/releases/download/v0.2.0/agent-shepherd.zip

Running app hot-loads it; a stopped app loads it on the next start.

Two ways this fails quietly:

  • Not the source archive. .../archive/refs/tags/v0.2.0.zip does contain a
    manifest, but at agent_shepherd/adapters/qwenpaw/plugin.json, while both
    installers look in the archive root or one top-level directory — so it is
    rejected. Use the release asset above.
  • A bundle is not a supervisor. The zip is only the in-process observer; it
    POSTs to 127.0.0.1:4890 and fails open when nothing listens there, so it
    loads, registers, and supervises nothing. The supervisor is a Python package,
    and this release carries it too — the wheel needs no git client:
pip install https://github.com/CallMeHFK/agent-shepherd/releases/download/v0.2.0/agent_shepherd-0.2.0-py3-none-any.whl
shepherd start-bg && shepherd status
Assets 6
Loading