eval-watch

Discipline layer over the agent stack. Wraps each sibling's existing eval via subprocess, tracks regression and drift in SQLite, commits a STATUS.md report back to the repo on a monthly GitHub Actions cron.

Metric	V1.1
Per-run cost	~$2.80 (8-golden ARA subset + 2 OCC sequences; SM + MPA pytest are $0)
Wall-clock per run	~20 min (sequential composition, Tier 1 safe)
Operational cost	~$35/year (monthly cadence)
License	MIT

What it does

On a monthly GitHub Actions cron (or workflow_dispatch manual trigger), eval-watch:

1. Subprocess-invokes ARA's evals/eval_runner.py against an 8-golden subset
2. Subprocess-invokes OCC's evals/eval_runner.py (4 governance goldens — 2 refusals at $0, 2 live sequences)
3. Subprocess-invokes SM's pytest evals/replay/
4. Subprocess-invokes MPA's pytest evals/
5. Aggregates pass/fail + cost + wall-clock per sibling into SQLite
6. Compares to an anchored baseline — flags any previously-passing golden that now fails
7. Renders unicode-block histograms of per-dimension scores over the last 6 runs
8. Writes STATUS.md to repo root + commits + pushes via the workflow's bot identity

The discipline is auditable from outside the repo: GH Actions runs are visible at /actions; STATUS.md is rendered by GitHub at the front door.

Sample STATUS

The current latest is always at STATUS.md. Historical snapshots live under reports/. One annotated excerpt:

# eval-watch status

> ✓ All siblings green · $2.560 total · 14m wall-clock
> Last updated: 2026-06-10 09:01 UTC · Triggered by: cron · Run: 2026-06-10T09-00-00Z

## Per-sibling

| Sibling | Goldens | Cost | Wall-clock | Notes |
|---|---|---|---|---|
| ARA | 8/8 ✓ | $2.560 | 13m | python evals/eval_runner.py --domain <golden> × 8 |
| SM  | 5/5 ✓ | $0   | 610ms | pytest evals/replay/ |
| MPA | 22/22 ✓ | $0 | 720ms | pytest evals/ |

## Regression gate

Baseline: 2026-06-09T22-55-45Z
✓ No regressions vs baseline.

## Score histograms (last 6 runs, oldest → newest)

### ARA

| Dimension | Median trend | Current | Baseline | Drift |
|---|---|---|---|---|
| fit_score              | ▁▃▅▅█▇▇ | 4.10 | 4.00 | ↑2.5% |
| icp_confidence         | ▆▇▇█▇▇▇ | 0.84 | 0.82 | ↑2.4% |
| revops_maturity        | ▄▄▅▅▅▅▅ | 3.50 | 3.50 | —    |
| critic_score           | ▆▆▇▆▇▇▇ | 4.20 | 4.10 | ↑2.4% |

The agent stack

Repo	Answers	Framework	Status
account-research-agent	Who fits	LangGraph	v1.1.0
signal-monitor	When to act	LangGraph	v1.1.0
meeting-prep-agent	What to say	LangGraph	v1.0
outbound-campaign-crew	What to send	CrewAI	v1.0
eval-watch (this repo)	How do you know your agents — all of them — are right?	—	v1.1

V1.1 added OCC — the first non-LangGraph sibling. The adapter contract turned out to be framework-agnostic by construction: anything with a subprocess-runnable eval that exits non-zero on failure plugs in, whatever orchestrates it underneath.

eval-watch is the sixth instance of the code-enforced-rule pattern across this portfolio (OCC's Flow gates + task guardrails are the seventh):

1. Disqualifier clamp (ARA)
2. Length compliance (sister project — the blog autopilot)
3. Three clamps (SM): evidence_traceable / spend_cap / identity_key dedup
4. Concurrency clamp in SM's runtime
5. Four clamps + sequential composition (MPA)
6. Eval discipline enforced in CI (this repo)
7. Flow gates + task guardrails, same rules in CrewAI primitives (OCC)

Not a one-shot. A way of thinking that shows up across surfaces — and across frameworks.

Eight-golden ARA subset

Covers the four diagnostic classes that matter, ~60% cost reduction vs full 20-golden eval (subset corrected 2026-06-11 — see CLAUDE.md):

Golden	Class	Drift to catch
retool.com	Buyer-class mid fit (only validated buyer-class pass)	Regression on the genuine buyer case
stripe.com	Stage disqualifier honored (Series D+ at scale)	Stage discipline on well-known companies
vercel.com	Disqualifier override	The architectural differentiator (Series F overridden)
gymshark.com	Sector disqualifier honored (D2C non-buyer)	False fits creeping in
clay.com	Peer-gtm-vendor disqualified	Vendor/buyer confusion
apollo.io	Peer-gtm-vendor disqualified	Same
deloitte.com	Hard disqualifier (consultancy)	Baseline for "honored cleanly"
mckinsey.com	Hard disqualifier (consultancy)	Redundancy w/ Deloitte

Setup

Requires Python 3.11+. eval-watch invokes each sibling via subprocess — all four sibling repos must be cloned and configured.

# 1. Clone all five repos
git clone https://github.com/aehirota/account-research-agent
git clone https://github.com/aehirota/signal-monitor
git clone https://github.com/aehirota/meeting-prep-agent
git clone https://github.com/aehirota/outbound-campaign-crew
git clone https://github.com/aehirota/eval-watch
cd eval-watch

# 2. Set up the venv
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 3. Configure .env (sibling paths + API keys passed through to ARA/OCC)
cp .env.example .env
# Edit .env — set ARA_REPO_PATH, SM_REPO_PATH, MPA_REPO_PATH, OCC_REPO_PATH
# ANTHROPIC_API_KEY is needed for ARA + OCC; FIRECRAWL_API_KEY for ARA only

# 4. Smoke test the cheap path first (SM + MPA only — offline pytest, $0)
python run.py --sibling sm    # → 5/5 passed, $0
python run.py --sibling mpa   # → 22/22 passed, $0

# 5. Anchor a baseline for the regression gate
python run.py --list-runs
python run.py --set-baseline <run_id>

Usage

# Full run (ARA + OCC + SM + MPA; takes ~20 min, ~$2.80)
python run.py

# Single sibling
python run.py --sibling ara

# CI mode (workflow_dispatch + cron use this)
python run.py --triggered-by ci

# Baseline management
python run.py --set-baseline <run_id>
python run.py --get-baseline
python run.py --list-runs

V1 ship gates (all five pass at v1.0)

1. ✓ Day 1 end-to-end smoke succeeded (SM 5/5 + MPA 22/22)
2. ✓ Day 2 checkpoint criteria all green (report renders, regression gate works against baseline)
3. ✓ First live STATUS.md generated and committed (Day 3)
4. ✓ README + KNOWN_ISSUES + docs/architecture.md + tool page live (Day 3)
5. ✓ GH Actions workflow runs successfully on workflow_dispatch (Day 3 manual verification)

See also

• STATUS.md — the latest live status (updated monthly + on workflow_dispatch)
• docs/architecture.md — state machine, adapter pattern, cadence reasoning
• KNOWN_ISSUES.md — honest curation disclosure (V1 limits, deferred features)
• The pre-outbound stack tool pages on gtmsystemslab.com — the four tools in context

License

MIT. See LICENSE.