Smythe v0.4.0
Evidence. This release is what the benchmark campaign found and fixed — every number, loss, and null result is published in benchmarks/. No breaking API changes.
The story
We benchmarked dynamic topology against a single agent and a hardcoded pipeline (3 runs per cell, blind cross-model judge, raw records committed). It lost — badly. The losses were mechanistically diagnosed, fixed, and each fix measured:
| Dynamic-topology quality | before | payload → roots | payload → all nodes | + right-sizing |
|---|---|---|---|---|
| code-review | 1.7 | 4.3 | 9.0 | 8.3 (−45% cost) |
| acquisition-diligence | 2.0 | 8.3 | 9.0 | 8.7 |
| mean (5 tasks) | 4.7 | 6.3 | 8.2 | 8.0 (−14% cost) |
Dynamic topology now sits near parity with a strong fixed pipeline (8.0 vs 8.6) — the headline claim isn't proven yet, and the write-up says so.
Fixed
- Every node receives the task payload — generated node labels never carried the task's source material, so specialists worked blind and verifiers hedged claims about artifacts they couldn't see
- Terminal nodes are told their output is the deliverable — final nodes referenced upstream findings instead of reproducing them
- The Architect right-sizes graphs — cost-aware planning guidance; single-artifact tasks dropped from 5-node fork-joins to 3-node serial-adversarial graphs
Added
- Benchmark harness:
--runs Nwith mean/range reporting, an independent judge model, ablation flags, source documents for the diligence task - Memory A/B benchmark with observable recall wiring — first memory-on/off numbers (null on a homogeneous family, and the write-up explains why)
- Full v2→v5 records and judge reasoning committed
Full changelog: CHANGELOG.md
pip install --upgrade smythe