New research — A Measurably Self-Improving Multi-Agent Forecasting System: Five Feedback Loops, No Retraining

[vybe]

🎥 Video walkthrough: https://youtu.be/_BFKrp9GnNQ
📖 Paper + data: https://github.com/Abilityai/trinity/tree/main/research/closed-loop-forecasting

A multi-agent forecasting system that has been running on Trinity since February 2026 and improves while it runs — no retraining, no fine-tuning. Seven specialized forecasting agents (the oracle fleet), a meta-analyst (Cleon) that measures quality and routes corrections, and a knowledge-base agent (Cornelius) that turns surprising failures into insights served back before future predictions.

The record so far:

• 96,161 logged predictions, 73,552 resolved outcomes
• Brier 0.194 vs 0.250 always-0.5 and 0.217 base-rate baselines; skill score +0.105
• 93.5–99.7% of prediction error is noise, not bias — calibration alone can't fix that
• 56,347 scored hidden-state hypotheses, with a monotone win-rate gradient across lifecycle tiers (strong 75.0%, emerging 68.2%, fading 59.0%)
• Three of five compared agents improved both Brier score and surprise rate; two moved against their feedback — a limitation reported, not hidden

The five feedback loops:

1. Noise-adaptive calibration — routes to process-consistency guidance when noise dominates
2. Surprise-driven insight extraction — confident misses become knowledge
3. Knowledge retrieval before prediction
4. Hidden-state hypothesis injection — theses about unobserved world state, scored by the predictions that use them
5. A generative loop that decides what is worth forecasting — abducing theses from cross-domain anomalies and commissioning falsifiable predictions

Live dashboard: https://brierhq.ai — predictions, resolutions, and Brier scores published as they land.

Built and running entirely on Trinity — free and open source (Apache 2.0). Questions and critique welcome — the split effect on the corrective loops in particular is an open discussion.