A reproducible framework for benchmarking LLM native forecasting via knowledge cutoff and temporal masking

$$\Large \text{Forecasting} = \text{Gathering} \times \text{Synthesis} \times \text{Judgment} \times \text{Decision}$$

Traditional benchmarks ask: “Can you recall the answer?”
OracleProto asks: “Can you predict the future?”

May every forecast be reproducible, may AI truly become decision support
In service of every person’s judgments and choices for a good life

English | 中文文档 | Hugging Face

View Our Paper: arXiv

Visit Our Leaderboards: oracleproto.com

---

Overview

• Background & Challenges: Evaluating LLM forecasting faces a dilemma: live benchmarks expire easily, and retrospective benchmarks suffer from data leakage. Prompting cannot establish a genuine knowledge boundary.

• Architecture & Methods: The OracleProto framework combines model knowledge cutoffs and temporal masking to rigorously reconstruct historical events into reproducible, time-bounded forecasting samples.

• Experimental Results: Tests on six contemporary LLMs show that OracleProto effectively distinguishes models' forecasting quality, stability, and cost efficiency. It reduces the leakage rate to 1%, providing a controlled signal source for model comparison, supervised fine-tuning, and reinforcement learning.

Framework of OracleProto

Online Leaderboard Overview of OracleProto

---

1. Code map

forecast_eval/                       # core package
├─ runner.py                         # build_task_plan + scheduler
├─ react.py                          # ReAct loop + Tavily end_date injection
├─ leak_filter.py                    # retrieval-content auditor
├─ llm.py                            # OpenAI-compatible client; enforces no provider-side browsing
├─ search.py                         # Tavily wrapper
├─ analysis/                         # scoring and diagnostics: accuracy, FSS, BI, composite, behavior
├─ prompts.py / parser.py            # input renderer R / output parser Ψ
├─ types.py / errors.py / config.py  # data models / typed exceptions / Settings
├─ db.py / loader.py                 # SQLite schema migrations / dataset sync
└─ tavily_keys.py / tools.py         # API-key rotation / tool schemas
evaluation.py                        # entrypoint
scripts/                             # offline tooling
tests/                               # tests
runs/, logs/                         # run artefacts
forecast_eval_set_example.db         # bundled example dataset

---

2. Quickstart

2.1 Environment

Use uv :

uv sync
source .venv/bin/activate

or use Conda :

conda env create -f environment.yml
conda activate oracleproto

2.2 Configure .env

cp .env.example .env

Fill LLM_API_KEY, LLM_BASE_URL, MODELS, MODEL_TRAINING_CUTOFFS, TAVILY_API_KEY, LEAK_DETECTOR_API_KEY, LEAK_DETECTOR_BASE_URL, LEAK_DETECTOR_MODEL. The inline notes in .env.example cover the rest.

2.3 Tests

pytest tests/ -q

2.4 Run

python evaluation.py

Each invocation creates runs/{run_id}/ with run_id of the form YYYYMMDD-HHMMSS-{4-char hex}. Set RUN_ID=<existing-id> in .env to resume that run in place; completed or ineligible questions are skipped, and transient errors retry under the original backoff policy.

---

3. Bring your own dataset

The bundled forecast_eval_set_example.db carries 80 curated questions across three question types, with dates spanning 2026-03-12 to 2026-04-14. To plug in another corpus, swap SOURCE_DB and SOURCE_TABLE in .env.

---

4. Outputs

runs/{run_id}/
├─ manifest.json          # run-level metadata and hash chain
├─ db/{model_slug}.db     # one SQLite per model, independently replayable
├─ analysis/              # CSV/JSON regenerated from the raw DB
└─ logs/{run_id}.log

The DB stores raw observations only. Every aggregate ($\text{pass@1}$, FSS, BI, composite, …) is recomputed by forecast_eval/analysis/, which runs at the end of evaluation.py and can also be invoked standalone:

python -m forecast_eval.analysis runs/{run_id}

When the run is a virtual panel (manifest.is_virtual_panel = true), the analysis path additionally emits the panel-wide statistical envelope. See panels/README.md for the field-by-field schema and panels/example.json for a 2-model template.

---

5. Contact

For questions about code usage, dataset construction, or reproducing results, please reach out to the developers directly:

• Yiding Ma: yidingma@bupt.edu.cn
• Chengyun Ruan: ruanchengyun815@bupt.edu.cn

For joint research, dataset and benchmark co-development, or paper collaboration, please contact the principal investigators:

• Kaibo Huang (corresponding author): huangkaibo@bupt.edu.cn
• Zhongliang Yang (corresponding author): yangzl@bupt.edu.cn

---

6. Paper

View Our Paper: arXiv

---

7. Citation

If you use this project in your research, please cite our paper:

@article{OracleProto,
  title={OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking},
  author={Yiding Ma, Chengyun Ruan, Kaibo Huang, Zhongliang Yang, Linna Zhou},
  journal={arXiv preprint arXiv:2605.03762},
  year={2026}
}

---