Futuresim is a forecasting simulator for LLM agents. It advances a dated question market, exposes only the information available at each simulated date, records agent forecasts, and scores them over time.

For background, see the Futuresim blogpost and paper.

git clone https://github.com/OpenForecaster/futuresim.git
cd futuresim

uv sync
source .venv/bin/activate

cp .env.example .env
# Edit .env if you need OpenRouter keys, custom output paths, or local artifacts.

python scripts/run_forecast_sim.py --config configs/shared/default_sim.yaml

The default search-enabled configs use LanceDB and require FSIM_SEARCH_DB. For a no-retrieval smoke run, use:

python scripts/run_forecast_sim.py --config configs/shared/default_nosearch_sim.yaml

scripts/run_forecast_sim.py loads .env from the repo root. Shell exports override .env values. ${FSIM_REPO_DIR} expands to this checkout.

Common variables:

One-off override example:

FSIM_OUTPUT_BASE=/scratch/$USER/futuresim-runs \
python scripts/run_forecast_sim.py --config configs/shared/default_sim.yaml

OpenForesight questions load from Hugging Face by default. The default config uses the aljazeera2026Q1 split.

Futuresim separates the question market from agent retrieval:

• The environment owns dates, visible questions, visible article files, forecast ingestion, answer matching, and scoring.
• Agents own their retrieval strategy. They can use the filesystem article corpus, the bundled LanceDB hybrid search tool, or a custom tool.

Download the prebuilt LanceDB artifact for the bundled hybrid search configs:

export FSIM_SEARCH_DB=${FSIM_SEARCH_DB:-$(pwd)/artifacts/forecast-news-embeddings}

hf download shash42/forecast-news-embeddings \
  --repo-type dataset \
  --local-dir "$FSIM_SEARCH_DB" \
  --max-workers 8

python scripts/check_search_readiness.py --db-path "$FSIM_SEARCH_DB"

The public embedding model used with this index is Qwen/Qwen3-Embedding-8B. Set FSIM_EMBEDDING_MODEL to a local checkout, a model id, or an embedding server target supported by your search backend.

Download the browsable article corpus separately:

export FSIM_ARTICLES_BASE=${FSIM_ARTICLES_BASE:-$(pwd)/artifacts/forecast-news}

hf download shash42/forecast-news \
  --repo-type dataset \
  --local-dir "$FSIM_ARTICLES_BASE" \
  --include '2025/12/**' \
  --include '2026/**' \
  --max-workers 8

FSIM_ARTICLES_BASE must point to a dated tree: YYYY/MM/DD/articles.jsonl. Rows should include title, source, date, and content; date_publish, url, id, and date_modify are optional.

Use --dataset custom --dataset_path <file-or-dir> with CSV, JSONL, JSON, or Parquet. A directory may contain split files such as test.jsonl, test.parquet, or test-*.parquet.

Required columns:

Optional columns: background, resolution_criteria, answer_type, options, source_split, and prompt.

Example:

python scripts/run_forecast_sim.py \
  --dataset custom \
  --dataset_path /path/to/questions.jsonl \
  --split test

To use a custom search backend, implement the BaseSearchTool contract in agents/search_tools/base.py. For LanceDB, semantic/hybrid search needs an articles table with chunk ids, article ids, date fields, content, optional metadata, and vectors built with the configured embedding model.

Futuresim includes adapters for Prime Intellect Verifiers and OpenReward/ORS. They use the same SimulationEnvironment and run a MinimalHarness-compatible CLI agent through the packaged MCP server:

python -m futuresim_agents.minimalHarnessAgent.mcp_server

Important defaults:

• The filesystem article corpus is the default information source.
• Hybrid LanceDB search is opt-in via futuresim.enable_hybrid_search: true.
• Hosted runs only accept forecasts submitted through MCP submit_forecasts and finalized with next_day.
• Sandboxes block general internet by default to avoid future leakage.
• Codex/Claude CLI reproductions require each user to provide their own private CLI/provider credentials through platform secrets or an equivalent private setup.

See integrations/README.md for sandbox image requirements, credential handling, network/egress guidance, and publication steps for Verifiers and OpenReward.

# Default shared simulation
python scripts/run_forecast_sim.py --config configs/shared/default_sim.yaml

# No-retrieval variant
python scripts/run_forecast_sim.py --config configs/shared/default_nosearch_sim.yaml

# Resume from the last day in a run directory
python scripts/run_forecast_sim.py --resume /path/to/output_dir

# Restart from a specific day while preserving prior forecasts
python scripts/run_forecast_sim.py \
  --restart_from /path/to/original/run \
  --restart_from_day 2025-04-05

Scaffold selection is explicit in config under defaults.scaffold:

• basic, allQ, allqd: base chat-tools scaffolds.
• qwenbasic, qwenallq: Qwen-named compatibility wrappers.
• minimalHarness: external CLI backends such as Codex, Claude Code, and OpenCode.

Runs are saved to FSIM_OUTPUT_BASE/<sim_name>/<timestamp>/.

Key files:

If FSIM_SIM_MATCHER_CACHE_DIR is set, split: "test" runs reuse <cache_dir>/<matcher_slug>.json and merge new entries back when the run exits. Other splits can opt in with top-level YAML: matcher_cache: {enabled: true, path: null}.

• timegap_days changes the simulator from daily wakeups to one session every N days. Metrics for active questions are evaluated through the end of that wakeup interval.
• OpenForesight configs can prepend train-split questions with prepend_train_resolution_start, prepend_train_resolution_end, and subsample_per_month.
• Each OpenForesight question carries a source_split tag so split-specific metrics can be logged without a separate loader path.

• agents/search_tools/README.md: search tool contract.
• agents/allQAgent/README.md: AllQ scaffold notes.
• agents/minimalHarnessAgent/README.md: external CLI harness notes.
• integrations/README.md: Verifiers and OpenReward integration details.