ReasonCast

Official code for ReasonCast, a recipe that trains a single language model to forecast a univariate time series and justify the forecast with a structured, verifiable reasoning chain, and ReasonTS-Bench, the benchmark it is trained and evaluated on.

Every model output couples a prediction with an explanation drawn from the same sequence, and four metric families score them jointly:

• Error — forecast accuracy (MAE / MASE).
• Fidelity — whether the stated reasoning fields match the closed-form truth.
• Consistency — whether the forecast obeys the model's own stated rule (needs no ground truth).
• Sensitivity — whether the reasoning tracks a single intervened generative parameter (counterfactual probe).

ReasonTS-Bench is built from five generative primitives (periodicity, trend, temporal dependence, multi-periodicity, structural break), plus an Unknown no-pattern set and an OOD-novel held-out set. Each sample carries an algorithmically derived ground-truth reasoning chain.

This repository contains the main-experiment code only. Ablations, extended analyses, and the multi-node dispatch/fleet tooling are not included. The generated datasets are not shipped; regenerate them with the benchmark scripts below.

Repository layout

reasoncast/
  benchmark/   generate the ReasonTS-Bench splits (no data files shipped)
  train/       ReasonCast supervised fine-tuning
  eval/        forecast + reasoning metrics, and the counterfactual probe
  baselines/   trivial and deep time-series forecasting baselines
docs/
  BENCHMARK_SPEC.md   full benchmark specification

Installation

pip install -r requirements.txt
# run everything as a package from the repo root:
export PYTHONPATH=$PWD

1. Build the benchmark

No .jsonl files are shipped. Each primitive, the no-pattern set, the OOD set, and the counterfactual pairs have their own generator:

# one primitive (train split); repeat for val/test with --seed and --n-samples
python -m reasoncast.benchmark.gen_L1_sine       --n-samples 12000 --seed 0 --out-jsonl data/L1_sine_train.jsonl
python -m reasoncast.benchmark.gen_L2_sine_trend --n-samples 12000 --seed 0 --out-jsonl data/L2_sine_trend_train.jsonl
python -m reasoncast.benchmark.gen_L3_ar1        --n-samples 12000 --seed 0 --out-jsonl data/L3_ar1_train.jsonl
python -m reasoncast.benchmark.gen_L4_multifreq  --n-samples 12000 --seed 0 --out-jsonl data/L4_multifreq_train.jsonl
python -m reasoncast.benchmark.gen_L5_changepoint --n-samples 12000 --seed 0 --out-jsonl data/L5_changepoint_train.jsonl

# no-pattern (Unknown) and held-out (OOD-novel) sets
python -m reasoncast.benchmark.gen_L0_unknown    --out-jsonl data/L0_unknown_train.jsonl
python -m reasoncast.benchmark.gen_OOD_novel     --out-jsonl data/OOD_novel_test.jsonl

# counterfactual pairs for the Sensitivity probe
python -m reasoncast.benchmark.gen_counterfactual_multi --out-jsonl data/L2_sine_trend_test_cf.jsonl

See docs/BENCHMARK_SPEC.md for the generative equations, parameter ranges, and reasoning schema of every pattern.

2. Train ReasonCast

python -m reasoncast.train.train_reasoning_sft \
    --model-name Qwen/Qwen2.5-3B-Instruct \
    --train-jsonl data/L1_sine_train.jsonl \
    --val-jsonl   data/L1_sine_val.jsonl \
    --out-dir     outputs/reasoncast_3b \
    --epochs 2 --batch-size 4 --lr 5e-5

Train on the union of all five primitives (plus the Unknown set) for the routed single-model configuration; train on a single primitive for a per-pattern specialist.

3. Evaluate

# forecast + Fidelity + Consistency
python -m reasoncast.eval.eval_reasoning \
    --model-dir outputs/reasoncast_3b --test-jsonl data/L1_sine_test.jsonl \
    --out-json outputs/reasoncast_3b/eval_L1_sine.json

# Sensitivity (counterfactual probe)
python -m reasoncast.eval.eval_counterfactual \
    --model-dir outputs/reasoncast_3b --cf-jsonl data/L1_sine_test_cf.jsonl \
    --out-json outputs/reasoncast_3b/cf_L1_sine.json --n-pairs 100

reasoncast.eval.eval_incontext runs the few-shot (in-context) LLM baseline with the same output format, and reasoncast.eval.scoring_per_level holds the per-field matching and metric definitions.

4. Baselines

# trivial predictors (naive-last, seasonal-naive, linear extrapolation)
python -m reasoncast.baselines.trivial \
    --test-jsonl data/L1_sine_test.jsonl --out-json outputs/trivial_L1.json

# deep time-series models (dlinear, patchtst)
python -m reasoncast.baselines.deep_ts --model dlinear \
    --train-jsonl data/L1_sine_train.jsonl --val-jsonl data/L1_sine_val.jsonl \
    --test-jsonl data/L1_sine_test.jsonl --out-json outputs/dlinear_L1.json

The iTransformer and TimeXer baselines in the paper are run through the external Time-Series-Library and are not vendored here.

Notes

• The instruction-tuned LLM backbones (Qwen, Llama, Phi, Gemma) are downloaded from Hugging Face by --model-name; nothing is bundled.
• Reasoning metrics apply only to models that emit the structured schema; the deep time-series baselines produce forecasts only.