Skip to content

Lab and Validation

Savvas Leousis edited this page Sep 26, 2026 · 2 revisions

Lab and Validation

The lab decides whether a strategy deserves trust. Most backtests look good by luck; survival tests try to prove that.

flowchart LR
  H[Hypothesis<br/>and premortem] --> PF[Preflight<br/>data checks]
  PF --> SP{Split}
  SP -->|train 70%| TU[Tune params<br/>every trial recorded]
  TU --> FI[Fit]
  SP -->|embargo, then validation 30%| ST
  FI --> ST[Survival tests]
  ST --> V{All passed?}
  V -->|yes| P[pass<br/>optional register in shadow]
  V -->|no| F[fail]
Loading

Steps

  1. Record: --hypothesis (the edge, and who pays for it) and --premortem (how it will fail) are stored before tuning.
  2. Split: --train-ratio (default 0.7) of the window is train. --embargo-bars skips bars between train and validation so labels cannot leak.
  3. Tune: a tuner tries parameter sets on train and keeps the best score. Tuners: grid (default) or random. Objectives: sharpe (default), cagr, final_return. Trials run on all cores (--workers, [lab.parallel]). Every trial lands in lab_trials.
  4. Fit: ML strategies train on the train window only.
  5. Survival suite: each test returns pass or fail with metrics.
  6. Verdict: pass only if every test passed. A reproducibility manifest (data, params, versions, seeds) is saved.

Presets

Preset Tests
quick (default) oos, period_stability
standard quick + perturbation, walk_forward, deflated_sharpe, cost_stress
promotion oos, walk_forward, deflated_sharpe, pbo, mc_trades, cost_stress, plateau, cross_instrument, benchmark_relative, mcpt (200 permutations)

--register uses promotion unless you pass --preset or --tests.

Survival tests

All 18 tests, found automatically in lab/survival/. Pick any with --tests a,b,c.

Test Passes when (defaults)
oos Probabilistic Sharpe on validation >= 0.95, at least 20 trades, drawdown no worse than -30%.
period_stability Sharpe in 3 validation sub-windows never below 0 and not too spread out.
perturbation Equity curves stay correlated (>= 0.8) when prices get small noise.
drift Feature distributions on train and validation stay similar (PSI).
runs_test Wins and losses are not suspiciously clustered or alternating.
mcpt Real bars beat shuffled bars: p-value <= 0.05.
walk_forward Re-tuned on rolling windows: enough folds score above 0, and walk-forward efficiency >= 0.5.
walk_forward_mcpt The whole walk-forward process beats the same process on shuffled bars.
deflated_sharpe Deflated Sharpe >= 0.95, counting every trial of this strategy class.
pbo Probability of backtest overfitting <= 0.2.
mc_trades Resampled trade years: risk of ruin <= 10%, profit likely, return well above drawdown.
cost_stress The edge survives costs at 2x, and costs do not eat most of the Sharpe.
plateau Nudging each parameter a little keeps most of the score.
cross_instrument Works on most tickers alone, and no single ticker makes most of the profit.
benchmark_relative Information ratio and excess CAGR against the benchmark >= 0.
event_study Entries beat simply holding the same tickers (baseline drift).
vs_random The tuner finds clearly more in real bars than in noise shaped like them.
signal_ic Informational: reports IC, ICIR and quantile spreads; never fails.

Tune a test with --test-option test.key=value (for example --test-option pbo.max_pbo=0.1).

Examples

# Everyday check
uv run stonks lab run momentum --tickers AAPL.US,MSFT.US --start 2023-01-01 --end 2026-09-01

# Strict: the promotion suite, then register if it passes
uv run stonks lab run ewmac_trend --tickers SPY.US,GLD.US,BTC-USD.CC \
  --start 2018-01-01 --end 2026-09-01 --embargo-bars 21 \
  --hypothesis "trends persist as investors under-react" --register

# Pick tests by hand, with walk-forward and a permutation test
uv run stonks lab run donchian_breakout --params '{"ticker": "SPY.US"}' --tickers SPY.US \
  --start 2020-01-01 --end 2026-09-01 --tests oos,runs_test --walk-forward --mcpt

# Every catalogued strategy except the wrappers, on one basket, summary as CSV
uv run stonks lab sweep --tickers AAPL.US,MSFT.US --start 2023-01-01 --end 2026-09-01 --csv-out sweep.csv

Other flags: --interval 1h, --tuner random --budget 50, --grid-size 5, --seed, --objective cagr, --cost-model config|zero|realistic, --benchmark QQQ.US, --wf-splits, --wf-test-days, --wf-anchored/--wf-rolling, --wf-min-wfe, --wf-matrix, --mcpt-retune, --mcpt-permutations, --mcpt-max-p, --json-out.

Notes

  • --tickers defaults to [production].universe.
  • Costs come from [backtest.costs] (realistic by default). A zero-cost run logs a warning.
  • --register saves a passing strategy as shadow. It never becomes active on its own.
  • Strategies are named by catalog id, class name, or module:Class. See Strategies.

Clone this wiki locally