Repository navigation
Lab and Validation
Savvas Leousis edited this page Sep 26, 2026
·
2 revisions
The lab decides whether a strategy deserves trust. Most backtests look good by luck; survival tests try to prove that.
flowchart LR
H[Hypothesis<br/>and premortem] --> PF[Preflight<br/>data checks]
PF --> SP{Split}
SP -->|train 70%| TU[Tune params<br/>every trial recorded]
TU --> FI[Fit]
SP -->|embargo, then validation 30%| ST
FI --> ST[Survival tests]
ST --> V{All passed?}
V -->|yes| P[pass<br/>optional register in shadow]
V -->|no| F[fail]
-
Record:
--hypothesis(the edge, and who pays for it) and--premortem(how it will fail) are stored before tuning. -
Split:
--train-ratio(default 0.7) of the window is train.--embargo-barsskips bars between train and validation so labels cannot leak. -
Tune: a tuner tries parameter sets on train and keeps the best score. Tuners:
grid(default) orrandom. Objectives:sharpe(default),cagr,final_return. Trials run on all cores (--workers,[lab.parallel]). Every trial lands inlab_trials. - Fit: ML strategies train on the train window only.
- Survival suite: each test returns pass or fail with metrics.
-
Verdict:
passonly if every test passed. A reproducibility manifest (data, params, versions, seeds) is saved.
| Preset | Tests |
|---|---|
quick (default) |
oos, period_stability
|
standard |
quick + perturbation, walk_forward, deflated_sharpe, cost_stress
|
promotion |
oos, walk_forward, deflated_sharpe, pbo, mc_trades, cost_stress, plateau, cross_instrument, benchmark_relative, mcpt (200 permutations) |
--register uses promotion unless you pass --preset or --tests.
All 18 tests, found automatically in lab/survival/. Pick any with --tests a,b,c.
| Test | Passes when (defaults) |
|---|---|
oos |
Probabilistic Sharpe on validation >= 0.95, at least 20 trades, drawdown no worse than -30%. |
period_stability |
Sharpe in 3 validation sub-windows never below 0 and not too spread out. |
perturbation |
Equity curves stay correlated (>= 0.8) when prices get small noise. |
drift |
Feature distributions on train and validation stay similar (PSI). |
runs_test |
Wins and losses are not suspiciously clustered or alternating. |
mcpt |
Real bars beat shuffled bars: p-value <= 0.05. |
walk_forward |
Re-tuned on rolling windows: enough folds score above 0, and walk-forward efficiency >= 0.5. |
walk_forward_mcpt |
The whole walk-forward process beats the same process on shuffled bars. |
deflated_sharpe |
Deflated Sharpe >= 0.95, counting every trial of this strategy class. |
pbo |
Probability of backtest overfitting <= 0.2. |
mc_trades |
Resampled trade years: risk of ruin <= 10%, profit likely, return well above drawdown. |
cost_stress |
The edge survives costs at 2x, and costs do not eat most of the Sharpe. |
plateau |
Nudging each parameter a little keeps most of the score. |
cross_instrument |
Works on most tickers alone, and no single ticker makes most of the profit. |
benchmark_relative |
Information ratio and excess CAGR against the benchmark >= 0. |
event_study |
Entries beat simply holding the same tickers (baseline drift). |
vs_random |
The tuner finds clearly more in real bars than in noise shaped like them. |
signal_ic |
Informational: reports IC, ICIR and quantile spreads; never fails. |
Tune a test with --test-option test.key=value (for example --test-option pbo.max_pbo=0.1).
# Everyday check
uv run stonks lab run momentum --tickers AAPL.US,MSFT.US --start 2023-01-01 --end 2026-09-01
# Strict: the promotion suite, then register if it passes
uv run stonks lab run ewmac_trend --tickers SPY.US,GLD.US,BTC-USD.CC \
--start 2018-01-01 --end 2026-09-01 --embargo-bars 21 \
--hypothesis "trends persist as investors under-react" --register
# Pick tests by hand, with walk-forward and a permutation test
uv run stonks lab run donchian_breakout --params '{"ticker": "SPY.US"}' --tickers SPY.US \
--start 2020-01-01 --end 2026-09-01 --tests oos,runs_test --walk-forward --mcpt
# Every catalogued strategy except the wrappers, on one basket, summary as CSV
uv run stonks lab sweep --tickers AAPL.US,MSFT.US --start 2023-01-01 --end 2026-09-01 --csv-out sweep.csvOther flags: --interval 1h, --tuner random --budget 50, --grid-size 5, --seed, --objective cagr, --cost-model config|zero|realistic, --benchmark QQQ.US, --wf-splits, --wf-test-days, --wf-anchored/--wf-rolling, --wf-min-wfe, --wf-matrix, --mcpt-retune, --mcpt-permutations, --mcpt-max-p, --json-out.
-
--tickersdefaults to[production].universe. - Costs come from
[backtest.costs](realistic by default). A zero-cost run logs a warning. -
--registersaves a passing strategy asshadow. It never becomesactiveon its own. - Strategies are named by catalog id, class name, or
module:Class. See Strategies.
Source: sleousis/Stonks. Plans and status: docs/roadmap.md.
Start
Concepts
- Architecture
- Data and Storage
- Strategies
- Book Strategies
- Lab and Validation
- Backtesting
- Accounts and Modes
Using Stonks
- Strategy Studio
- Trading Tick
- Risk and Go-Live
- Operations
- Scheduling and Backups
- Deploy
- API and MCP
- Web Console
Reference