Skip to content

v0.3.2 -- Backtest harness: charter empirical gate

Choose a tag to compare

@baladithyab baladithyab released this 14 May 02:56
· 880 commits to main since this release

v0.3.2 -- Backtest harness: the charter's empirical gate

The release the charter has been waiting for.

Charter: "if your three-analyst committee on BTC can't beat buy-and-hold
risk-adjusted on paper, more analysts won't fix it."

Until now that question required 4-8 weeks of wall-clock paper trading
to answer. v0.3.2 makes it computable in seconds via historical replay
through the production advisor pipeline.

hermes quant backtest --symbol BTC/USDT --asset-class crypto
--timeframe 1h --bars-file ~/.hermes/quant/cache/btc-1h.csv

Output: report.md + result.json + equity_curve.csv + decisions.jsonl
in ~/.hermes/quant/backtests//.

The line that matters: "Excess return vs buy-and-hold: +X.XX%"

Highlights

  • ADR-0020: REPLAY (not simulation). Uses production advisor pipeline
    verbatim. Charter Reproducibility honored: same bars + same config_hash
    -> same equity curve, byte-identical.
  • backtest.replay() walks bars chronologically through advisor.recommend()
    with lookahead-safe as_of cutoff at every bar.
  • PaperPortfolio: minimal mark-to-market book with slippage, commission,
    lot-matching realized P&L, position flips.
  • Buy-and-hold baseline + Sharpe + Deflated Sharpe (PSR for n_trials=1)
    • max drawdown computed inside replay().
  • BacktestResult.excess_return_vs_buy_hold_pct is THE charter-gating
    headline. Negative over multi-month backtest = "fix analysts before
    RL aggregator work" per charter.

Tests: 530 passed, 1 skipped (was 506 -> +24, zero regressions).

What this unblocks

v0.4+ RL aggregator work was charter-gated on "paper-trade for 4-8
weeks first." With replay, operators can now answer:

  • Does the three-analyst committee beat buy-and-hold on BTC/USDT 1h
    over 2024? Run a backtest. (Seconds.)
  • What's the Deflated Sharpe accounting for backtest-overfitting? In
    the report.
  • What's the per-bar equity curve? equity_curve.csv ready to plot.

The charter's empirical gate is now computable. v0.4's RL work can
proceed iff the v0.3.x committee actually beats buy-and-hold on
real data -- which is exactly what the charter requires.