Repository navigation
Backtesting with NautilusTrader and ClickHouse — Without Parquet #5285
Replies: 1 comment
|
Thanks for sharing your testing results, @nicezic. Do note the open-source scope in the roadmap: any ClickHouse integration remains out of scope for implementation or discussion here. I'm closing this discussion to keep that boundary clear. Further correctness or performance reports for the core engine remain welcome. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi everyone! 👋
I use NautilusTrader for backtesting, but I keep my market data in ClickHouse instead of Parquet. I wanted to share how I connected the two.
The idea is simple: prepare the data when it's collected, then use NautilusTrader's existing backtest engine.
How it works
flowchart TB subgraph CUSTOM["My ClickHouse path"] direction TB WS(["Live WebSocket"]) --> AR["Live Archiver"] BF(["Historical Backfill"]) --> BI["Backfill Importer"] AR --> CH[("ClickHouse")] BI --> CH end subgraph STANDARD["Existing Parquet path"] direction TB P[("Parquet")] end subgraph NT["NautilusTrader"] direction TB CR["ClickHouseReader"] --> C["CatalogReader interface"] PR["Built-in Reader"] --> C C --> N["BacktestNode"] N --> E["BacktestEngine"] E --> S["Strategy · Orders · PnL"] end CH --> CR P --> PR classDef source fill:#f59e0b,fill-opacity:0.14,stroke-width:3px classDef ingest fill:#f97316,fill-opacity:0.17,stroke-width:3px classDef downstream fill:#a855f7,fill-opacity:0.17,stroke-width:3px classDef storage fill:#22c55e,fill-opacity:0.14,stroke-width:3px classDef native fill:#3b82f6,fill-opacity:0.14,stroke-width:3px class WS,BF source class AR,BI ingest class CR downstream class CH,P storage class PR,C,N,E,S native style CUSTOM fill:#808080,fill-opacity:0.08,stroke-width:3px style STANDARD fill:#808080,fill-opacity:0.08,stroke-width:3px style NT fill:#808080,fill-opacity:0.08,stroke-width:3px linkStyle default stroke-width:5pxI added two pieces:
1. Ingestion: Live WebSocket data goes through an archiver; historical data goes through a backfill importer. Both store columns that match NautilusTrader's
OrderBookDeltaandTradeTickmodels, not serialized Nautilus objects.2. ClickHouseReader: I wrote a read-only Rust reader that implements NautilusTrader's
CatalogReaderinterface. It reads ClickHouse data with ArrowStream and turns the rows back into native NautilusTrader data.From there,
BacktestNodeuses the normal backtest path. The sameBacktestEnginehandles strategy execution, simulated orders, fills, positions, and PnL.One detail for order books
I store individual book delta rows and rebuild the original batches when reading them. The
F_LASTflag marks the end of each batch.I'm using an rc6-based fork with the Catalog/Arrow v2 architecture (#4971) and the newer book-delta batching changes from #5215 backported.
The ClickHouse integration itself is a downstream addition, not part of the official NautilusTrader release.
Why this works well for me
ClickHouse is already where I store market data. Reading from it directly means I don't need a separate Parquet export step just for backtesting.
I'm not suggesting that ClickHouse should replace Parquet. It's simply another way to use NautilusTrader's existing design.
The ingestion pipeline stores the data, the reader loads it, and NautilusTrader runs the backtest.
Thanks to the maintainers for making this kind of extension possible! I'd love to hear if anyone else has built a custom catalog reader.
All reactions