ARCHIVED - Future improvements will be scoped into the pro version bundled with Flow master
Plugin-based backtesting for HFT strategies — where every nanosecond counts.
~15K lines Rust (9 crates) + ~3K lines GPU (CUDA + OpenCL + Metal). Full LOB reconstruction, nanosecond-precision tick replay, and a custom DSL compiled to bytecode.
┌──────────┐ ┌───────────────┐ ┌─────────┐ ┌──────────┐ ┌─────────┐
│ bt-data │───>│ bt-tick-emul. │───>│ bt-oms │───>│bt-strategy│───>│ Results │
│ DuckDB │ │ LOB + Exec. │ │ VWAP │ │ DSL VM │ │ PnL/CSV │
└──────────┘ └───────────────┘ └─────────┘ └──────────┘ └─────────┘
│
┌───────────┴───────────┐
│ GPU (CUDA / OpenCL) │
│ N scenarios parallel │
└───────────────────────┘
Same cargo bench workloads, two host CPUs side by side. Bold = faster.
| Operation | Apple M4 | RTX 6000 host (Linux) |
|---|---|---|
| DSL VM step (arithmetic, 4 signals) | 10.2M/s · 98ns | 19.8M/s · 51ns |
| DSL VM step (SMA, any window — O(1)) | 25M/s · 40ns | 46M/s · 22ns |
| DSL VM (full momentum strategy) | 7.8M/s · 129ns | 14.1M/s · 71ns |
| DSL VM (stdev + zscore + percentile) | 11M/s · 91ns | 19.3M/s · 52ns |
| DSL compile (lex + parse + compile) | 164K/s · 6.1µs | 163K/s · 6.1µs |
| LOB snapshot (200 levels) | 165K/s · 6.1µs | 197K/s · 5.1µs |
| LOB delta (1 update) | 108M/s · 9.3ns | 108M/s · 9.3ns |
| LOB best bid/ask | 54M/s · 18ns | 106M/s · 9.4ns |
| LOB mixed stream (1 snap + 100 deltas) | 92K cycles/s | 103K cycles/s |
cargo bench --bench cpu_vm and cargo bench --bench lob_replay. Apple M4 — 10 cores, 16 GB. RTX 6000 host — Linux box, the GPU optimizer numbers are in the GPU section.
| Crate | What |
|---|---|
| bt-core | FastDecimal (i128, 10^18), LOB BTreeMap, Order/Fill/Trade types, TimeseriesBuffer |
| bt-plugin | Traits: DataLoader, Emulator, OMS, Strategy, Optimizer |
| bt-data | DuckDB loader, columnar LRU buffers, ingesters (Bybit LOB JSON, BTCUSD CSV, TAQ, OHLC) |
| bt-tick-emulator | LOB engine (snapshot+delta), execution heuristic (volume-capped, direction-matched), market impact models (linear, sqrt), 4 timing modes |
| bt-oms | Order router (Direct passthrough, VWAP time-sliced), meta-order lifecycle |
| bt-dsl | Aria-style DSL: lexer, parser, 46-opcode bytecode compiler, register-machine CPU VM |
| bt-strategy | DSL strategy runner, risk manager (position/drawdown/stop-loss), portfolio coordinator |
| bt-optimizer | Grid/random/Bayesian search, walk-forward scenarios, GPU batch execution, Pareto aggregation |
| bt-engine | CLI binary, TOML config, pipeline orchestrator, GPU FFI dispatch, sharding IPC, CSV/JSON output |
signal sma_fast = sma(close, 10)
signal sma_slow = sma(close, 50)
signal up = cross_up(sma_fast, sma_slow)
signal down = cross_down(sma_fast, sma_slow)
let qty = 0.01
strategy momentum {
when up -> buy(qty)
when down -> sell(qty)
}
Built-in functions: sma, ema, cross_up, cross_down, run_max, run_min, stdev, zscore, percentile, abs, max, min, exp, log, prev
Order actions: buy(qty), sell(qty), flatten(), limit_buy(qty, price), limit_sell(qty, price), vwap(qty, duration_secs)
Operators: +, -, *, /, %, >, <, >=, <=, ==, !=, &&, ||, !, |> (pipe)
Mutable state: var count = 0.0 then count := count + 1.0
All examples below run against the bundled synthetic dataset
(data/synth_oss.duckdb — 11 tickers, 4 asset classes, deterministic seed),
so they work out of the box with no external data ingestion.
A handful of additional configs in
examples/(btcusdc_hft.toml,cuda_real_hft.toml,aave_*,sp500_ohlc.toml, …) target real ingested data underdata/duckdb/db1.duckdb. Skip those until you've run the matching ingestion pipeline; the synth ones are the right entry point for first-time use.
# Build & test
cargo test --workspace
# Single-config tick replay (CPU only — see GPU section)
cargo run --release --bin bt-engine -- examples/oss_crypto_hft.toml
# Single-config OHLC backtest (CPU only)
cargo run --release --bin bt-engine -- examples/oss_equity_ohlc.toml
# GPU param sweep — this is the path that actually drives a GPU backend
cargo run --release --bin bt-engine -- optimize examples/cuda_optimize_momentum.toml \
--csv /tmp/sweep.csv --json /tmp/sweep.json --top-n 10
# Benchmarks
cargo bench --bench cpu_vm
cargo bench --bench lob_replayAll data lives in DuckDB. Schema: data/synth_oss_schema.sql.
SourceId convention: md5_number('ByBit'), md5_number('Yahoo'), md5_number('Synth') — deterministic UHUGEINT hash per source.
| Source | Format | Ingestor |
|---|---|---|
| Bybit HFT | Newline-delimited JSON (orderbook snapshots + deltas) | ingest::bybit_lob |
| Bybit trades | CSV (id, timestamp, price, volume, side) | via DuckDB CLI |
| BTCUSD perpetual | CSV (10-level bid/ask) | ingest::btcusd_lob |
| NYSE TAQ | Pipe-delimited equity trades | ingest::taq_trades |
| Yahoo Finance | CSV OHLC per ticker | ingest::ohlc |
| XAUUSD | Semicolon-delimited MT5 OHLC | ingest::ohlc |
| Synthetic | Generated (11 tickers, 4 asset classes) | bt-engine synth-data |
Configuration
All parameters are configurable via TOML. No hardcoded values. Example based on examples/oss_crypto_hft.toml (which runs against the bundled data/synth_oss.duckdb):
[general]
tickers = ["BTCUSDC"]
start_time = "2026-01-01"
end_time = "2026-12-31"
mode = "tick" # tick | ohlc
[data]
duckdb_path = "data/synth_oss.duckdb"
frequency = "1d" # for ohlc mode
[strategy]
file = "strategies/momentum.aria"
[risk]
max_position = 1.0
max_drawdown = 0.20
stop_loss = 500.0
max_open_positions = 5
[execution]
volume_multiplier = 0.10 # max 10% of observed market volume
lob_fill_fraction = 0.50 # limit order fill fraction of top-of-book
timing_mode = "full_speed" # full_speed | realistic | latency_test | min_latency
computation_offset_ns = 1000000
latency_multiplier = 2.0
min_latency_ns = 5000000
default_quantity = 0.01
[impact]
model = "none" # none | linear | sqrt
alpha = 0.1 # linear: impact per unit of order/depth ratio
beta = 0.5 # linear: fraction of spread as slippage
daily_sigma = 0.02 # sqrt: daily volatility
adv = 1000000.0 # sqrt: average daily volume
[portfolio]
max_total_exposure = 1000000.0
max_positions = 10
[gpu]
backend = "none" # none | cuda | opencl | metal
num_scenarios = 10000
[sharding]
mode = "none" # none | scenario | time
num_workers = 4
num_gpus = 2
[output]
pnl_csv = "results/pnl.csv"
trades_csv = "results/trades.csv"
json_summary = trueCUDA, OpenCL, and Metal kernels implement the full DSL VM (46 opcodes), LOB replay, execution matching, and reduction kernels (PnL, Sharpe, drawdown).
Rust loads GPU libraries at runtime via libloading — no CUDA/Metal/OpenCL dependency at compile time.
Where the GPU actually kicks in: only the
bt-engine optimize <cfg>param-sweep path probes a GPU backend. Single-config tick/ohlc runs (bt-engine <cfg>) always go throughCpuPipelineregardless of any[gpu]section — there's no GPU code path for sequential single-stream replay. The binary's stdout reflects this.
# CUDA (Linux)
cd gpu && mkdir -p build && cd build
cmake .. -DENABLE_CUDA=ON -DENABLE_OPENCL=ON
make -j$(nproc)
# -> libbt_gpu_cuda.so, libbt_gpu_opencl.so
# Metal (macOS)
cd gpu && mkdir -p build-metal && cd build-metal
cmake .. -DENABLE_CUDA=OFF -DENABLE_OPENCL=OFF -DENABLE_METAL=ON
make -j$(sysctl -n hw.ncpu)
# -> libbt_gpu_metal.dylibTested on: NVIDIA RTX PRO 6000 Blackwell Max-Q (96GB, sm_90; CUDA + OpenCL fp64), Apple M4 (Metal + OpenCL, fp32 on Metal)
bt_engine::gpu_runner::GpuScenarioRunner replays N bar-aligned market scenarios in parallel. The Metal and OpenCL backends batch K vm_step dispatches per submission, amortizing per-launch sync overhead.
# Quickstart: 8 OHLCV-derived Monte Carlo scenarios on the first available backend
cargo run --release --example gpu_runner_quickstart -p bt-engineWired into the optimize CLI via [gpu] backend = "auto" (probes metal → cuda → opencl):
[gpu]
backend = "auto" # or explicit: "metal" / "cuda" / "opencl" / "none"Release-mode optimizer wallclock on a 96 param × 20 scenario × 200 LOB-event SMA-crossover sweep (Apple M4):
| Backend | Wallclock | Notes |
|---|---|---|
| Metal batched | 4.9s | ~2x faster than CPU baseline |
| CPU | 9.5s | Workspace baseline |
| OpenCL batched | 28.7s | Apple's deprecated OpenCL |
| Metal single-step (pre-batching) | 74.3s | Per-launch sync dominated |
Same workload — examples/cuda_optimize_momentum.toml (96 params × 20 scenarios over data/synth_oss.duckdb) — on NVIDIA RTX PRO 6000 Blackwell Max-Q (sm_90, fp64):
| Backend | Wallclock | Speedup vs CPU |
|---|---|---|
| CUDA batched | 0.69s | 10.1× |
| OpenCL batched | 2.45s | 2.9× |
| CPU (release, rayon) | 7.00s | 1.0× (baseline) |
Reproduce:
target/release/bt-engine optimize examples/cuda_optimize_momentum.toml --top-n 3
target/release/bt-engine optimize examples/opencl_optimize_momentum.toml --top-n 3
# CPU baseline: copy the CUDA toml, set [gpu] backend = "none"The optimize command prints the top 10 to stdout and can also dump the full all_results grid for downstream analysis:
[output]
optimizer_csv = "results/sweep.csv" # one row per param set, score + 11 metrics + flat param columns
optimizer_json = "results/sweep.json" # same data as JSON array, params nested under "params"Both files are sorted by score descending. CSV is spreadsheet-friendly; JSON parses cleanly into Pandas / serde without column-name acrobatics.
CLI flags can override the TOML config without editing it (e.g. for ad-hoc sweeps):
bt-engine optimize config.toml \
--csv results/sweep.csv \
--json results/sweep.json \
--top-n 25Two benchmark suites via Criterion.rs. Run with scripts/bench.sh to capture platform info + results.
scripts/bench.sh # all benchmarks + platform report
scripts/bench.sh cpu_vm # DSL VM only
scripts/bench.sh lob_replay # LOB onlyResults are saved in results/bench_<host>_<date>.txt and target/criterion/ (HTML).
CPU VM benchmarks — Apple M4 vs RTX 6000 host CPU
| Benchmark | Apple M4 | RTX 6000 host | Notes |
|---|---|---|---|
vm_arithmetic_4signals |
98 ns · 10.2M/s | 51 ns · 19.8M/s | 4 signal computations per step — pure ALU bound |
vm_sma/5 |
40 ns · 25M/s | 22 ns · 46M/s | O(1) ring buffer SMA |
vm_sma/200 |
40 ns · 25M/s | 22 ns · 46M/s | Same as SMA/5 — confirms O(1) at any window |
vm_momentum_strategy_full |
129 ns · 7.8M/s | 71 ns · 14.1M/s | Full strategy (2× SMA + cross + orders) — ~3× arithmetic cost |
vm_stdev_zscore_percentile |
91 ns · 11M/s | 52 ns · 19.3M/s | Statistical functions close to arithmetic cost |
vm_throughput/1 |
50 ns · 20M elem/s | 23 ns · 43M elem/s | Base overhead per VM step |
vm_throughput/20 |
476 ns · 42M e/s | 335 ns · 60M e/s | Cache-friendly: 20 signals cheaper than 20× one signal |
compile_full_strategy |
6.1 µs · 164K/s | 6.1 µs · 163K/s | Lex + parse + compile — not a bottleneck on either platform |
LOB replay benchmarks — Apple M4 vs RTX 6000 host CPU
| Benchmark | Apple M4 | RTX 6000 host | Notes |
|---|---|---|---|
lob_snapshot/10 |
165 ns · 61M lvl/s | 115 ns · 87M lvl/s | 10-level book replacement — BTreeMap insert dominated |
lob_snapshot/50 |
— | 958 ns · 52M lvl/s | 50-level — sub-µs |
lob_snapshot/200 |
6.1 µs · 33M lvl/s | 5.1 µs · 39M lvl/s | 200-level full book — scales ~linearly with depth |
lob_delta/1 |
9.3 ns · 108M/s | 9.3 ns · 108M/s | Single price level update — near-cache-line speed |
lob_delta/5 |
65 ns · 77M/s | 53 ns · 95M/s | 5 updates incl. removals — still sub-100ns |
lob_delta/20 |
229 ns · 85M/s | 220 ns · 91M/s | 20 updates — BTreeMap amortizes well |
lob_best_bid_ask |
18 ns · 54M/s | 9.4 ns · 106M/s | bid + ask + mid + spread (BTreeMap first/last) |
lob_mixed_stream_100 |
10.9 µs · 92K cyc/s | 9.7 µs · 103K cyc/s | Realistic feed: 1 snapshot + 100 deltas per cycle |
Key takeaway: delta updates (the common production case) are ~550× faster than full 200-level snapshots on both platforms. A realistic 100-event feed cycle finishes in under 11 µs — fast enough for sub-millisecond tick replay regardless of host.
Testing
# Full test suite
cargo test --workspace # 398 tests
# GPU conformance — CPU vs every available backend (CUDA, OpenCL, Metal)
# Requires gpu/build/libbt_gpu_*.{so,dylib} pre-built.
cargo test --test gpu_conformance --release -- --ignored --test-threads=1
cargo test --test gpu_conformance_advanced --release -- --ignored --test-threads=1
# Metal conformance (macOS only)
cargo test --test metal_conformance -- --ignored
# Specific crate
cargo test -p bt-core
cargo test -p bt-dsl
cargo test -p bt-data
# Regenerate the bundled synthetic dataset
cargo run --release --bin bt-engine -- synth-data data/synth_oss.duckdbSee TESTING.md for the full breakdown — every test, what it covers, and how to invoke it.
├── Cargo.toml # Workspace root
├── crates/
│ ├── bt-core/ # Types, decimal, LOB, orders
│ ├── bt-plugin/ # Plugin traits
│ ├── bt-data/ # DuckDB loader, ingesters, synthetic data
│ ├── bt-tick-emulator/ # LOB engine, execution, timing, impact
│ ├── bt-oms/ # Order routing, VWAP
│ ├── bt-dsl/ # DSL compiler + VM
│ ├── bt-strategy/ # Strategy runner, risk, portfolio
│ ├── bt-optimizer/ # Parameter search, scenarios, batch
│ └── bt-engine/ # CLI, config, pipeline, GPU FFI, output
├── gpu/
│ ├── CMakeLists.txt
│ ├── include/ # bt_gpu.h, bt_decimal.h
│ └── src/
│ ├── cuda/ # VM, LOB, execution, reduction kernels
│ ├── opencl/ # VM, LOB, reduction kernels
│ ├── metal/ # Metal shaders (macOS)
│ └── common/ # Shared decimal utilities
├── strategies/ # .aria DSL files
├── examples/ # .toml configs
├── data/ # Market data, DuckDB, synthetic datasets
│ └── synth_oss_schema.sql
└── TODO.md # Remaining work