Note
This is a public portion of a larger private project, published as a reference for people learning this domain. The proprietary side (strategies, their parameters, and the research around them) stays private and is not here. What is here is the machinery underneath: an event bus, market-data ingestion, cointegration analytics, a fill simulator, a risk gate and a live runtime, complete enough to build, test and run end to end.
If you are trying to understand how an algorithmic-trading system is actually put together, and where the honest limits of each piece are, that is what this repository is for. Execution is simulated at present, and the docs are explicit about which numbers are measured and which cannot be until there is a venue behind them.
AlgoStream is an event-driven platform for researching statistical-arbitrage strategies: live market-data ingestion, cointegration and mean-reversion analytics, a backtest engine, and a live runtime that drives the same strategy code against a real feed.
Execution is simulated. There is no venue connectivity in this repository today: no exchange
credentials, no request signing, no trading endpoint, so nothing here can place a real order. Fills
are simulated against live quotes by the same engine the backtester uses, and
test/runtime/test_parity.exe asserts the live runtime and the backtest produce identical results
from one fixture. That equivalence is the claim the project makes today; real execution latency and
fill quality are marked unmeasured wherever they appear, and stay that way until there is a venue
behind them.
| Reproduce with | ||
|---|---|---|
| Event-bus latency | 0.07 ms p50 · 0.15 ms p99 | make paced-bench RATE=50000 |
| Ingestion throughput | 427k ev/s | make ingest-bench |
| Bus throughput | ~850k ev/s | make bench |
| Test suite | 709 cases across 25 suites | dune runtest |
Apple Silicon, release profile. Latency is publish-to-handler through the priority queue at a stated offered load; there is no venue leg to include, so this is not an order-execution figure.
The other latency benchmark, event_bus_latency, saturates the bus on purpose and reports queueing
delay in the tens of milliseconds by construction. Both are real and they answer different
questions. paced-bench is the one comparable to a latency target.
An event bus sits at the centre: ingestion publishes into a four-band priority queue, a dispatcher
Domain fans out to subscribers, and each processor drains into its own Domain and publishes an
immutable snapshot other Domains read atomically. Strategies are pure functions over events, so the
same Strategy.S runs under the backtester and the live runtime.
| Library | What it does | Guide |
|---|---|---|
infrastructure/event_bus |
Priority queue, filters, append-only log and replay | event bus |
infrastructure/{lwt_host,network} |
Single Lwt scheduler; HTTP API and SSE dashboard | dashboard |
infrastructure/{auth,persistence} |
Bearer keys with scopes; hash-chained audit log | security |
domain |
Orders, trades, portfolio, positions, pairs | domain models |
data_ingestion |
Binance and Coinbase WebSocket connectors | ingestion |
normalization, time_series |
Canonical symbols, bar building, alignment | normalization · time series |
analytics, pairs |
Rolling statistics; cointegration, hedge ratio, z-score | analytics · pairs |
advanced_models |
Ornstein-Uhlenbeck, Kalman, GARCH, regime detection | advanced models |
strategy, backtest |
The strategy contract and the fill simulator | backtesting |
order_management, risk_management |
Routing, sizing, execution quality; VaR and limits | order management · risk |
montecarlo, optimization, performance |
Simulation, walk-forward, attribution | monte carlo · optimization |
runtime, telemetry, reporting |
Live paper runner, metrics, report export | live runtime · telemetry |
Written in OCaml 5.x, using Domain for parallelism, Atomic for publication and Lwt for I/O. The
numerics are hand-rolled rather than pulled from a linear-algebra dependency; the reasoning is in
the relevant guides.
- OCaml 5.0 or newer. The project uses
Domain, so 4.x will not build it. CI covers 5.0.x and 5.1.x on Linux and macOS. If your active switch is a 4.x one,maketargets fail withLibrary ... not foundrather than anything naming the real cause. - OPAM package manager
- Docker (optional, for
perf/valgrind/gprofprofiling on macOS)
git clone https://github.com/chizy7/AlgoStream.git
cd AlgoStream
# Easiest path: creates a local 5.1.1 switch and installs everything
./scripts/setup-dev.sh
eval $(opam env)
# Or do it manually
opam switch create . ocaml-base-compiler.5.1.1
eval $(opam env)
opam install . --deps-only --with-test --with-dev-setup
make build # dune build
make test # dune runtest
make fmt # dune build @fmt --auto-promote
make fmt-check # CI-style check (no auto-fix); fails if anything would changemake docker-dev # build Dockerfile.dev + start container
make docker-shell # bash into the container
# inside the container:
make perf-record # produces perf.data
make valgrind-massif # produces massif.out.* (heap profile)
make memtrace BIN=bin/event_replay.exe ARGS="--log-file replay.bin"The quickest look at a running system:
make dashThen http://127.0.0.1:8080/dashboard/. Note the path; / is the landing page.
No keystore means no credential required, and the listener is loopback-only.
# The key is printed ONCE. Only its SHA-256 goes into the keystore.
dune exec bin/keyctl.exe -- add --label laptop --scopes read,control
# A read-only one as well, to watch the control buttons grey out
dune exec bin/keyctl.exe -- add --label viewer --scopes read
dune exec bin/keyctl.exe -- listKeys land in $XDG_CONFIG_HOME/algostream/keys.json (~/.config/...), mode 0600. The daemon
refuses to start if the permissions are wider, rather than warning.
dune exec bin/algostream.exe -- \
--auth-keys ~/.config/algostream/keys.json \
--audit-dir /tmp/algostream-audit \
--static site/ --http-port 8080The dashboard now prompts for a key. Paste the read-only one first: the stream goes live and all three control buttons disable with "this key is read-only". Sign out, paste the control key, and they enable.
Directly:
KEY='ask_...'
curl -s localhost:8080/api/health | jq # public; shows auth_required
curl -i localhost:8080/api/telemetry # 401 + WWW-Authenticate
curl -i -XPOST -H "Authorization: Bearer $KEY" localhost:8080/api/strategies/pairs-1/stop/api/health stays public deliberately, so a client can tell a live daemon from an unreachable one
without holding a credential.
dune exec bin/auditctl.exe -- tail /tmp/algostream-audit
dune exec bin/auditctl.exe -- verify /tmp/algostream-audit # exit 1 on a break, cron-friendly
dune exec bin/auditctl.exe -- head /tmp/algostream-audithead prints the anchor. Copy it somewhere the daemon cannot write. The chain is unkeyed, so
anyone who can write the log can recompute it and hand you a file that verifies perfectly. The
security guide shows how to demonstrate that, and what an anchor buys you.
curl -s -H "Authorization: Bearer $KEY" localhost:8080/metrics | head
make stack-up # daemon + Prometheus + Grafana + Alertmanager; mints keys on first run
make stack-downScope-gated like every other observation endpoint. Grafana :3000 (anonymous viewer), Prometheus
:9090, Alertmanager :9093, all bound to loopback.
- Getting Started Guide — setup, first run, troubleshooting
- Event Bus Guide — Publish/subscribe, filters, replay, instrumentation, memtrace
- Domain Models Guide — Complete guide to core models
- Backtesting Guide — strategy contract, fill model, step ordering
- Monte Carlo Guide — RNG determinism, bootstrap, stress, parallel speedup
- Performance Analytics — metric conventions and consolidation
- Strategy Optimization — walk-forward, purged CV, overfitting statistics
- Security — threat model, keys, scopes, the audit chain
- Deployment — container, observability stack, Kubernetes
- Operations runbook — what to do when something is wrong
- Performance tuning — GC, pinning, machine-level settings
- Statistical arbitrage theory — mathematical foundations
All 24 guides are published at chizy7.github.io/AlgoStream/guides, and the generated API reference at /api.
make test # alcotest, the full suite
# or: dune runtest
# A few of the suites
dune exec test/domain/test_runner.exe # 11 domain tests
dune exec test/infrastructure/event_bus/test_runner.exe # 14 event-bus tests
dune exec test/backtest/test_runner.exe # 27 backtest tests
dune exec test/montecarlo/test_runner.exe # 26 Monte Carlo tests
dune exec test/optimization/test_runner.exe # 44 optimization tests
# Lints: clock leaks in pure layers, banned RNGs, metric duplication
make determinism-lint
# Security lints: constant-time comparison, no deterministic RNG in auth,
# append-only audit log, no credentials committed
python3 scripts/security-lint.py
# Kubernetes and alert-rule schemas (Docker)
make k8s-validate
# Performance benchmarks (release profile)
make bench # text output
make bench-json # JSON output for github-action-benchmark
make paced-bench # latency at a stated offered load; the figure to quoteFull detail in docs/guides/deployment.md, which records what has actually been exercised and what has not.
make docker-release # multi-stage build: non-root, no sudo, read-only rootfs
make stack-up # daemon + Prometheus + Grafana + Alertmanager
make stack-downDockerfile (root) is the release image. Dockerfile.dev is a profiling shell (passwordless
sudo, SYS_ADMIN, seccomp:unconfined), which is correct for running perf and valgrind and
disqualifying for anything unattended. Never derive one from the other.
A container must bind 0.0.0.0, since a pod's loopback is unreachable from outside its network
namespace. That is exactly the case the daemon refuses by default, so --auth-keys is mandatory
in a container and it exits 2 without one.
make k8s-validate # kubeconform -strict + promtool
kubectl create secret generic algostream-keys --from-file=keys.json
kubectl apply -f k8s/
scripts/blue-green.sh --image algostream:v2 # health-gates before flipping the Service
scripts/blue-green.sh --rollbackWhat has been exercised: the manifests and the blue/green script have been applied to a single-node kind cluster, where the pod reached Ready through its own
/api/healthprobes and a promotion and a rollback both completed without the Service losing its endpoint. Multi-node is unproven, and the resource figures are reasoned from the benchmarks rather than measured under load. TheSecretink8s/service.yamlis a template carrying no key material.One real constraint: the audit volume is
ReadWriteOnceand both slots mount it, so a cross-node blue/green handover blocks until the old pod releases it. Pin both slots to one node, use aReadWriteManyclass, or accept a brief gap.
replicas is 1 per slot on purpose. Each instance opens its own exchange connections and runs its
own copy of the strategy, so a second replica means duplicate subscriptions and two independent
paper portfolios rather than shared load. Blue/green buys availability across a deploy; it does
not make this a clustered system.
Contributions are welcome. See CONTRIBUTING.md for the build setup, code style and test expectations. Security issues should go through SECURITY.md rather than a public issue.
MIT. See LICENSE.