Architecture

Architecture

Diagrams of everything on this page: C4 model. The user's view of the package: Library.

Everything on this page but the Protocol and Server sections is sbt2.core, in the sbt2-core distribution; module names below are relative to it. The other distributions of the workspace are listed on Library.

Modules

One module per thing it owns: a piece of state or a set of files on disk, not one module per step. A module is a package of small files, and no class in it does more than one step. That, not the number of modules, is what keeps v1's god object from coming back.

Module	Owns	Inside
Data	raw files, the catalog	sources (the source adapters), download (network → raw), ingest (raw → catalog, the only catalog writer), catalog (the reader: coverage from nautilus's catalog, notebook loaders; the writer ingest uses)
Spec	what to run and on which dates	parse and expand spec files, resolve and hash, splits, venue profiles
Strategy	the strategy base	the thin base over nautilus's Strategy (metadata, search space, bar subscriptions, warm-up gate), strategy loading and parameter resolution, the drawdown guard
Run	executing backtests	pre-flight (fetching missing data, checks, the test lock), execute (one backtest in-process: the engine), batch (a forked process per run, the launcher, parallelism)
Results	stored runs	store (RunOutput and the output sink, the ResultStore interface and its parquet backend), metrics, pricing (priced runs and benchmarks), costs (costs and exposure), comparisons, report (the Report and tearsheets)
optimise	nothing	the optimiser interface, grid search, objectives; runs its candidates through run.batch
api	nothing	sbt2.core.backtest: the one core both front-ends call
CLI	nothing	thin commands over api and data

The engine is a function, not a module: run.execute(resolved_spec, sink) runs one backtest in the current process, which is all tests need. run.batch(...) wraps it in processes forked from a forkserver, one per run.

Plug-ins

The core never names a specific exchange or asset class. Plug-in points carry everything specific:

• Data sources (data/sources/*): how to fetch and parse one source's raw data, and its known data gaps.
• Venue profiles: the simulated exchange's settings, and the source that serves its instruments (see Spec). They are configuration, not code: they live in the config folder (below), never in a package.
• Asset classes (assets/*): one profile per nautilus AssetClass + InstrumentClass pair: instrument type, calendar, annualization, carry (nautilus-native, or a custom module for a gap), margin defaults, valuation price, buy-and-hold convention.

Each plug-in point defines its interface in its own package (data/sources/base.py, assets/base.py). Plug-ins depend only on nautilus. Sources are discovered through the entry point sbt2.sources, so other packages can add them; sbt2 registers its own the same way. Modules that need one look it up from the resolved run spec (a source by name, an asset profile by its nautilus classes), never by importing a specific plug-in.

Data flow

 Python: sbt2.core.backtest(...) ─┐
 CLI: sbt2 run spec.toml ─────────┴─► api ──► spec: parse · expand · resolve · hash
                                                       │
                                                       ▼
      data sources ──► data: download · ingest ◄── run: pre-flight ──► results store
                              │                  (fetch missing days,   (test lock: earlier
                              ▼                   coverage, types, lock)  test runs)
                        root/catalog                   │  one forked process per run
                              │                        ▼
                              └──────────────► run: execute ◄── user's strategy (nautilus Strategy on the sbt2 base)
                                                       │   ◄── venue profile (nautilus settles carry)
                                                       ▼
                              results: output sink ──► runs/{run_id}/ ──► summary written last
                                                                               │
                                        optimise: rank candidates on validation ◄┤
                                                                               ▼
                                                  results: Report, metrics, tearsheets, benchmarks

Dependency rules

Dependencies point one way only:

data/sources/*, assets/*, strategy ─► nautilus only (strategy names results' Benchmark as a type only)
user strategies ─► strategy (the strategy base), nautilus
spec     ─► strategy (metadata only), assets
data     ─► assets
results  ─► spec, data, assets             (resolved spec type, benchmark prices, annualization)
run      ─► spec, data, strategy, results, assets
optimise ─► run, results, spec
api      ─► spec, run, optimise, results
cli      ─► api, data

• Type-only imports may cross layers. An import under TYPE_CHECKING exists only for the type checker, so the import-linter contracts ignore it (exclude_type_checking_imports). This is how the strategy base types its benchmark as results' Benchmark without importing results at run time.
• Each type lives with the module that owns it: RunSpec in Spec, RunOutput, the output sink and Report in Results, the strategy base in Strategy. There is no shared-types module.
• spec imports strategies only for metadata: parameter schema, defaults, search space, warm-up. Strategies run only inside run.execute, in a child process.
• download and ingest never import each other. They share only each source adapter's conventions: URLs, raw paths and the instrument-id format.
• ingest is the only writer to the catalog. Everyone else reads through data.
• run.execute knows nothing about processes, the front-ends or where the sink writes.
• Strategies use nautilus directly, but never reach the engine or the node; they see only what a nautilus Strategy sees.
• Strategies live in the user's own installed package. sbt2 reaches them only through the class passed to backtest or the import path in a spec file, never by importing one directly.

Root

Everything a user's data and runs produce lives under one data root, which every command that reads or writes data needs: --data PATH, else the SBT2_DATA environment variable; the Python API takes it per call (root=). There is no default and no fallback to the working directory: without either, a command exits with a usage error naming both, and creates nothing. A container mounts its data volume at a fixed path and sets SBT2_DATA. A different disk per folder is done with symlinks or volume mounts.

$SBT2_DATA/
├── settings.toml  run settings (memory budget, per-run cap, chunk sizes); optional, defaults built in
├── known_gaps.toml  days each source confirmed it lacks, one list per source; optional, none when missing
├── raw/           downloads, one folder per source, the source's own file names kept
├── catalog/       nautilus ParquetDataCatalog
└── results/       default ResultStore backend
    ├── runs/{run_id}/           one folder per run: the resolved run spec, written when the run starts; equity, fills, positions, carry ledger, [orders] as parquet and summary.parquet (written last) when it ends
    └── index                    DuckDB view over runs/*/summary.parquet; nothing writes to it

Venue profiles are not under the root: they are configuration, in the config folder (below). The known gaps describe the data held, so they live with it. Results hold no absolute paths, so a data root can be moved or mounted elsewhere and its runs still read back.

Config folder

The configuration a run reads lives in one config folder: sbt2 run --config PATH, else the SBT2_CONFIG environment variable (ConfigFolder in config.py, like Root for data). It is required, with no default and no fallback to the working directory or $XDG_CONFIG_HOME: without either, run exits with a usage error naming both, and creates nothing.

$SBT2_CONFIG/
└── venues.toml    the venue profiles, one table per profile (see [[Spec]])

sbt2-core ships no defaults: no venue profile lives in the package. The defaults live in the GUI, which installs them through the server's API when it connects to a server whose config folder is empty; until then the README shows a minimal venues.toml to write. A missing venues.toml fails the run with MissingConfigError, naming the file. The folder is read on every run, never cached, so an edit, from the GUI or by hand, applies to the next run; a run already started is unaffected, since its resolved spec, venue arguments included, is stored in its spec.json.

Protocol

The GUI talks to sbt2 only through the server (sbt2-server, see Library), in protobuf messages over a WebSocket. The protocol is the GUI's contract, and its single source of truth is the .proto files of sbt2-protocol/ in this repository, in package sbt2.protocol.v1.

sbt2-protocol/
├── buf.yaml            buf's lint and breaking rules
├── COPYING, COPYING.LESSER
└── sbt2/protocol/v1/
    ├── envelope.proto  ClientMessage, ServerMessage, the Hello/Welcome handshake
    ├── types.proto     messages shared by every area: Error and its ErrorCode
    └── <area>.proto    one file per area: runs, studies, config, strategies, results

• Its own folder: the protocol is public and LGPL-3.0-or-later. The repository's prek hooks format and lint it and check it for breaking changes against sbt2-protocol/ on main, and its pipeline runs its Python and Rust tests. A message for a new area and the server code that uses it land in one pull request of this repository, in one component or in both.
• Envelope: each binary WebSocket frame is one message: a ClientMessage from the GUI, a ServerMessage from the server, each with a request_id and a oneof body. The client picks a non-zero request_id per request, and every answer carries it back. A server push has request_id = 0 and the subscription_id it belongs to.
• Errors: Error { code, message, details } is a ServerMessage body answering the failed request. It lives in types.proto, not envelope.proto: the envelope imports every area, so an area that embeds an Error (say, per failed item) would otherwise import the file that imports it.
• Handshake: a connection starts with the client's Hello (its version), answered with Welcome (the server's version and its Capability list).
• Field rules: prices, quantities and money are decimal strings, never double; timestamps are google.protobuf.Timestamp; a field whose unset value means something other than zero or empty is proto3 optional; a removed field's number and name are reserved and never reused.
• Versions: the package is the protocol's major version. A change that buf breaking rejects needs a new package, sbt2.protocol.v2, next to sbt2.protocol.v1.
• Code generation, offline with locked tools: scripts/generate-protocol.sh runs protoc from grpcio-tools with mypy-protobuf's plugin, both pinned in sbt2-backend/uv.lock, and writes the Python modules and their stubs to sbt2-backend/packages/sbt2-server/src/sbt2/protocol/v1/, which is committed and never edited by hand. The GUI reads sbt2-protocol/ from a tag of this repository and generates its Rust types in its build.rs, with protox (a pure-Rust .proto compiler) and prost-build, both pinned in Cargo.lock; nothing in sbt2 serves it. Neither side uses Buf's registry or any other online service to generate code.
• sbt2-core knows nothing of the protocol: the server translates between the messages and sbt2-core's types.

Server

sbt2.server, in sbt2-server, serves the protocol to the GUI. It uses only sbt2-core's public interface, and an import-linter contract keeps sbt2-core, sbt2-strategies and sbt2-papers from importing it or sbt2.protocol; another keeps the generated sbt2.protocol from importing any of sbt2.

sbt2/server/
├── auth.py        BearerToken: the API token, compared in constant time
├── routing.py     Router, Handler, Outbox and offloaded
├── connection.py  one client's session: the Hello gate, a task per request, the outgoing queue
├── server.py      Server and Settings: the Hello route, health and auth before the upgrade
├── results/       the results area: routes(root) and its private encoding, chunking and tearsheet cache
├── config/        the config area: routes(config, root) and its private encoding
├── transport/     the Transport, Channel and Endpoint interfaces, and WebsocketsTransport
└── cli.py         the sbt2-server command (see [[CLI]])

• Transport: binary WebSocket frames, one envelope per frame, in plain ws://. sbt2 ships no reverse proxy: TLS, and with it wss://, is the operator's to add in front of the server (see Deployment below). The websockets library (asyncio) serves them behind abstract base classes of the server's own: a Transport serves an Endpoint on an address, and hands it each upgraded connection as a Channel (its frames, send, close). A faster library, such as picows, would replace WebsocketsTransport by implementing them, if profiling ever shows the socket layer matters. Only sbt2.server.transport imports its internals.
• Before the upgrade: GET /health is answered 200 OK on the same port, with no token. Any other request must carry Authorization: Bearer <token> with the server's one API token, SBT2_SERVER_TOKEN, or gets HTTP 401; the token is meant for one user or a trusted team. A handshake that doesn't ask for the subprotocol sbt2.v1 gets HTTP 400.
• Hello first: until a connection's first Hello, any other request gets Error INVALID_MESSAGE with its request_id; the connection stays open, and is usable once Hello arrives. Hello is answered with Welcome: server_version is the sbt2-server package version, capabilities the areas the server was built with; sbt2-server serves CAPABILITY_RESULTS and CAPABILITY_CONFIG. A repeated Hello gets Welcome again.
• Routing: the router is an explicit registry from a body's field name in ClientMessage to its handler, async def handler(request: ClientMessage, outbox: Outbox) -> ServerMessage; the router sets the reply's request_id to the request's. The server registers hello itself; each area adds its handlers and capability with Server(settings, routes, capabilities), so the server knows no area; the sbt2-server command is where the areas are put together. A route for a name that isn't a body of ClientMessage fails when the router is built (UnknownBodyError).
• Errors, never a dropped connection: a frame that isn't a ClientMessage, or a text frame, gets INVALID_MESSAGE with request_id = 0; a request without a body INVALID_MESSAGE; request_id = 0 INVALID_ARGUMENT; a body without a handler UNIMPLEMENTED; a handler that raises INTERNAL, with a generic message: the traceback goes to the server's log only. The connection stays open in every case.
• The event loop never blocks: each request runs in a task of its own, so a slow one holds up neither the next request nor other connections. A handler runs file, store or CPU-heavy work off the loop: offloaded(fn) is fn run through asyncio.to_thread; backtests run in their own processes (see Run).
• Bounded outgoing queue: replies and pushes go through one queue per connection, of Settings.queue_size messages (256 by default). A push (Outbox.push) never waits: a client that falls that far behind is closed with WebSocket code 1013 (try again later), so a slow client can't grow the server's memory. A reply and the chunks before it (Outbox.send) wait for room in the queue instead, so a long answer slows to the client's pace; a disconnect cancels the waiting request and releases what it holds.
• Chunked replies: an answer too large for one envelope comes as numbered chunks sharing the request's request_id: each body has an index from 0 and a last flag set on the final one only, and every whole ServerMessage serializes to at most 1 MiB. An empty answer is still one chunk with last set. A failure after some chunks ends the answer with an Error for the same request_id, and the chunks before it are void.
• One replica, one worker: one process serves every connection on one event loop.

Config area

sbt2.server.config.routes(config, root) answers the requests of config.proto: the venue profiles in the config folder's venues.toml, the known gaps in the data root's known_gaps.toml, and the venue model kinds. It uses sbt2-core's public API only: spec.venue_profiles, put_venue_profile, delete_venue_profile, model_kinds, and data.listed_gaps, add_known_gaps, remove_known_gaps. Each answer is one message.

Request	Answer
ListVenueProfiles	VenueProfiles: every profile as stored, without the asset class's venue defaults
PutVenueProfile	ConfigWritten; creates the profile, or replaces the one of the same name
DeleteVenueProfile	ConfigWritten
ListKnownGaps	KnownGaps: every gap, with its source, symbol, data type and day
AddKnownGaps, RemoveKnownGaps	ConfigWritten; a batch, all or nothing; a gap listed twice is kept once, and removing a gap not listed is skipped
ListModelKinds	ModelKinds: each model argument's kinds, and each kind's parameters with type, whether required, and default

• Profiles: a VenueProfile has typed fields for its name, the nautilus venue name, source, asset class and instrument class, each model argument as {kind, config} (modules as a list), and the other BacktestVenueConfig arguments in a google.protobuf.Struct. A Struct holds numbers as doubles, so the server stores an integral number as an integer.
• No defaults: the server holds none. An empty config folder answers empty lists; the GUI installs its defaults.
• Validated by core, atomic: sbt2-core checks a write as a run reads it before the file changes, and writes the file to a temp file in the same folder, then replaces it. Writes to one file run one at a time, under a lock per file, and off the event loop; reads go to the files every time, so the next run sees each write. Rewriting a file drops its comments and may reorder its keys.
• Model parameters: their types come from the model builders' signatures; a parameter whose type can't be told is a string.
• Errors: an invalid profile or gap (an unknown model kind or source, a bad model config, a missing field, a bad day) is INVALID_ARGUMENT, deleting an unknown profile NOT_FOUND, each with sbt2-core's message.

Results area

sbt2.server.results.routes(root) answers the browsing requests of results.proto from the data root's store, through sbt2-core's public results API only (store_at, RunFilter, full_metrics, equity_curve, tearsheet); every store query, parquet read, calculation and rendering runs through offloaded.

Request	Answer
ListRuns	RunList chunks of RunSummary, the finished runs RunFilter selects, oldest first. A run_ids left out selects every run; present and empty, none.
GetRun	One RunSummary: the stored summary, with its headline metrics as decimal strings.
GetMetrics	Metrics chunks: each of full_metrics' groups, the per-instrument PnL statistics and the probabilistic Sharpe ratio. An undefined value (none, NaN, infinite) is left out, never sent as zero.
GetSeries	Series chunks whose bytes, joined, are one Arrow IPC stream. EQUITY is the equity curve the metrics use, on the run's grid: columns ts_event, currency, total_equity. FILLS is the stored fills table as written, indexed by client_order_id; a run without fills sends an empty table with its columns.
GetTearsheet	Tearsheet chunks whose bytes, joined, are the UTF-8 HTML sbt2 report tearsheet writes.

• Benchmarks: a tearsheet is drawn against the strategy's default benchmark unless the request picks NONE, BUY_AND_HOLD (on its instrument_id, else the run's first instrument) or EQUAL_WEIGHT, as the CLI's --benchmark does; external files are the CLI's only. When the benchmark's strategy or prices are missing the request fails; the server never falls back to no benchmark.
• Tearsheet cache: each rendered tearsheet is kept in the run's folder, under tearsheets/, one file per benchmark choice and sbt2-server version, since a finished run never changes. It is rendered to a temporary file beside it and renamed into place, so a failed render leaves no file behind, and concurrent requests for the same tearsheet render it once.
• Errors: an unknown or malformed run id, a missing table or missing prices are NOT_FOUND; an unset series kind or a benchmark choice that doesn't fit (an instrument on anything but BUY_AND_HOLD, an instrument id that doesn't parse) INVALID_ARGUMENT; one record larger than an envelope RESOURCE_EXHAUSTED. Messages never name server paths.

Deployment

The server image is built from sbt2-backend/Containerfile, with sbt2-backend/ as its build context. sbt2-backend/compose.yaml runs it.

• The image: a multi-stage build. The first stage installs sbt2[server] from sbt2-backend/uv.lock (uv sync --locked --no-dev --extra server --no-editable) into a virtual environment, with uv copied in from its pinned image; the second copies that environment onto the same slim Python 3.14 image, without uv or build tools. It is built for linux/amd64 only, the one platform of the lock and of Nautech's nautilus wheels. The build context has no git history, so the version every member reports (and Welcome.server_version with it) comes from the build argument SBT2_VERSION, 0.0.0+local when it isn't given.
• At run time: the server runs as the non-root user sbt2 (uid 10001), bound to 0.0.0.0:8765 (SBT2_SERVER_HOST, SBT2_SERVER_PORT), with SBT2_DATA=/data and SBT2_CONFIG=/config, two folders the image creates and gives to sbt2, so an empty named volume mounted there is writable. SBT2_SERVER_TOKEN is never baked in: a container started without it exits with status 2, naming it.
• Health: the image's HEALTHCHECK calls GET /health with the image's Python, since a slim image has no curl: every 10 s, after a 30 s start period. HEALTHCHECK is part of the Docker image format only, and podman builds OCI images by default, so a podman build that should keep it passes --format docker; compose.yaml declares the same check itself.
• compose.yaml: one service, server, built from the Containerfile, with the named volumes data on /data and config on /config, the token from the deploying shell's SBT2_SERVER_TOKEN (compose refuses to start without it), port 8765 published, and restart: unless-stopped. No reverse proxy and no TLS: the server speaks plain ws:// and GET /health on the published port, so the token crosses the network in clear text unless the operator puts TLS in front of it.
• Runs execute inside the container, which bounds what an uploaded strategy can reach. A container has no systemd user session, so they use the default uncapped launcher, and the container engine limits memory: the operator sets the container's memory limit (mem_limit in compose, --memory on podman run).

Deploy from the repository root:

SBT2_SERVER_TOKEN=<token> SBT2_VERSION=<version> podman compose -f sbt2-backend/compose.yaml up -d --build

From sbt2-backend/, use podman compose up -d --build. Docker Compose accepts the same file and options. The data and config live in the compose project's volumes and survive down; down -v deletes them.

Key concepts

Full list of terms: Glossary.

• Run spec: one backtest as written: strategy, parameters, instruments, period, split and part, venue profile, capital, seed, equity interval. A spec file may contain lists and expands into many run specs.
• Resolved run spec: the run spec with every name replaced by its contents: instrument ids, asset profile, the part's dates, venue profile values, parameters with defaults filled in, warm-up. This is what gets hashed, so the same hash always means the same backtest request.
• RunOutput: the raw facts a run produced: equity series, fills, positions, carry ledger and status.
• run_id: a UUIDv7, unique per execution. Rerunning the same spec gives the same spec hash and a new run_id.
• Part: train, validation or test. See Spec.