Data

Data

Raw files and the catalog. One pipeline, grouped by stage: sources (the source adapters, see Data sources), download (network → raw files), ingest (raw files → catalog) and catalog (the catalog's reader, and the writer only ingest uses). download and ingest never import each other.

Both take a request built on one DayRange: symbols over an inclusive range of UTC days, and the data types by nautilus type name (empty means every type the source serves). DayRange.plan(source) yields each (symbol, data type, day) the source should have, skipping its known gaps, and both stages walk that plan. DownloadRequest adds the instrument snapshot's date, IngestRequest the re-ingest switch. Both return a Tally of their per-file or per-day results, in plan order, filtered by outcome with having.

download

Network in, raw files out. No parsing.

Each data source (an exchange, a vendor, a public dump) is a source adapter that knows where its files live and how to fetch them. download itself is source-agnostic: it schedules the fetches and guarantees the behaviour below. Source adapters are listed on Data sources.

A source can provide any of:

Data	Needed for
Trades	trade-driven strategies, and bars built from trades by the engine
1-minute candles	bars built from candles by the engine, for runs with bars = "candles" (see Spec)
L2 order book	book strategies, realistic fills
Instrument specs	real tick size, precision, margin. Fetched with nautilus's adapter for the source where one exists. Saved as dated snapshots, never overwritten.
Carry data (e.g. funding rates)	carry settlement by nautilus's simulated venue
Reference prices (e.g. mark price)	carry, margin, valuation and benchmarks

• Files keep the source's own names under the root's raw/{source}/….
• Where nautilus's adapter for the source already fetches something (e.g. instruments, funding history), download uses it and saves the returned objects as JSON (to_dict). That is still raw: nothing is merged, fixed or filtered.
• Atomic and idempotent. Each file is written to .part and renamed on completion, with the size checked against content-length. Re-runs skip files that are already complete.
• A day the source has no file for (404) is reported as missing, never treated as fatal.
• Downloads run concurrently, with retries and backoff.

ingest

Raw files in, nautilus catalog out. No network. The only module that writes to the catalog.

Output	From	Notes
Instruments	a pinned, dated instrument-spec snapshot	Already a nautilus instrument of the asset class's type (e.g. CryptoPerpetual) when the source has a nautilus instrument provider. Fails loudly if the data doesn't fit the spec.
OrderBookDelta	L2 files	Streamed, never loaded whole.
TradeTick	trade files	Sorted, with exact nanosecond timestamps.
Bar	candle files	1-minute EXTERNAL bars, one bar type per instrument ({instrument}-1-MINUTE-LAST-EXTERNAL), each stamped 1 ns before its close.
Carry and reference data	carry files	e.g. FundingRateUpdate, MarkPriceUpdate. Streamed with the run like any other data; nautilus's simulated venue settles funding from them.

Only 1-minute candles are stored; nautilus builds every declared bar at run time. A run's bars key (see Spec) picks what from. From trades, nautilus aggregates INTERNAL bar types from the streamed trades. From candles, it aggregates composite bar types such as …-1-HOUR-LAST-INTERNAL@1-MINUTE-EXTERNAL from the stored candles. Either way bar edges follow nautilus's own rules, and sbt2 has no resampling code. Coarser candles are never stored: 1 minute is the finest interval sources serve, every whole-minute bar up to a day is built from it, and a month of BTCUSDT candles is 2.5 MB.

Each source adapter supplies the parsers that turn its raw format into these types. Format-specific rules (timestamps, snapshot handling, quirks) live on the source's page under Data sources.

L2 conversion rules (all sources)

• a snapshot → CLEAR + ADDs, flagged F_SNAPSHOT
• level size 0 → DELETE, otherwise UPDATE
• last delta of every source message → F_LAST, so nautilus knows the batch is complete
• ts_event = the exchange's matching-engine time, ts_init = the time the update was published
• trimmed to the UTC day

Instrument-spec snapshots

Every spec fetch is kept with its date. Ingest writes the snapshot it used into the catalog as the nautilus instrument, with ts_init set to the snapshot date, so the catalog's instrument is the record of which snapshot was used. If a newer snapshot differs from the instrument in the catalog, ingest fails loudly unless a re-ingest is requested, so old runs stay reproducible.

• The snapshot used is the newest one in raw/ for the symbol.
• Specs are compared without their timestamps: a newer snapshot with the same spec keeps the catalog's instrument, and its older date.
• A re-ingest (sbt2 ingest --reingest) removes the instrument and all its day files, then writes the new instrument and the requested days. Price and size precision shape every tick, so days built with the old spec don't stay.

Writing

• Nautilus's public writers. v2's catalog is Rust; its typed writers (write_trade_ticks, write_order_book_deltas, write_mark_price_updates, write_bars, write_instruments, …) take explicit start / end bounds, which is all the day naming below needs. No private internals.
• Funding rates are the exception. v2 has no Python writer for FundingRateUpdate, so ingest writes that file itself, taking the schema from nautilus (serialization.get_arrow_schema_bytes(FundingRateUpdate)) rather than hard-coding it, in nautilus's layout (data/funding_rates/{instrument_id}/). The schema already changed between rc5 and rc6 (timestamp columns, a new identifier column), and again in 2.0.0rc6.dev20260930: the instrument moved from an instrument_id column into the schema metadata, as for the other types, and the reader fails on a file without it. Funding files written before that read only with the older nightly; delete data/funding_rates/ and pre-flight fetches them again. A round-trip test checks that nautilus streams it back unchanged (see Quality). It switches to a public writer once one exists.
• One file per (instrument, data type, UTC day), named with the whole day's bounds ([day start, day end − 1 ns]), not the first and last timestamps in it. Nautilus files bars under their bar type, not the instrument id (data/bars/{bar_type}/), so ingest, coverage, status and re-ingest name an instrument's candles by its candle bar type. Nautilus reads a catalog's coverage from file names, so the days then join into one continuous interval and a missing day shows up as an exact gap (see catalog below).
• Empty days get a zero-row file. A day the source serves but that holds no rows (e.g. no trades) is still written, as a file with nautilus's schema and no rows, named with the day's bounds. Nautilus counts it as covered, streams across it and returns no rows for it, so the day is not a gap. The typed writers write no file for an empty list, so ingest writes this file itself with pyarrow, like funding. Pinned by the characterization test test_empty_day.py (see Quality). The file must carry the schema metadata nautilus's writers add (instrument_id, bar_type for bars, and price_precision / size_precision for the types that have them): without it, a query over a zero-row file with no non-empty neighbour fails with Missing metadata key: instrument_id.
• Day-atomic. A day file appears only once complete, so a crash never leaves a partial day, and re-runs skip days whose file already exists. Nautilus's typed writers already write under <file>#N and rename when done, and its reader ignores such names (pinned by test_atomic_write.py, see Quality). The files ingest writes with pyarrow follow the same pattern with a #sbt2 suffix.
• One writer call per day, with the whole day in memory. On BTCUSDT's busiest January 2025 day (8.8M trades) ingest peaks at about 6.5 GB: 3.7 GB of TradeTick objects and 2.6 GB inside write_trade_ticks. Chunked writes, whose boundaries never split a timestamp, come with L2 (M6), when a day no longer fits.
• Every record's ts_init must fall inside its day's bounds, else ingest raises OutsideDayError and writes nothing for that day.
• A day without a raw file is reported as missing and skipped; known-gap days are skipped silently.

Verification

A built-in check replays a day's deltas through a nautilus OrderBook and compares the result with the next snapshot.

catalog

The catalog on disk. A package of a reader, which everyone reads the catalog through, and a writer, which only ingest uses. How nautilus lays files out is its internal detail.

Stored types

Everything the catalog knows about one data type lives in one stored type: a subclass of the abstract StoredType (Trades, MarkPrices, Funding, Candles), listed once in the package's registry tuple. A stored type holds its nautilus type, its directory under data/, the name an instrument's files go under (the candle bar type for Candles, the instrument id otherwise), the precisions its zero-row files carry as metadata, how a day is written (nautilus's typed writer, or its own table for Funding) and the frame its records load as. The writer and the reader look a data type up in the registry and hold no per-type code; writing or loading a data type sbt2 doesn't store raises UnstoredDataTypeError. Coverage takes any nautilus data type, filing one sbt2 doesn't store under its instrument id, so a run that streams such a type fails at the source, which can't serve it. A new data type is one new subclass and one registry entry.

There is no manifest. Nautilus's catalog answers everything from its file names and stored instruments, without reading any data:

Question	Answered by	Used by
Is the data there for every trading day of a window?	catalog.get_missing_intervals_for_request(start, end, data_cls, instrument_id), minus the source's known-gap days	Run coverage check, sbt2 data status
Does each instrument's type match the run's asset class?	catalog.instruments(instrument_ids=…)	Run pre-flight
Which instrument-spec snapshot was used?	the instrument stored in the catalog (ts_init = snapshot date)	ingest

Notebook loaders

Catalog.frame(instrument_id, data_type, window) loads an instrument's data of one stored type over a window (a half-open Window(start, end) of tz-aware datetimes; Window.of_days(first, last) covers whole UTC days, both included) as a pandas DataFrame indexed by ts_event. Its columns come from the stored type: trades as price, size, side, trade_id, mark prices as price, funding as rate, interval, candles as open, high, low, close, volume. Event-driven L2 backtests are slow, so ideas are cheaper to explore in pandas first.