File Format

File Format

The VTX file format is a self-describing binary format built for speed, streaming, and long-term data durability. This page is the wiki-level overview. For the bit-exact layout see docs/FILE_FORMAT.md in the repo.

File structure

A VTX file has four sections.

Header

Format versioning, schema versioning, and source metadata. A reader can identify what it is looking at and whether it has a compatible schema immediately, before parsing any frames. See Stability for how versioning works and what changes are considered breaking.

The magic bytes identify the serialisation backend:

• VTXP for Protocol Buffers.
• VTXF for FlatBuffers.

VTX::OpenReplayFile auto-detects the backend from these magic bytes. Integrators do not need to know in advance which one was used.

Embedded schema

The complete property-to-name-and-type mapping for this file. The schema acts as a self-contained decoder. Property ID 1 maps to Health, property ID 2 maps to Position, and so on.

Embedding the schema decouples data durability from the software development lifecycle. A file recorded today can be read later without needing the original application code or an external config: the decoder information travels with the data.

Chunks

Frame data is segmented into compressed chunks. Each chunk holds multiple frames, serialised with the chosen binary format (Protobuf or FlatBuffers) and compressed with Zstd.

Frames are stored as complete per-frame blobs, not as inter-frame deltas. Each frame in a chunk is a full state snapshot that can be decoded independently. The reader exposes raw per-frame bytes directly (GetRawFrameBytes(frame_index)), which would not be possible if frames were delta-encoded.

The Diff Engine (see SDK Architecture) operates on decoded frame pairs at read-time to drive playback and spawn/destroy/update logic. It is a runtime facility, not a storage format.

Chunks are the unit of streaming, compression, and on-demand download.

Footer / time index

The timeline metadata is separated from the frame data. The footer maps game timestamps (and frame indices) to file byte offsets.

Scrubbing a timeline binary-searches the footer to find the chunk that contains the target frame, decompresses that one chunk, and returns. Cost: O(log n) to find the chunk plus one chunk decompress. Sub-millisecond when the target chunk is warm in the reader's cache; tens of milliseconds on a cold seek while the chunk decompresses. It is not literally O(1): the "instant" experience is the combination of a tiny search + a single decompress, not a zero-cost lookup.

If you need to bound worst-case scrub latency, it is bounded by your chunk size and decompression speed. See Performance and File Format, Chunk size.

Encoding, endianness, and portability

All multi-byte integers and floats are stored little-endian, matching the native layout of the x86 / x64 / ARM64 targets the SDK builds on. The SDK does not currently include byte-swap shims for big-endian hosts; if you need to target such a platform, file an issue.

The on-disk format is fixed-endianness and stable across platforms. A file produced on Windows and a file produced on Linux are interchangeable: the same reader on either platform consumes either file identically.

Serialisation backends

The VTX SDK is not locked to a single binary serialisation format. The writer, reader, and differ all support both Protocol Buffers and FlatBuffers out of the box. You pick per file; the file announces the choice in its magic bytes.

The architecture is extensible. Another binary format can be plugged in by implementing the serialisation interface (and a matching diff implementation). Schemaless formats (Cereal, Bits) were evaluated and rejected: the diffing system requires files to be self-describing.

Protobuf vs. FlatBuffers

	Protocol Buffers	FlatBuffers
File size	Smaller (varint encoding, efficient sparse data)	Larger (vtable overhead for nested structures)
Data-access cost	Parse-on-read	Lower (in-place reads against the decompressed buffer; "zero-copy" within the chunk)
Diff performance	Standard	Faster
Best for	Storage, network transmission, archival	Real-time playback where diff speed is critical
Language support	Broad (C++, Python, Go, Rust, Java, JS, ...)	Broad (C++, Python, Go, Rust, Java, JS, ...)
Schema evolution	Excellent (unknown fields safely ignored)	Good (vtable-based field access)

Note: "zero-copy" for FlatBuffers refers to access against the decompressed chunk in memory. Zstd still has to run on the way in. On a cold seek, both backends pay one chunk-decompress; FlatBuffers then avoids a second parse pass, Protobuf does not.

In practice: Protobuf produces smaller files, FlatBuffers is faster at diffing and data access. Storage-constrained or network-heavy applications benefit from Protobuf. Latency-sensitive playback applications may prefer FlatBuffers. See Performance, Choosing a serialisation backend for decision guidance.

Compression

Chunks are compressed with Zstd (Zstandard). Strong compression ratio with very fast decompression. This matters for:

• Reducing file sizes for storage and network transfer.
• Fast seeking. Only the target chunk needs decompressing, not the whole file.
• Operating on memory-constrained platforms.

Compression is on by default (WriterFacadeConfig::use_compression = true). You can disable it when the writing pipeline is CPU-starved or when disk space is cheaper than CPU cycles.

Why not Unreal's FArchive?

Legacy replay tooling at Zenos used Unreal's FArchive serialisation, which tied replay files to a specific Unreal Engine version. Moving to standard binary formats (Protobuf / FlatBuffers):

• Removes the UE version dependency.
• Makes replay files readable outside the engine (external tools, web viewers, Python notebooks).
• Ensures a stable long-term format not subject to engine changes.
• Makes the data engine-agnostic.
• Lets SDK users pick the serialisation format that best fits their use case.

Chunk size

Chunk boundaries are decided by the writer and tuned for the balance between:

• Compression efficiency. Larger chunks compress better.
• Seek responsiveness. Smaller chunks mean less data to decompress per seek.
• Memory budget, especially on constrained platforms.

Writer defaults:

• chunk_max_frames: 1000 (chunk rolls at 1000 frames).
• chunk_max_bytes: 10 MB (chunk rolls at 10 MB, whichever limit hits first).

Both are configurable on WriterFacadeConfig. See Performance, Cache-window sizing for the reader-side cache that pairs with chunk layout.

Streaming vs. storage

The same format works in two modes:

• Streaming: chunks are written and flushed live; downstream consumers (broadcast overlays, analytics pipelines) read them as they land. Low latency.
• Storage / replay: full matches persisted for later scrubbing, analysis, or training. Random access by frame or timestamp goes through the footer index.

A tool written once reads either.