Replays

[mworzala]

Goals

• Format
  • Support very long runs (some form of chunking/streaming)
  • Support storing metadata in the recording (like anticheat flags)
  • Allow forwards and backwards playback, as well as arbitrary seeking.
• Race against a ghost of a previous recording of a map completion in a new run (yours or another persons, like the top time)
• View any recording as a staff member
  • Multiplayer viewing
  • Playback controls
  • Possibly allow anybody in spectator mode to review recordings

Recording

Recordings have the following structure

Recording (header + metadata + index)
 > Segment (list of chunks, only present during recording until compaction)
  > Chunk (list of ticks)
   > Tick (single game tick, may be skipped when empty)

Within a chunk there are two potential pieces of data:

• An event stream per tick within the chunk.
• A snapshot (optional) containing the full state of the world.

When recording, we write segments every {x} time (or based on size, non-empty tick count, etc), and update the index whenever we write a segment.

File format

Name	Type	Note
Header	256 bytes
Metadata	variable	NBT compound containing arbitrary metadata
Index	variable	Chunk index for scanning
Chunk 0..N	variable	Each chunk is separately compressed, see index.

Header

Name	Type	Note
Magic number	4 bytes	TBD
Version	1 byte	Format version, starting at 1
Flags	2 bytes	unused
World ID	16 bytes (uuid)	An arbitrary identifier for the world
World version	16 bytes	An arbitrary identifier for the world version
Timestamp	8 bytes	Unix milliseconds when the recording was started
Dictionary ID	2 bytes	The dictionary in use, or 0 if no dictionary
Metadata length	4 bytes	Starts at byte offset 256
Index length	4 bytes	Starts at offset 256 + {metadata length}
Tick count	4 bytes	Total ticks in the file
Chunk count	4 bytes	Total chunks in the file
Padding	-	Padding to 256 bytes

Index

The index is Chunk count (see header) sequential entries with the following format. It is sorted ascending by tick.

Name	Type	Note
Start tick	var int
Tick count	var int	Number of ticks in this chunk
Flags	1 byte	bit 0: has snapshot
Byte offset	8 bytes	Start offset in bytes into the file for compacted recordings. For segmented recordings, contains `(segment offset << 32)
Compressed length	var int
Uncompressed length	var int

Chunk

The chunk is an optional snapshot (present if index says so) followed by an event stream. Each chunk is separately compressed with zstd.

Snapshot entry formats are defined by their implementation but handle things like:

• Absolute set of edited blocks.
• Absolute set of active entities and their positions.

The event stream has the following format (repeated for each tick in the chunk)

Name	Type	Note
Tick	var int	The tick (relative to the chunk start) of the following events. Note: empty ticks are skipped so this number may not increase by 1.
Event count	2 bytes	Number of events following (not variable since its appended later)

The format of each event is defined by its implementation, for example:

• Delta move
• Spawn entity
• Set block
• Swing hand

Partial recording format

A recording may represent a very long period over many distinct play sessions, so it's not feasible to write the entire thing at once. Instead, we store it as a separate metadata file, and N segments, each of which contains some number of chunks.

The metadata file contains the header, metadata, index sections from the main format. It is always overwritten completely when being updated.

Segment files are append-only. When the server has accumulated some number of chunks or a player leaves, it writes them as a segment.

After a run is completed, the metadata file and segments should all be compacted into a single file. That file can

• use a higher compression ratio
• append extra metadata from analysis of the recording
• reduce read cost by avoiding lots of tiny expensive s3 reads.

Playback

TODO write some more here

Some notes:

• This implementation intends to make heavy use of range reads of the recording files
  • To start: Read first 256kb of recording to capture header, metadata, index (probably, can re-read if those are bigger)
  • When seeking, index tells us where to go for the chunks we care about (including the snapshot nearest)
• Seeking can be accomplished by skipping to the closest snapshot and replaying events from there.

Notes

• Tick rate is not considered variable (can change later if needed).

Open questions

• !! Event data versioning
• !! Game data versioning
  • Having a recording start on version x but be later appended on version y is tricky.
  • Probably need to drop data version down to the chunk level.
• Limits
  • How long/what recordings do we retain?
  • Different for hypercube vs not?
• For staff viewing
  • In the main world? Or a separate world?
  • How to allow for multiplayer viewing
• Stetch: it would be interesting to be able to do a shadow play-like recording where we keep the last 5m of a players gameplay in a ring buffer always and save it when they get reported