Format Overview

Format Overview

A summary of the JAZMIN file format, version 1.0, for the curious and for anyone writing a reader in another language. The normative specification is the repository's format RFC draft, and the catalog's messages are defined in spec/jazmin.proto (Protocol Buffers).

Media type	application/vnd.jazmin
Extension	.jzm
Magic	JZM1 (4A 5A 4D 31) at offset 0 and in the last 4 bytes
Byte order	Little-endian for fixed-size fields

Layout

+---------------------------+  offset 0
| Preamble (64 bytes)       |  magic, flags, file id, salt, KDF iterations
+---------------------------+  offset 64
| Chunk parts               |  the rows, column by column
| Index sections            |  sorted index pages, trigram indexes
| Statistics sections       |  min / max / nulls per chunk and column
| Embedded-file sections    |  files in 256 KiB blocks
| Catalog sections          |  directories, column definitions, ...
| Key slots      (shared)   |
| Header                    |  tables, columns, metadata, keyring
| Signature      (shared)   |
+---------------------------+
| Trailer (44 bytes)        |  where the header, key slots and signature are; CRC; magic
+---------------------------+  end of file

Readers find everything through the trailer (at a fixed place: the end) and the offsets recorded in the catalog. Writing the catalog last is what lets a writer stream any number of rows with one chunk in memory.

Preamble flags: ENCRYPTED, PASSWORD, ACCESS (shared file), APPENDED. Unknown flag bits make readers refuse the file. A pre-release draft file starts with JZMN and is refused with a clear message.

Sections

Every section (a chunk part, an index page, the header...) is stored in a 16-byte envelope:

Field	Size	Meaning
codec	1	0 none, 1 Deflate (raw), 2 Brotli; 3 reserved for Zstandard
flags	1	bit 0: encrypted
reserved	2	0
raw_length	4	Length after decryption and decompression
payload_length	4	Length of the stored payload
crc32	4	CRC-32 of the stored payload

The payload is raw → compress → encrypt; a section that doesn't get smaller by compression is stored with codec 0. Readers check, in order: the CRC, the authentication tag, and that the decompressed length equals raw_length. Each section has a text identifier (such as 0/chunk/12/* or 0/index/account/sorted/page/3) that selects its key and is bound into its encryption.

Chunks

A chunk part stores one stream per column of its column group, each prefixed with its length so a reader can skip the columns it doesn't need:

stream = flags (encoding + HAS_NULLS) | null bitmap (if any nulls) | values of the non-null rows

Encoding	Column types	Body
plain	all	Each value as is
delta	int, datetime	The first value, then each difference from the previous, as zigzag varints
dictionary	string, decimal	The distinct values once, then one index per row
bitmap	bool	One bit per value
scaled float	float	Short decimals as an integer and a power of ten; others as 8-byte doubles

Writers choose per stream: delta for ints and dates when smaller, dictionary when at most half the values are distinct, scaled float when most values are short decimals. This is why sorted data and repeated values compress so well.

Catalog

The header (a Protocol Buffers message, encrypted in an encrypted file) holds the tables, their columns (name, type, nullability, description, attributes), metadata, sort order, index list, package settings and, in encrypted files, the keyring: three random secrets for the data, index and files key groups. Directories record each chunk's position, row count and partition; statistics record each chunk's min, max and null count per column.

Indexes

• Sorted: keys in sorted order with the rows that hold them, in pages of about 64 KiB under a small directory, so a lookup reads the directory and one page. With the index-deltas reader feature (compact indexes, 1.2), keys and row ids are stored as differences from the previous entry.
• Trigram: every three-character sequence of a string column, with the rows containing it, for contains and icontains.

Encryption

master key   = the key's 32 bytes, or PBKDF2-HMAC-SHA256(password, salt, iterations)
header key   = HKDF-SHA256(master, salt, "JAZMIN/1/header")
section key  = HKDF-SHA256(master, keyring[group], "JAZMIN/1/" || section id)
AAD          = file id || envelope bytes 0..7 || section id

Key text forms: jzk1- + base64url(key ‖ first 4 bytes of SHA-256(key)); access keys jza1- + base64url(secret ‖ owner fingerprint ‖ checksum). Shared files add partition and column-group secrets, key slots and an ECDSA P-256 signature: see Security Model.

Changing files

• Rewrites (update, compact) produce a new file with a new file id and salt, swapped in atomically.
• Appends add new sections, a small delta listing what changed (new chunks, deleted row ids), a new header and a new trailer at the end, and set the APPENDED flag. Existing bytes are never modified. A reader that finds an incomplete append at the end uses the last complete trailer before it.

Extending the format

• Optional information goes into new catalog fields; readers ignore fields they don't know.
• Anything a reader must understand (a codec, an encoding, an index format) is announced as a reader feature in the header, only when the file uses it. Readers refuse files with unknown reader features, naming them; writers refuse to modify files with unknown writer features. Defined so far: index-deltas.
• Reserved names: zstd, bitpacked, byte-stream-split, roaring, bloom-filter, datetime-us, datetime-ns.
• The magic JZM1 changes only if the container itself changes incompatibly.

Conformance

The repository's spec/fixtures folder holds files written by each library (js-*.jzm, dotnet-*.jzm: plain, key, password, Brotli, shared, appended, several tables, embedded files, paged indexes), with their test keys (public by design) and expected contents. Each library's tests read the other's files, and damaged files that must be refused. The browser writer is checked to write exactly the bytes the JavaScript library writes, given the same inputs and random bytes.