Why JAZMIN

Why JAZMIN

JSON is the default way to move lists of records around. It is readable and works everywhere, but it gets expensive as data grows: every field name is repeated in every record, nothing is compressed, nothing is indexed, and to find one record you parse them all. JAZMIN keeps JSON's simplicity (an array of records in, an array of records out) and fixes those costs.

When to use JAZMIN

• Large exports, archives, caches and hand-overs between systems.
• One large file processed a section at a time, such as one PDF per client statement: see Recipes.
• Files you query: one record, or a filtered subset, without loading everything.
• Data that must be encrypted at rest, without a separate encryption step.
• One file shared by many people who may each see only part of it: see Access Control.
• Replacing a "JSON + gzip + encrypt" pipeline with one format and one call.

When to keep something else

Use	Instead of JAZMIN, when
JSON	Small API bodies that browsers or third parties read directly; configuration; anything people edit by hand; payloads of a few hundred bytes (a JAZMIN file has about 300 bytes of fixed overhead)
CSV	A spreadsheet user must open the file directly (JAZMIN converts to CSV in one call)
Parquet	Analytics that aggregate whole columns across billions of rows in a data lake
A database	Many writers changing the same records concurrently, or ad-hoc joins across many tables

Compared at a glance

	JSON	gzipped JSON	CSV	JAZMIN
Size (200,000 records)	30.5 MB	3.3 MB	about JSON's	1.3 MB (deflate), 0.8 MB (brotli)
Find one record	parse everything	decompress and parse everything	parse everything	reads one chunk or index page
Types	strings, numbers, bools	same	text only	8 types, exact decimals, UTC dates
Schema	none	none	header only	names, types, nullability, descriptions, attributes
Encryption	separate step	separate step	separate step	built in, AES-256-GCM, authenticated
Per-person access to parts	no	no	no	yes, rows and columns per key
Streaming write and read	with special parsers	with special parsers	yes	yes, one chunk in memory
Attachments	base64 inside	base64 inside	no	embedded files, read by byte range

Measured

On 200,000 customer records with 8 fields, Intel i7-12700H, Windows 11 (best of 3 runs after a warm-up). Full tables and how to re-run them: Performance and Benchmarks.

.NET 10	Newtonsoft.Json	System.Text.Json	JAZMIN
File size	30,529 KB		1,297 KB
Find one record by id (open the file → result)	265 ms	104 ms	1.4 ms
Filter Country == "NA" && Age > 80	276 ms		13 ms
Memory allocated for one lookup	196 MB		1.1 MB
Deserialize every record	211 ms	91 ms	32 ms
Serialize every record	103 ms	57 ms	42 ms

Node.js 24	JSON	JAZMIN
File size	31,307 KB	1,292 KB
Find one record by id	109 ms	1.7 ms
Read every record	109 ms	45 ms
Text search (contains)	112 ms	21 ms
Memory to answer a lookup	42 MB	under 0.1 MB

And at a size JSON can't handle in memory: 1,000,000 rows × 300 columns is 6.3 GB of JSON and 643 MB of JAZMIN. Looking up one id took 52 s by streaming the JSON in Node, and 0.03 s in JAZMIN.

See it on your own machine

This example writes the same 100,000 records as JSON, gzipped JSON and JAZMIN, and compares the sizes and the cost of finding one record.

import { gzipSync } from 'node:zlib';
import { open, write } from '@smithsoft-studios/jazmin';

const countries = ['ZA', 'NA', 'BW', 'KE', 'NG', 'GB', 'US', 'DE'];
const rows = Array.from({ length: 100_000 }, (_, i) => ({
  id: i + 1,
  name: `Customer ${i + 1}`,
  country: countries[i % countries.length],
  age: 18 + (i % 70),
  balance: Math.round(((i * 7919) % 100000) * 1.37) / 100,
  active: i % 3 !== 0,
}));

const json = Buffer.from(JSON.stringify(rows));
const jzm = write(null, rows, { sortedBy: ['id'] }); // null: return the bytes instead of writing a file
const kb = (n) => `${Math.round(n / 1024).toLocaleString('en-US')} KB`;
console.log('JSON        ', kb(json.length));
console.log('JSON + gzip ', kb(gzipSync(json).length));
console.log('JAZMIN      ', kb(jzm.length));

// Finding one record: JSON must parse everything. The file is sorted by id, so chunk statistics find the one chunk.
const reader = open(jzm);
const { rows: found, chunksRead, indexPagesRead } = reader.explain({ id: 76543 }, { analyze: true });
console.log(`found ${found} row, reading ${chunksRead} of ${reader.chunkCount} chunks and ${indexPagesRead} index sections`);

JSON         9,002 KB
JSON + gzip  1,237 KB
JAZMIN       566 KB
found 1 row, reading 1 of 25 chunks and 0 index sections

About 16× smaller than the JSON and less than half the gzipped JSON, and one chunk read out of 25. How much smaller depends on the data: repeated values (countries, statuses, dates in order) compress very well; unique random text (names, references) less so. Sorting by the column you look up (sortedBy) also means no index is needed for it.

What you give up

• Not human-readable. Use toJSON() or the command-line tool to look inside, or the viewer to browse a file.
• Not for small messages. The fixed overhead (about 300 bytes) is more than many API payloads.
• Files are written, then read. Changes are made with update or append, which is fast, but JAZMIN is not a database with many concurrent writers.