Core Concepts

Core Concepts

Five ideas explain how JAZMIN stays small and reads so little: columns, chunks, statistics, indexes and the catalog. Once you know them, you can predict what a query will cost.

What is inside a file

flowchart LR
  subgraph File[".jzm file"]
    direction LR
    D["Data: chunks of rows<br/>(4,096 rows each by default)<br/>stored column by column,<br/>compressed, optionally encrypted"]
    C["Catalog (at the end)<br/>header: tables and columns<br/>metadata: your key/values<br/>directories: where each chunk is,<br/>with min/max per column<br/>indexes: sorted and trigram<br/>keyring: secrets for sections"]
  end
  D --- C

Loading

• Chunks. Rows are cut into chunks of 4,096 (chunkRows). Each chunk is compressed, and optionally encrypted, on its own. To read row 765,432, the library reads only the chunk that holds it.
• Columns. Inside a chunk, each column is stored separately, as typed binary values. A query decodes only the columns it uses: on a 300-column table, asking for 3 columns decodes 3.
• Statistics. For every chunk and column, the catalog records the minimum, maximum and null count. A filter such as { amount: { gt: 1000 } } skips every chunk whose maximum is 1,000 or less, without reading it.
• Indexes. Optional, per column. A sorted index maps values to rows (equality, ranges, prefixes); a trigram index finds text that contains a fragment. Large sorted indexes are stored in pages of about 64 KB, so a lookup reads a small directory and one page.
• The catalog sits at the end of the file. That lets a writer stream millions of rows to disk with only one chunk in memory, and write the directories and indexes last. Opening a file reads the catalog only.

Column types

Type	JavaScript value	.NET value	Notes
bool	boolean	bool
int	number (bigint beyond ±2^53)	long	64-bit signed
float	number	double	64-bit IEEE 754
decimal	string, e.g. "1520.75"	string in rows; decimal on your classes	Exact, never rounded; 12.5 equals 12.50 in filters
string	string	string	UTF-8
datetime	Date	DateTime (UTC)	Millisecond precision
binary	Buffer	byte[]
json	any JSON value	JsonNode	Nested objects and arrays, any depth

Every column also has nullable (default true), and optionally a description and free-form attributes, such as { "unit": "kPa" }. They travel with the file, so the next reader knows what the data means.

import { open, write } from '@smithsoft-studios/jazmin';

const bytes = write(null, [{
  ok: true, count: 42, ratio: 0.25, price: '19.990', name: 'Ann', at: new Date('2026-01-02T03:04:05Z'),
  raw: Buffer.from([1, 2, 3]), extra: { tags: ['a', 'b'], address: { city: 'Cape Town' } },
}], {
  columns: [
    { name: 'ok', type: 'bool' },
    { name: 'count', type: 'int' },
    { name: 'ratio', type: 'float' },
    { name: 'price', type: 'decimal', description: 'Unit price', attributes: { currency: 'ZAR' } },
    { name: 'name', type: 'string' },
    { name: 'at', type: 'datetime' },
    { name: 'raw', type: 'binary' },
    { name: 'extra', type: 'json' },
  ],
});

const reader = open(bytes);
const row = reader.get(0);
for (const c of reader.columns) {
  const value = row[c.name];
  const kind = value instanceof Date ? 'Date' : Buffer.isBuffer(value) ? 'Buffer' : typeof value;
  console.log(c.type.padEnd(8), kind.padEnd(7), JSON.stringify(value));
}
console.log(reader.columns.find((c) => c.name === 'price'));

bool     boolean true
int      number  42
float    number  0.25
decimal  string  "19.990"
string   string  "Ann"
datetime Date    "2026-01-02T03:04:05.000Z"
binary   Buffer  {"type":"Buffer","data":[1,2,3]}
json     object  {"tags":["a","b"],"address":{"city":"Cape Town"}}
{
  name: 'price',
  type: 'decimal',
  nullable: true,
  description: 'Unit price',
  attributes: { currency: 'ZAR' }
}

Note that the decimal kept its trailing zero ("19.990"): decimals are stored exactly as given.

Chunks and statistics at work

The next example writes 100,000 transactions in date order (sortedBy), with an index on account and a trigram index on memo, then asks explain() how three queries will run. With { analyze: true }, explain() also runs the query and reports what it read.

import { open, write } from '@smithsoft-studios/jazmin';

const start = Date.UTC(2026, 0, 1);
write('transactions.jzm', Array.from({ length: 100_000 }, (_, i) => ({
  at: new Date(start + i * 300_000), // one every 5 minutes
  account: `ACC-${String((i * 7919) % 20000).padStart(5, '0')}`, // 20,000 accounts, 5 rows each
  amount: ((i * 7919) % 50000) / 100,
  memo: i % 25000 === 0 ? 'refund for order' : 'card payment',
})), {
  columns: [
    { name: 'at', type: 'datetime', nullable: false },
    { name: 'account', type: 'string', index: 'sorted' },
    { name: 'amount', type: 'float' },
    { name: 'memo', type: 'string', index: 'trigram' },
  ],
  sortedBy: ['at'],
});

const show = (label, filter, options = {}) => {
  const reader = open('transactions.jzm'); // a fresh reader, so the cost is the query's own
  const { ms, ...plan } = reader.explain(filter, { ...options, analyze: true });
  console.log(label.padEnd(30), JSON.stringify(plan));
  reader.close();
};
const reader = open('transactions.jzm');
console.log(`${reader.rowCount} rows in ${reader.chunkCount} chunks; indexes:`, reader.indexes);
reader.close();
show('one day (sort order)', { at: { gte: new Date('2026-03-01'), lt: new Date('2026-03-02') } });
show('one account (sorted index)', { account: 'ACC-00042' });
show('memo contains (trigram)', { memo: { contains: 'refund' } });
show('amount > 499 (no index)', { amount: { gt: 499 } });
show('same, select amount', { amount: { gt: 499 } }, { select: ['amount'] });

100000 rows in 25 chunks; indexes: [
  { column: 'account', kind: 'sorted' },
  { column: 'memo', kind: 'trigram' }
]
one day (sort order)           {"strategy":"scan","chunks":25,"chunksSkipped":24,"rows":288,"bytesRead":24264,"chunksRead":1,"indexPagesRead":0,"columnsDecoded":4}
one account (sorted index)     {"strategy":"index","candidateRows":5,"rows":5,"bytesRead":135086,"chunksRead":5,"indexPagesRead":2,"columnsDecoded":20}
memo contains (trigram)        {"strategy":"index","candidateRows":4,"rows":4,"bytesRead":97811,"chunksRead":4,"indexPagesRead":1,"columnsDecoded":16}
amount > 499 (no index)        {"strategy":"scan","chunks":25,"chunksSkipped":0,"rows":198,"bytesRead":587982,"chunksRead":25,"indexPagesRead":0,"columnsDecoded":100}
same, select amount            {"strategy":"scan","chunks":25,"chunksSkipped":0,"rows":198,"bytesRead":587982,"chunksRead":25,"indexPagesRead":0,"columnsDecoded":25}

What the lines say:

• One day: the file is sorted by at, so chunk statistics rule out 24 of the 25 chunks. No index is needed.
• One account: the sorted index names 5 rows, and only the 5 chunks holding them are read.
• Text search: the trigram index finds the 4 matching memos, and only their 4 chunks are read.
• No index, no sort order: amounts are spread through every chunk, so every chunk is read and every column of every chunk is decoded (4 × 25 = 100 column streams).
• The same with select: ['amount']: the chunks are still read, but only one column is decoded in each (25 column streams). Decoding is most of a query's work, so on wide tables select is the biggest single saving.

The fix for a query that reads too much is almost always one of three things: sort the file by the column you filter on, add an index, or use smaller chunks. Tuning Speed and Memory shows how to choose, and reader.advise() suggests a layout from the statistics alone.

Indexes

Index	Speeds up	Allowed on
sorted	eq, in, gt, gte, lt, lte, startsWith, isNull	bool, int, float, decimal, string, datetime
trigram	contains, icontains	string

• Cost: indexes make the file bigger (about 20–30% for three indexed columns) and writing slower. Add them only to columns you search on.
• No index needed on the sort column: a sorted index on the first sortedBy column is not written, because chunk statistics already find its values.
• Indexes only narrow the scan. The reader works out which chunks a scan would read, and uses an index only when it reads less than that. Results are the same either way.
• Compact indexes (compactIndexes: true, New in 1.2) store each key as the difference from the previous one: much smaller indexes on whole numbers and dates. Readers before 1.2 can't read them, so turn them on once every reader is on 1.2 (see Versions and Compatibility).

Encryption

Pass a key (32 random bytes, as jzk1-... text with a checksum) or a password (turned into a key with PBKDF2, 600,000 iterations). The key unlocks the catalog; the catalog holds secrets that, combined with the key, give every chunk and index its own key. Everything is authenticated, so any change to the file is detected. Encryption and Keys has the details.

Several things in one file

A file holds one table by default. It can also hold:

• several tables that share keys and access rules: Several Tables;
• embedded files, such as PDFs, images or a whole web page: Embedded Files and Documents;
• metadata, your own key/value information (source, export date, owner): metadata: { source: 'crm' };
• access rules, so each key sees only its rows and columns: Access Control.

Reading never loads the whole file

Operation	Reads
open()	The catalog only
get(rowId)	The chunk that holds the row
find(filter)	Index pages that help, then only chunks that can match, and only the columns used
find(filter, { select })	As above, decoding only the selected columns (and the filter's)
count(filter)	Often nothing but the index or chunk statistics
rows({ offset, limit })	Skips whole chunks before the offset without reading them

find() and rows() are generators: one chunk is held in memory at a time, however many rows match.