dd-json

Deduplicated JSON — a JSON superset that compresses repetitive data by replacing repeated values with short @ references.

{"name":"Alice","friends":[{"name":"Bob"},{"name":"Alice"}]}

{"name":"Alice","friends":[{@0:"Bob"},{@0:@1}]}

The encoder builds an implicit scope as it walks the data. Each unique value gets an auto-incrementing index on first encounter. When the same value appears again, it's replaced with @ followed by that index in base-36. The decoder maintains the same table as it parses — no scope header, no byte offsets, no second pass.

Why not just gzip/zstd the JSON?

You can — and you can gzip a .ddj file too (they stack). But dd-json solves a different problem:

• Still text — you can read it, grep it, diff it, pipe it, pretty-print it with syntax highlighting. Compressed JSON is an opaque blob.
• No decompression step — the decoder builds JavaScript objects directly as it parses. zstd+JSON requires decompress → full JSON string in memory → parse. dd-json skips the intermediate string.
• Streamable — single-pass, top-to-bottom. A decoder can start building objects before the full input arrives.
• Inspectable — cat data.ddj | ddj format gives you colored, indented output. Try that with data.json.zst.

If you just want smallest-possible cold storage, use zstd. dd-json is for when JSON is your interface format — APIs, configs, build manifests, caches — and you want it smaller without losing the properties that make JSON useful.

Format

dd-json is a strict superset of JSON. Every valid JSON document is valid dd-json. The only extension is @ references — strings, numbers, objects, and arrays use standard JSON syntax.

• @ references — @0, @1, ..., @a (10), @z (35), @10 (36) — base-36 scope index
• Whitespace-tolerant — the decoder skips whitespace and ANSI escape codes, so pretty-printed and syntax-highlighted output round-trips cleanly

Implicit scope

Both encoder and decoder maintain a scope table — an ordered list of values, indexed from 0. The core rule is simple:

Inline value → gets a slot. @ reference → no slot. null, true, false → no slot.

That's the entire protocol. Here's what that means in practice:

Gets a scope slot (advances the index):

• Strings written inline
• Numbers written inline
• Objects {...} written inline
• Arrays [...] written inline

Does NOT get a scope slot:

• @N references (they look up an existing slot, they don't create one)
• null, true, false (always written as literals, never referenced)

Ordering rule: Children get their slots before their parent. A string inside an object gets a lower index than the object itself. This is natural depth-first order — values are added to the scope as they finish parsing.

Why this matters: The encoder and decoder must agree on exactly which values occupy which slots. If the encoder writes a value inline — even a repeat it chose not to deduplicate — that burns a slot. The decoder will see it, push it, and advance its index. If either side counts differently, every subsequent @ reference points to the wrong value.

Worked example

Input:

{"name":"Alice","friends":[{"name":"Bob"},{"name":"Alice"}]}

Encoding trace:

{                        → (start object, no slot yet)
  "name"                 → string, gets slot 0
  "Alice"                → string, gets slot 1
  "friends"              → string, gets slot 2
  [                      → (start array, no slot yet)
    {                    → (start object)
      @0                 → reference to slot 0 ("name") — no new slot
      "Bob"              → string, gets slot 3
    }                    → object done, gets slot 4
    {                    → (start object)
      @0                 → reference to slot 0 ("name") — no new slot
      @1                 → reference to slot 1 ("Alice") — no new slot
    }                    → object done, gets slot 5
  ]                      → array done, gets slot 6
}                        → root object done, gets slot 7

Output: {"name":"Alice","friends":[{@0:"Bob"},{@0:@1}]}

The decoder builds the same table as it parses, so @0 always means "name" and @1 always means "Alice".

Install

bun add dd-json

Library

import { encode, decode, format } from "dd-json";

// Encode: JavaScript value → dd-json string
const ddj = encode({ name: "Alice", friends: [{ name: "Bob" }, { name: "Alice" }] });
// '{"name":"Alice","friends":[{@0:"Bob"},{@0:@1}]}'

// Decode: dd-json string (or JSON) → JavaScript value
const obj = decode(ddj);
const obj2 = decode('{"name": "Alice"}'); // JSON works too

// Pretty-print with optional syntax highlighting
const pretty = format(ddj);

// With inkglow theme colors
import { getTheme, colorize } from "inkglow";
const colored = format(ddj, { theme: getTheme("Inkglow"), colorize });

API

encode(value: unknown): string

Encodes a JavaScript value to compact dd-json. Single-pass, no intermediate data structures beyond hash tables for dedup lookups.

decode(input: string): unknown

Decodes dd-json (or plain JSON, or pretty-printed, or ANSI-colored) back to a JavaScript value. Skips whitespace and ANSI escape sequences automatically.

format(input: string, options?): string

Pretty-prints a dd-json or JSON string with indentation and optional syntax highlighting.

Options:

• indent — spaces per level (default: 2)
• theme — an inkglow theme object
• colorize — the colorize function from inkglow

CLI

ddj [file] [flags]

Input is auto-detected — JSON is encoded to DDJ, DDJ is passed through. Reads from stdin if no file given.

ddj data.json                # JSON → DDJ (multiline + color on terminal)
ddj data.json -s             # JSON → DDJ (single-line + color on terminal)
ddj data.ddj -J              # DDJ → JSON
cat data.json | ddj          # pipe: JSON → DDJ (single-line, no color)
cat data.ddj | ddj -J        # pipe: DDJ → JSON
ddj data.ddj -c | less -R   # force color into pager

Flags

Short	Long	Description
-J	--json	Output as JSON (default: DDJ)
-s	--single-line	Force compact single-line output
-m	--multiline	Force multiline indented output
-c	--color	Force color output
-n	--no-color	Disable color
-t	--theme	Inkglow theme name (default: Inkglow)
-h	--help	Show help

Defaults: TTY → multiline + color, pipe → single-line + no color.

Syntax highlighting

Colors use inkglow themes when available. Install it for color support:

bun add inkglow

If inkglow is not installed, all output is plain text. The colorize option in the format() API lets you plug in any colorizer.

Performance

Benchmarked on real-world JSON documents (Apple M4 Pro, Bun 1.3 / Node 22):

Document	Type	JSON	DDJ	Reduction	Encode	Decode
Docs site metadata	Paths manifest	121.5 MB	14.9 MB	87.8%	338ms	69ms
Kubernetes pods (5K)	API response	2.1 MB	322 KB	85.2%	9ms	2ms
Nobel laureates	API response	3.6 MB	1.3 MB	63.9%	19ms	8ms
IoT telemetry (50K)	Sensor data	7.8 MB	2.8 MB	63.8%	44ms	24ms
npm package metadata	Registry API	782 KB	288 KB	63.2%	3ms	3ms
SF city lots	GeoJSON	152 MB	63 MB	58.5%	1762ms	348ms
GitHub events (10K)	API response	28.3 MB	12.6 MB	55.3%	108ms	56ms
Reddit thread	API response	277 KB	129 KB	53.6%	1ms	1ms
World countries	GeoJSON	12.6 MB	10.4 MB	18.0%	276ms	60ms

Reduction depends on data repetitiveness. Structured data with repeated shapes (manifests, K8s resources, config blobs) sees 80–88%. API responses and sensor data typically see 50–65%. Highly unique data (GeoJSON coordinates, JSON Schema descriptions) sees 16–18% — just key deduplication — but is never larger than the input.

License

MIT