Structural and merge replacements, Hypothesis property tests

[simonw]

Three features that extend replacements beyond substring matching, built for llm's condensed response payload storage.

Three key changes:

• Replacements of {"a": {"object": ["nested", "list"]}}` now work - previously the value had to be a string, now it can be any JSON structure.
• For objects in that replacement list, the resulting condensed structure can specify merge operations - take that object, add or change these keys, delete these other keys.
• Hypothesis testing to help ensure round-trips across complex examples.

Claude Fable 5 description

Structural replacements

A replacement value may now be a dict or a list. Container values match structurally - any subtree equal to the value in canonical JSON form is replaced whole with the existing {"$": id} marker, so key order and serialization bytes never matter. Matching is outermost-wins and strictly structural (a string containing the JSON serialization of a value never matches). Resolution substitutes independent deep copies; round trips are structurally equal rather than byte-identical, since the original key order of a matched subtree is not recorded.

Merge references

Dict replacements double as merge bases: a dict that is mostly equal to a base - a static envelope with a few varying fields, like API response metadata - is stored as a dict-valued marker:

{"$": {"m": "base_id", "u": {"added or changed": "keys"}, "d": ["absent keys"]}}

Condensing needs no similarity heuristic: for each base the patch is computed and both encodings are measured, and the merge form is emitted only when it is smaller - unrelated bases price themselves out. Deletion is the explicit d list rather than a JSON Merge Patch-style null sentinel, because null is a legitimate payload value. The dict-valued marker is still a single-key $ dict, so the existing $raw escaping already protects input that looks like a merge reference. Measured on real OpenAI Responses payloads in llm, merge references took a typical reply from 22% to 44% saved, since the ~15 static top-level envelope keys collapse to one reference.

Performance was benchmarked and tuned: a key-overlap early-out skips the cost computation for nodes sharing no keys with any base (without it, deeply nested documents went quadratic - 29ms for a 36KB doc, now 0.6ms), base canonical forms are computed once per call, and input subtree canonicals are memoized. A realistic payload condenses in under 0.1ms; adding bases to a large string-replacement workload costs nothing measurable.

Property-based tests

Six Hypothesis properties: the round-trip contract under arbitrary, document-derived, and merge-targeted replacements; output always JSON-serializable; repeated application lossless; and uncondense_json on arbitrary input either resolves or raises UncondenseError, never anything else. The strategies were validated by mutation testing - three planted bugs (equality-semantics, dropped escaping, raw-instead-of-processed patches) are all caught. That process exposed and fixed two real weaknesses: a confusables generator now draws True/False/0/1/0.0/1.0 frequently, and assertions compare canonical JSON alongside ==, because True == 1 makes bool/int corruption invisible to == alone. HYPOTHESIS_PROFILE=thorough runs 2,000 examples per property.

Notes for review

• First-ID-wins now holds for merge bases too (kept in mapping order, equivalent values deduplicated), matching the documented rule for other replacement kinds; cost ties also go to the earlier entry.
• Patches are flat by design: a nested value that differs at all travels whole in u (condensed recursively on the way). Nested bases provide depth when wanted.
• uv run pytest: 71 passed. uv run mypy condense_json tests: clean. Black formatted. README documents both new behaviors with executable examples.

🤖 Generated with Claude Code