Skip to content

Add property-based round-trip tests and the RFC 6902 conformance corpus#51

Merged
Korijn merged 1 commit into
masterfrom
claude/test-hardening
Jul 10, 2026
Merged

Add property-based round-trip tests and the RFC 6902 conformance corpus#51
Korijn merged 1 commit into
masterfrom
claude/test-hardening

Conversation

@Korijn

@Korijn Korijn commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Track D of the roadmap — test the library's core promises directly instead of spot-checking them. Independent of #50 (both merge cleanly in either order).

Property tests (new hypothesis dev dependency)

Hypothesis generates arbitrary nested structures of dicts, lists, sets, tuples, frozensets and scalars, and verifies:

  • diff + apply round-trips in both directions for any same-kind pair of documents.
  • apply never mutates its input; iapply mutates exactly its input and returns the same object; diff of equal objects is empty.
  • Serialization is lossless for JSON-shaped data: operations survive to_jsonjson.loadsPointer.from_str and still apply correctly, in both directions.
  • Pointer string rendering and RFC 6901 token escaping round-trip.

RFC 6902 conformance corpus

tests/rfc6902/ vendors tests.json and spec_tests.json verbatim from json-patch/json-patch-tests, the community conformance suite. All 50 applicable cases pass. The 62 cases outside patchdiff's subset-plus-extensions are skipped with explicit, per-case reasons: move/copy/test operations, operations on the document root, upstream-disabled cases, and invalid-patch validation cases (patchdiff applies trusted patches; it does not validate).

Bug found and fixed by these tests 🐛

Pointers parsed from strings carry string tokens, and Pointer.evaluate's strict parent walk passed them straight into list indexing — so any serialized patch whose path crossed a list could not be applied:

iapply(doc, [{"op": "remove", "path": Pointer.from_str("/baz/0/qux")}])
# TypeError: list indices must be integers or slices, not str

This made the documented serialize → parse → apply flow broken for nested-list paths (both the corpus and the JSON round-trip property caught it immediately). evaluate now retries string tokens as integer indices for list-like parents — mirroring what iapply already did at the leaf — with targeted regression tests for the fix, for non-numeric tokens on lists (ValueError), and for traversal into primitives (still TypeError).

Known limitation made explicit

Diffing two documents of different top-level kinds (a list against a dict) yields a whole-document replace at the root, which apply/iapply cannot execute. Rather than hiding it, the limitation is pinned in a dedicated test (a future fix must update it deliberately) and documented in the gotchas page. Worth deciding before 1.0.0 whether to support root replaces (e.g. by returning the new root from apply) — happy to take that on as a follow-up if you want it.

Verification

405 passed + 62 documented corpus skips, coverage stays at 100% (including the new evaluate branches), ruff check/format pass, docs build stays warning-free.

🤖 Generated with Claude Code

https://claude.ai/code/session_016Mu9vEBwU4fQLi8ZgkgdS2


Generated by Claude Code

Track D of the 1.0.0 roadmap: test the library's core promises
directly instead of spot-checking them.

Property tests (new hypothesis dev dependency):
- diff + apply round-trips in both directions for arbitrary nested
  structures of dicts, lists, sets, tuples, frozensets and scalars
- apply never mutates its input; iapply mutates exactly its input and
  returns the same object; diff of equal objects is empty
- serialization is lossless for JSON-shaped data: ops survive
  to_json -> json.loads -> Pointer.from_str and still apply, in both
  directions
- pointer string rendering and token escaping round-trip

Conformance corpus:
- tests/rfc6902/ vendors tests.json and spec_tests.json verbatim from
  json-patch/json-patch-tests; all 50 applicable cases pass, and the
  62 cases outside patchdiff's subset-plus-extensions are skipped with
  explicit reasons (move/copy/test ops, root operations, and
  invalid-patch validation cases)

Bug found and fixed by these tests: pointers parsed from strings carry
string tokens, and Pointer.evaluate's strict parent walk passed them
straight into list indexing - so any serialized patch whose path
crossed a list could not be applied (TypeError). evaluate now retries
string tokens as integer indices for list-like parents, mirroring what
iapply already did at the leaf. Regression tests added alongside the
existing pointer tests.

Known limitation made explicit: diffing documents of different
top-level kinds yields a whole-document replace at the root, which
apply/iapply cannot execute. Pinned in a dedicated test and documented
in the gotchas page.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Mu9vEBwU4fQLi8ZgkgdS2
@Korijn Korijn merged commit 420ff28 into master Jul 10, 2026
10 checks passed
@Korijn Korijn deleted the claude/test-hardening branch July 10, 2026 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants