test(canonicalization): a second vector for the form a careful implementer reaches

[lywinged]

test_every_adhoc_form_is_caught_by_some_vector asserts that every ad-hoc form is separated by some vector, and every form is. Counting how many vectors separate each one gives a different answer.

Ad-hoc form	Vectors that separate it
sort_keys_default	3
sort_keys_compact	3
sort_keys_compact_utf8	1

sort_keys_compact_utf8 is the form left after compact separators and ensure_ascii are both set correctly, which is to say the one a careful implementer actually reaches. It was separated by 03 alone. One vector is coverage until that vector changes: weaken or retire 03 and the closest non-conformant canonicalizer passes the whole set, with every other assertion in the module still green. That is the margin rule of #124 applied to the set that measures the section 3.2.2 MUST.

04 moves 03's divergence inside a nested object, so it is a distinct defect rather than a restatement. A canonicalizer that sorts by UTF-16 code units at the outer levels and by code points below them accepts 03 and rejects 04.

The margin this adds is positional rather than mechanistic, and worth naming as such: key ordering is the only RFC 8785 divergence this schema can reach at all, since no field is typed number and IEEE 754 serialization is therefore unreachable, which test_number_divergence_is_still_unreachable already pins.

Reproduction

git fetch origin pull/170/head:pr170 && git checkout pr170

The count in the table:

python - <<'PY'
import json, pathlib
from collections import Counter
c = Counter(n for p in sorted(pathlib.Path("examples/canonicalization-boundary").glob("*.json"))
            for n in json.loads(p.read_text())["diverges_under"])
for form, n in sorted(c.items()): print(f"{form:26} {n}")
PY

On HEAD~1 that prints sort_keys_compact_utf8 1; on this branch it prints 2.

The distinct defect. A canonicalizer sorting by code units only to a given depth, then falling back to the default sort:

python - <<'PY'
import json, rfc8785
def ser(o, d, correct_to):
    if isinstance(o, dict):
        ks = sorted(o, key=lambda s: s.encode("utf-16-be")) if d <= correct_to else sorted(o)
        return "{" + ",".join(json.dumps(k, ensure_ascii=False) + ":" + ser(o[k], d+1, correct_to)
                              for k in ks) + "}"
    if isinstance(o, list):
        return "[" + ",".join(ser(x, d+1, correct_to) for x in o) + "]"
    return json.dumps(o, ensure_ascii=False)

for name in ("03-utf16-key-order", "04-utf16-key-order-nested"):
    r = {k: v for k, v in
         json.load(open(f"examples/canonicalization-boundary/{name}.json"))["record"].items()
         if k != "signature"}
    ok = ser(r, 0, 2).encode() == rfc8785.dumps(r)
    print(f"{name:34} {'accepts' if ok else 'REJECTS'}")
PY

03-utf16-key-order                 accepts
04-utf16-key-order-nested          REJECTS

Load-bearing:

git rm -q examples/canonicalization-boundary/04-utf16-key-order-nested.json
pytest -q tests/test_canonicalization_boundary.py   # 2 failed, 8 passed
git checkout -- examples/canonicalization-boundary/04-utf16-key-order-nested.json
pytest -q tests/test_canonicalization_boundary.py   # 12 passed

Full suite on 3.11: 379 passed, 1 skipped. ruff check clean. Scope is examples/ and tests/ only.

[github-actions]

❔ Contributor Check: UNKNOWN

Check	Result
Profile	UNKNOWN
Credential	LOW
Overall	UNKNOWN

Automated check by AgenTrust Contributor Check.

[imran-siddique]

The new nested-key vector creates positional margin against a depth-dependent non-RFC-8785 sorter, the generator and committed fixture move together, and the test now requires every modeled ad-hoc form to be separated by at least two vectors.