Fix two-byte record corruption: cap maxOwnStructures at 32 (v5.0)

[kriszyp]

Summary

RecordEncoder set maxOwnStructures = 256. With msgpackr's default maxSharedStructures = 32, maxOwn + maxShared > 64 flips on msgpackr's two-byte record-id encoding, which mis-serializes over-cap "own" structures when a shared-structures store is present (every primary store has one): it writes an out-of-range record-id reference instead of inlining the structure. Once a table exceeds the shared cap, new records become undecodable — Record id is not defined for N. This fix lowers the cap to 32, keeping msgpackr on the correct one-byte path (which inlines over-cap shapes and bounds memory at least as tightly).

Why this matters (data integrity, shipped in v5.0.29)

The broken maxOwnStructures=256 is the OOM cap from #1146, shipped in v5.0.29. The corruption hits both configs:

• Default (typed / randomAccessFields=true)
• Opt-out (randomAccessFields=false, Add storage.randomAccessFields config to disable typed structures (v5.0) #1169)

i.e. any high-cardinality table (e.g. CDI's RaceEntry with undeclared dynamic attributes) writing records past the ~32-shape shared cap produces undecodable records. This is the Record id is not defined for N flood reported on dev/CDI (Slack thread, and Nathan Heskew's verification on #1169) — root-caused here to the two-byte path, not the #1157/#186 save-race.

Patch candidate for v5.0.30.

Repro (high confidence)

Pure-msgpackr matrix with a shared store + 3–5k distinct shapes:

• maxOwn=256 → ~2900/3000 records undecodable
• maxOwn=33 (just over the one-byte boundary) → also broken
• maxOwn=32 / default → 0 errors, dictionary bounded

Where to look

• resources/RecordEncoder.ts — the one-line cap change (256 → 32) + a comment explaining the one-byte invariant (maxOwn + maxShared ≤ 64).
• unitTests/resources/recordEncoder.test.js — new regression block: writes 500 distinct shapes through a shared store and asserts every record decodes, on both the typed and readOnlyStructures paths. (Fails at 256, passes at 32.)

Open / follow-up

• The deeper msgpackr two-byte over-cap fix (to make >32 shared structures correct, not just avoided) is in progress separately; this PR is the safe immediate stop-gap.
• Already-written data: records persisted on v5.0.29 after a table exceeded the shared cap may be permanently undecodable — needs a separate recovery assessment.
• Pre-existing on v5.0.29 (not from this PR): the Add storage.randomAccessFields config to disable typed structures (v5.0) #1169 recordEncoder.test.js config/byte-range tests flake under full-file ordering (config-state isolation), and the resources suite has the environmental "Data read" setup crash. The new regression tests pass in isolation and are robust to ambient config (they assert decodability, not byte ranges).

🤖 Generated by Claude (Opus 4.7). Codex review: clean — no actionable correctness issues introduced. (Gemini was daily-quota-limited on the related PRs.)

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[kriszyp]

Superseding this with #1179. The cap→32 here was a write-side stop-gap for the unfixed msgpackr two-byte path. Now that the root cause is fixed upstream (msgpackr 1.12.1, read-path only — kriszyp/msgpackr#189), #1179 just bumps the dep and keeps maxOwnStructures=256:

• It recovers records already written by v5.0.29 (the bytes were always correct; only the read was wrong), not just future writes.
• Keeping 256 is ~32% more compact than cap=32 for moderate-cardinality tables (shares shapes instead of re-inlining a definition per record), and within ~2% for extreme cardinality — so 256 is both correct and better.

The regression test from here is carried over to #1179. Closing in favor of #1179.

— Claude