Skip to content

Bound the typed-structure dictionary with maxOwnStructures (v1.11.x) - #185

Merged
kriszyp merged 1 commit into
v1.11.xfrom
kris/maxownstructures-v1
Jun 5, 2026
Merged

Bound the typed-structure dictionary with maxOwnStructures (v1.11.x)#185
kriszyp merged 1 commit into
v1.11.xfrom
kris/maxownstructures-v1

Conversation

@kriszyp

@kriszyp kriszyp commented Jun 5, 2026

Copy link
Copy Markdown
Owner

Summary

Backports an opt-in maxOwnStructures cap to the v1 randomAccessStructure typed-struct path (struct.js). Once typedStructs.length reaches the cap, novel record shapes fall back to plain msgpack encoding instead of minting new structures; the decode path is untouched, so existing and previously-persisted structs stay decodable. Default is uncapped — no behavior change unless a caller sets maxOwnStructures.

Why

The typed-struct encoder branches per field on the value's numeric width (num8/num32/num64, float32/float64) and string kind, so a wide/sparse schema mints a distinct structure per (key-set × width-combination) and grows typedStructs without bound — it's pinned on the long-lived encoder and never evicted. This OOM-crash-looped a production Harper cluster (per-table encoder + per-peer decoder dictionaries, multiplied across worker isolates). The cap bounds each dictionary. Harper will set it (planned 256).

This is the deployed-5.0.x path (Harper v5.0.x runs msgpackr v1, not structon); it mirrors the already-merged-in-review structon change (HarperFast/structon#4).

What to look at

  • struct.js writeStruct: the cap interacts with msgpackr's shared write position. Since the pack() callback advances it, the code can never return 0 after a pack() (it would corrupt the plain-object fallback) — so the cap is enforced before any pack() via a preflight, and the internal layout-retry passes structureKnown=true to re-encode the already-minted structure rather than re-bail.
  • Per-instance freeze: frozen is a local derived from this encoder's own typedStructs.length and passed explicitly to createTypeTransition (new frozen param) — never a shared module global, so a re-entrant encode on another instance can't lift the cap. forceTypeTransition (always-mint) is used in the queued-ref loop after the preflight.
  • Plumbing: reads packr.maxOwnStructures ?? Infinity (already on the instance via Object.assign in unpack.js). The classic shared-record cap (also maxOwnStructures, in pack.js, default 64) is unchanged — this only adds the typed-struct cap.
  • Bounded overshoot (by design): a record whose own nested encodes cross the cap mid-encode can exceed it by one record's worth before the next preflight catches it; converges, negligible at 256. Flat records are a strict hard bound.

Review notes

  • 9 new tests + 1 layout-retry test (121 passing, 0 regressions). Logic mirrors structon#4 (hardened over 10 Codex review rounds); a fresh Codex pass on this port found no issues.
  • Targets the new v1.11.x maintenance branch (off the v1.11.12 tag) for a 1.11.13 patch release.

🤖 Generated by Claude (Opus 4.7), porting the reviewed structon implementation. Please sanity-check the fast-path cap logic against v1's writeStruct/pack.js integration.

Backport of structon's maxOwnStructures cap to msgpackr v1's randomAccessStructure
typed-struct path. The typed-struct encoder branches per field on the value's
numeric width and string kind, so a wide/sparse schema mints a distinct structure
per (key-set x width-combo) and grows typedStructs unbounded (pinned on the
long-lived encoder, never evicted) — an OOM vector under shape-heterogeneous data.

Once typedStructs.length reaches maxOwnStructures, novel shapes fall back to plain
msgpack encoding instead of minting new structures; the read path is untouched so
existing and persisted structs stay decodable. Default is uncapped (no behavior
change unless a caller sets maxOwnStructures, which is already plumbed onto the
instance via Object.assign). This is the typed-struct cap only — the classic
shared-record cap (also maxOwnStructures, in pack.js) is unchanged.

The cap is enforced before any pack() advances the shared write position (a
preflight falls records that would mint back to plain), is per-instance (frozen
derived from this encoder's typedStructs.length, passed to createTypeTransition —
not a shared global), survives the layout-retry (structureKnown), and re-saves
combined structures so persistence isn't clobbered. Mirrors structon's
implementation (10 review rounds); flat records are a strict hard bound.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant