Bound the typed-structure dictionary with maxOwnStructures (v1.11.x) - #185
Merged
Conversation
Backport of structon's maxOwnStructures cap to msgpackr v1's randomAccessStructure typed-struct path. The typed-struct encoder branches per field on the value's numeric width and string kind, so a wide/sparse schema mints a distinct structure per (key-set x width-combo) and grows typedStructs unbounded (pinned on the long-lived encoder, never evicted) — an OOM vector under shape-heterogeneous data. Once typedStructs.length reaches maxOwnStructures, novel shapes fall back to plain msgpack encoding instead of minting new structures; the read path is untouched so existing and persisted structs stay decodable. Default is uncapped (no behavior change unless a caller sets maxOwnStructures, which is already plumbed onto the instance via Object.assign). This is the typed-struct cap only — the classic shared-record cap (also maxOwnStructures, in pack.js) is unchanged. The cap is enforced before any pack() advances the shared write position (a preflight falls records that would mint back to plain), is per-instance (frozen derived from this encoder's typedStructs.length, passed to createTypeTransition — not a shared global), survives the layout-retry (structureKnown), and re-saves combined structures so persistence isn't clobbered. Mirrors structon's implementation (10 review rounds); flat records are a strict hard bound. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backports an opt-in
maxOwnStructurescap to the v1randomAccessStructuretyped-struct path (struct.js). OncetypedStructs.lengthreaches the cap, novel record shapes fall back to plain msgpack encoding instead of minting new structures; the decode path is untouched, so existing and previously-persisted structs stay decodable. Default is uncapped — no behavior change unless a caller setsmaxOwnStructures.Why
The typed-struct encoder branches per field on the value's numeric width (num8/num32/num64, float32/float64) and string kind, so a wide/sparse schema mints a distinct structure per (key-set × width-combination) and grows
typedStructswithout bound — it's pinned on the long-lived encoder and never evicted. This OOM-crash-looped a production Harper cluster (per-table encoder + per-peer decoder dictionaries, multiplied across worker isolates). The cap bounds each dictionary. Harper will set it (planned 256).This is the deployed-5.0.x path (Harper v5.0.x runs msgpackr v1, not structon); it mirrors the already-merged-in-review structon change (HarperFast/structon#4).
What to look at
struct.jswriteStruct: the cap interacts with msgpackr's shared write position. Since thepack()callback advances it, the code can neverreturn 0after apack()(it would corrupt the plain-object fallback) — so the cap is enforced before anypack()via a preflight, and the internal layout-retry passesstructureKnown=trueto re-encode the already-minted structure rather than re-bail.frozenis a local derived from this encoder's owntypedStructs.lengthand passed explicitly tocreateTypeTransition(newfrozenparam) — never a shared module global, so a re-entrant encode on another instance can't lift the cap.forceTypeTransition(always-mint) is used in the queued-ref loop after the preflight.packr.maxOwnStructures ?? Infinity(already on the instance viaObject.assigninunpack.js). The classic shared-record cap (alsomaxOwnStructures, inpack.js, default 64) is unchanged — this only adds the typed-struct cap.Review notes
v1.11.xmaintenance branch (off thev1.11.12tag) for a 1.11.13 patch release.🤖 Generated by Claude (Opus 4.7), porting the reviewed structon implementation. Please sanity-check the fast-path cap logic against v1's
writeStruct/pack.jsintegration.