Skip to content

fix(runtime): give iterator helpers their own class id (#7576) - #7583

Merged
proggeramlug merged 3 commits into
mainfrom
fix/7576-iterator-helpers
Aug 7, 2026
Merged

fix(runtime): give iterator helpers their own class id (#7576)#7583
proggeramlug merged 3 commits into
mainfrom
fix/7576-iterator-helpers

Conversation

@proggeramlug

@proggeramlug proggeramlug commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Closes #7576.

Root cause: a class-id collision

crates/perry-runtime/src/iterator_helpers.rs:45 and
crates/perry-runtime/src/string/iter_object.rs:25 both defined 0xFFFF_0009:

pub const ITERATOR_HELPER_CLASS_ID: u32 = 0xFFFF_0009;   // "sits just past the Set iterator id (0xFFFF0008)"
pub const STRING_ITERATOR_CLASS_ID:  u32 = 0xFFFF_0009;   // "sits just past the Set iterator id (0xFFFF0008)"

Two constants, introduced independently, carrying the same comment. Neither
author checked whether the slot was taken.

Every dispatch tower matches these ids in a fixed order, and the String arm
comes first — object/native_call_method/handle_methods.rs:838 and
.../collection_methods.rs:396, both ahead of the helper arm at :855/:410.
So every iterator-helper object was dispatched as a String iterator:

call what actually ran result
.next() dispatch_string_iterator_method, reading the helper's op-kind field as a cursor index against a null backing array { done: true } on the first step, for every source kind
.map / .filter / .take / .drop / .flatMap / .toArray / .reduce / .forEach / .some / .every / .find the same dispatcher's _ => undefined arm undefined, so the next link in the chain threw

That is both symptoms in the issue from one cause, and it explains why row C
(the bare {next} driven directly) worked: it never went near the helper.

The helper now takes 0xFFFF_000B. 0xFFFF_000A is the RegExp-string iterator.

Second, independent defect on the same path

iterator_step resolved the source iterator's next with the inheriting
getter (js_object_get_field_by_name). Since #321 every built-in iterator
inherits .next from its shared %…IteratorPrototype% singleton
(object/iterator_prototypes.rs), and that inherited next is a thunk
that resolves its receiver from js_implicit_this_get() and dispatches by
class id.

So for an array / Map / Set / String iterator source the lookup found a
perfectly callable closure, iterator_step took the raw-closure-call branch —
js_closure_call1, which binds no this — and the thunk ran against whatever
was left in the thread-local. Observed directly: Method %IteratorPrototype%.next called on incompatible receiver.

array/iterator.rs::js_iterator_to_array already carries the own-field version
of this fix and documents exactly this hazard. iterator_step now matches it:
js_object_get_own_field_or_undef (also allocation-free, so it drops an intern
from the per-step path), and the call binds this to the iterator per
IteratorNext's Call(next, iterator) — which an own next() { return this.#impl.next(); } needs too.

Both halves are load-bearing. With the class id fixed and iterator_step
reverted, the new suite aborts on the brand-check throw.

Why this survived: the test existed and was skipped

test-files/test_gap_iterator_helpers_2874.ts covers most of this surface and
has been in test-parity/known_failures.json since 2026-07-04
("reason": "iterator helpers (#2874); standing per #5917."). It passes
byte-for-byte now, and this PR removes the skip.

Coverage

Per CLAUDE.md #5960 (integration suites under crates/*/tests/ only run
nightly/tag), the acceptance coverage is cargo-test-visible plus a gap test.

crates/perry-runtime/src/iterator_helpers/tests.rs — 16 unit tests.
Every behavioural one drives js_native_call_method, the tower a compiled
program reaches, not dispatch_iterator_helper_method directly. That is
deliberate: a test that called the helper dispatcher directly would have been
green throughout the entire outage, because the collision lives in the
tower's arm order, not in the dispatcher. Two structural tests:

  • iterator_class_ids_are_pairwise_distinct — enumerates all seven
    runtime-defined iterator class ids and fails on any duplicate, with an error
    message naming the pair. This is the test that would have caught the bug at
    the moment it was introduced.
  • an_own_next_method_is_called_with_the_iterator_as_this — pins the
    this-binding half independently of the own-vs-inherited lookup.

test-files/test_gap_iterator_helpers_7576.ts — 28 output lines, byte-identical
to node --experimental-strip-types on the pinned 26.5.1. Covers the issue's
reproducer verbatim (rows A–D), a helper stored in a local and hand-stepped
(the shape test_gap_iterator_helpers_2874.ts never exercised — it is all
inline chains), all six source kinds (bare {next}, generator, array, array
iterator, Map iterator, Set iterator, string iterator), every combinator and
every terminal, laziness over an unbounded generator, spread, and the String
iterator itself.

Validation

Local; CI has a deep backlog and has not reported.

  • cargo test -p perry-runtime --no-fail-fast1834 passed, 0 failed.
  • Issue reproducer + a 9-case probe: byte-identical to Node 26.5.1.
  • 30 iterator/spread/for-of/string-adjacent gap tests byte-compared against
    Node: 30/30 pass, including test_gap_7563_array_iterator_class_id_confusion,
    the three test_gap_7564_iter_result_*, and test_gap_iterator_helpers_2874
    (the un-skipped one).
  • python3 scripts/raw_handle_debt.py → 998 (baseline 998).
  • scripts/check_file_size.sh → OK.
  • cargo fmt --all -- --check → clean.

Sabotage evidence

  • Restore ITERATOR_HELPER_CLASS_ID = 0xFFFF_0009:
    iterator_class_ids_are_pairwise_distinct fails naming the
    iterator-helper/string pair, and to_array_drains_the_chain,
    take_..., reduce_..., some_every_find_..., map_..., filter_...
    all go red.
  • Revert iterator_step to the inherited lookup + unbound js_closure_call1
    (class id left fixed): the suite aborts on Method %IteratorPrototype%.next called on incompatible receiver.
  • Drop only the js_implicit_this_set pair:
    an_own_next_method_is_called_with_the_iterator_as_this reports a receiver
    of undefined.

Out of scope, but found while here

Two things a reviewer may want to pick up separately:

  1. RAW_JSON_CLASS_ID and JSX_NODE_CLASS_ID are both 0xFFFF_00A0
    (json/raw_json.rs:23, jsx.rs:33). Same shape as this bug. They are not
    in the iterator family so the new distinctness test does not cover them, and
    I have not established whether any tower matches both.
  2. python3 scripts/addr_class_inventory.py fails on main (verified on a
    clean tree at origin/main 1e971d2), on
    crates/perry-runtime/src/iter_result.rs:139 — a gcheader-cast introduced
    by perf(iterator): build one object per .next(), not five (#7564) #7579. It needs a mutable header write, so try_read_gc_header (which
    returns &'static) is not a drop-in; it wants either an allowlist entry or
    a gc_flags setter. Left alone here to keep this PR scoped.

Summary by CodeRabbit

  • Bug Fixes

    • Fixed iterator helper behavior across supported source types and chained operations.
    • Corrected iterator stepping, receiver binding, lazy consumption, spreading, and terminal operations.
    • Resolved conflicts affecting string iterators and iterator helpers.
  • Tests

    • Added comprehensive coverage for iterator creation, combinators, reductions, predicates, and identity behavior.
  • Chores

    • Updated the application version to 0.5.1330.

proggeramlug pushed a commit that referenced this pull request Aug 7, 2026
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 00f4a17f-45c8-4fde-add3-0f13f3db63a5

📥 Commits

Reviewing files that changed from the base of the PR and between 4e610b6 and 71317ab.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (8)
  • CLAUDE.md
  • Cargo.toml
  • changelog.d/7583-iterator-helpers-class-id-collision.md
  • crates/perry-runtime/src/iterator_helpers.rs
  • crates/perry-runtime/src/iterator_helpers/tests.rs
  • crates/perry-runtime/src/string/iter_object.rs
  • test-files/test_gap_iterator_helpers_7576.ts
  • test-parity/known_failures.json

📝 Walkthrough

Walkthrough

Iterator helper dispatch now uses a unique class ID, own next lookup, and correct receiver binding. Runtime and end-to-end tests cover iterator sources, combinators, terminal operations, laziness, spread iteration, and String iterator compatibility. The workspace version is updated to 0.5.1330.

Changes

Iterator dispatch correction

Layer / File(s) Summary
Iterator identity and next dispatch
crates/perry-runtime/src/iterator_helpers.rs, crates/perry-runtime/src/string/iter_object.rs
Iterator helpers use class ID 0xFFFF_000B. iterator_step performs allocation-free own-field lookup and invokes next with the iterator as this.
Runtime dispatch tower validation
crates/perry-runtime/src/iterator_helpers/tests.rs
Runtime tests validate class IDs, operation discriminants, source conversion, combinators, terminal methods, receiver binding, and String iterator dispatch.
Iterator helper regression coverage
test-files/test_gap_iterator_helpers_7576.ts, test-parity/known_failures.json, changelog.d/7583-iterator-helpers-class-id-collision.md, CLAUDE.md, Cargo.toml
End-to-end tests cover stored helpers, supported sources, combinators, terminal methods, laziness, spread iteration, String iterators, and helper identity. The known failure is removed, the changelog records the fix, and the version is updated.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant IteratorFrom
  participant iterator_step
  participant OwnNext
  IteratorFrom->>iterator_step: request next step
  iterator_step->>IteratorFrom: read own next field
  iterator_step->>OwnNext: invoke with iterator as this
  OwnNext-->>iterator_step: return iterator result
Loading

Possibly related PRs

  • PerryTS/perry#7495: Both modify iterator runtime behavior and GC-safe handle management.
  • PerryTS/perry#7569: Both address iterator-related class-ID and method-dispatch issues.
  • PerryTS/perry#7584: Both address GC-safe runtime pointer handling across iterator-related calls.

Suggested labels: bug, parity

Suggested reviewers: thehypnoo

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/7576-iterator-helpers

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Ralph Küpper added 3 commits August 7, 2026 09:08
`ITERATOR_HELPER_CLASS_ID` and `STRING_ITERATOR_CLASS_ID` were both
`0xFFFF_0009`. The two constants were introduced independently, each with a
comment reading "sits just past the Set iterator id (0xFFFF0008)", and neither
author checked whether the slot was taken.

Every dispatch tower matches these ids in a fixed order and tests the String
arm before the helper arm, so EVERY iterator-helper object was dispatched as a
String iterator:

  * `.next()` read the helper's op-kind field as a cursor index against a null
    backing array, so it answered `{ done: true }` on its first step —
    `Iterator.from(x)` was exhausted before it started, silently and for every
    source kind.
  * every other helper method (`map`/`filter`/`take`/`drop`/`flatMap`/
    `toArray`/`reduce`/`forEach`/`some`/`every`/`find`) fell into the String
    dispatcher's `_ => undefined` arm, so `Iterator.from(g).map(f)` was
    `undefined` and `.filter` on it threw.

The helper takes `0xFFFF_000B` (`0xFFFF_000A` is the RegExp-string iterator).

Second, independent defect on the same path: `iterator_step` resolved the
source's `next` with the INHERITING getter. Since #321 every built-in iterator
inherits `.next` from its shared `%…IteratorPrototype%` singleton, and that
inherited `next` is a thunk that resolves its receiver from
`js_implicit_this_get()`. So the lookup found a callable closure for an
array/Map/Set/String iterator source, took the raw-closure-call branch, and ran
the thunk with whatever `this` was left in the thread-local — `done` on the
first step, or `Method %IteratorPrototype%.next called on incompatible
receiver`. `js_iterator_to_array` already carries the own-field version of this
fix; `iterator_step` now uses `js_object_get_own_field_or_undef` too, and binds
`this` to the iterator around the call per `IteratorNext`'s
`Call(next, iterator)`. Both halves are load-bearing: with the class id fixed
and `iterator_step` reverted, the new tests abort on the brand-check throw.

Why nobody noticed: `test_gap_iterator_helpers_2874.ts` DID catch this, and has
been listed in `test-parity/known_failures.json` since 2026-07-04. It passes
byte-for-byte now and is un-skipped here.

Coverage, per CLAUDE.md #5960 (integration suites under `crates/*/tests/` only
run nightly/tag):

  * `crates/perry-runtime/src/iterator_helpers/tests.rs` — 16 `cargo-test`
    visible unit tests. They all drive `js_native_call_method`, the tower a
    compiled program reaches, NOT `dispatch_iterator_helper_method` directly:
    a test that called the helper dispatcher directly would have been green
    throughout the outage, because the collision lives in the tower's arm
    order. Includes `iterator_class_ids_are_pairwise_distinct`, which fails on
    any duplicate in the family, and
    `an_own_next_method_is_called_with_the_iterator_as_this`, which pins the
    `this`-binding half independently.
  * `test-files/test_gap_iterator_helpers_7576.ts` — 28 lines byte-identical
    to `node --experimental-strip-types` (26.5.1), covering the issue's
    reproducer, a stored hand-stepped helper, all six source kinds, every
    combinator and terminal, laziness over an unbounded generator, spread, and
    the String iterator itself.

Sabotage-verified both ways: restoring `0xFFFF_0009` fails
`iterator_class_ids_are_pairwise_distinct` plus 6 behavioural tests; reverting
`iterator_step` aborts the suite on the brand check.
@proggeramlug
proggeramlug force-pushed the fix/7576-iterator-helpers branch from cdf996d to 71317ab Compare August 7, 2026 07:08
@proggeramlug
proggeramlug merged commit e25a621 into main Aug 7, 2026
10 of 11 checks passed
@proggeramlug
proggeramlug deleted the fix/7576-iterator-helpers branch August 7, 2026 07:09
@proggeramlug

Copy link
Copy Markdown
Contributor Author

Audit before merge — verified, merged as v0.5.1330

Reproduced the fix end-to-end against node on my own build, with a test I
wrote rather than the PR's, covering the whole surface plus a control that the
colliding type still works:

from+toArray: [0,1,2,3]      map: [0,2,4,6]        filter: [0,2,4]
take: [0,1,2]                drop: [4,5]           flatMap: [0,0,1,1,2,2]
reduce: 10                   some/every/find: true true 3
string iter still ok: ["a","b","c"]

Byte-identical to node --experimental-strip-types, exit 0.

Sabotage-verified: restoring ITERATOR_HELPER_CLASS_ID = 0xFFFF_0009 fails
iterator_class_ids_are_pairwise_distinct. Gates re-run: addr_class_inventory
passes, raw_handle_debt 998, check_file_size.sh and cargo fmt --check
clean.

The root cause is worth restating because it is so cheap and so destructive: two
constants, defined independently in different files, each with a comment
explaining where it sits relative to its neighbour, both landing on
0xFFFF_0009. Every dispatch tower matches the String arm first, so every
helper object dispatched as a String iterator. One collision, an entire TC39
surface silently returning undefined.

The new test cannot catch the next one — and there IS a next one

iterator_class_ids_are_pairwise_distinct is scoped to the iterator family. I
checked the rest of the id space and found a second live collision:
JSX_NODE_CLASS_ID and RAW_JSON_CLASS_ID are both 0xFFFF_00A0, filed as
#7587. It is not merely latent — value/to_string.rs:1144 discriminates on
class_id alone, so String(JSON.rawJSON("123")) takes the JSX branch and
returns field 0. It gives a plausible answer only because both types happen to
store their payload in field 0.

A distinctness test that enumerates one family is structurally unable to fail
for the next family. The follow-up in #7587 is to assert pairwise distinctness
over every reserved 0xFFFF_00xx id, so the next constant minted by copying
a neighbour gets a red build rather than a cross-match nobody finds for years.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Iterator.from(x) returns an already-exhausted iterator; .map/.filter/.take return undefined

1 participant