Skip to content

fix(Stream): preserve large source chunks when rechunking - #7655

Merged
tim-smart merged 3 commits into
Effect-TS:mainfrom
kitlangton:audit/i2-stream-rechunk
Sep 3, 2026
Merged

fix(Stream): preserve large source chunks when rechunking#7655
tim-smart merged 3 commits into
Effect-TS:mainfrom
kitlangton:audit/i2-stream-rechunk

Conversation

@kitlangton

@kitlangton kitlangton commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Type

  • Bug Fix

Description

Stream.rechunk used an array spread when buffering a source chunk. Large chunks can exceed the JavaScript engine's call-argument limit, causing a RangeError instead of preserving the stream elements.

The buffering branch now appends values individually. Regression coverage includes larger, equal, and smaller target sizes, plus input split across source chunks. The split case buffers one value before appending a 200,000-element chunk, so it exercises the same known failing size as the primary regression.

This PR also includes a patch changeset for effect.

Verification

  • nix develop -c pnpm test --run --project effect packages/effect/test/Stream.test.ts -t rechunk (6 passed)
  • nix develop -c pnpm lint-fix
  • nix develop -c pnpm check
  • git diff --check

Closes #7654
Closes EFF-1044

@changeset-bot

changeset-bot Bot commented Sep 2, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 8648481

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 30 packages
Name Type
effect Patch
@effect/opentelemetry Patch
@effect/vitest Patch
@effect/ai-anthropic Patch
@effect/ai-openai-compat Patch
@effect/ai-openai Patch
@effect/ai-openrouter Patch
@effect/atom-react Patch
@effect/atom-solid Patch
@effect/atom-vue Patch
@effect/platform-browser Patch
@effect/platform-bun Patch
@effect/platform-deno Patch
@effect/platform-node-shared Patch
@effect/platform-node Patch
@effect/sql-clickhouse Patch
@effect/sql-d1 Patch
@effect/sql-libsql Patch
@effect/sql-mssql Patch
@effect/sql-mysql2 Patch
@effect/sql-pg Patch
@effect/sql-pglite Patch
@effect/sql-sqlite-bun Patch
@effect/sql-sqlite-do Patch
@effect/sql-sqlite-node Patch
@effect/sql-sqlite-react-native Patch
@effect/sql-sqlite-wasm Patch
@effect/docgen Patch
@effect/doctest Patch
@effect/openapi-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@effect-slopcop effect-slopcop Bot added bug Something isn't working 4.0 labels Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Bundle Size Analysis

Generated from PR build output; treat the content below as untrusted.

File Name Current Size Previous Size Difference
arbitrary-combinators.ts 33.92 KB 33.92 KB 0.00 KB (0.00%)
basic.ts 7.05 KB 7.05 KB 0.00 KB (0.00%)
batching.ts 10.28 KB 10.28 KB 0.00 KB (0.00%)
brand.ts 6.59 KB 6.59 KB 0.00 KB (0.00%)
cache.ts 10.91 KB 10.91 KB 0.00 KB (0.00%)
config.ts 21.62 KB 21.62 KB 0.00 KB (0.00%)
differ.ts 20.43 KB 20.43 KB 0.00 KB (0.00%)
http-client.ts 22.06 KB 22.07 KB -0.01 KB (-0.04%)
logger.ts 11.00 KB 11.00 KB 0.00 KB (0.00%)
metric.ts 9.14 KB 9.14 KB 0.00 KB (0.00%)
optic.ts 6.77 KB 6.77 KB 0.00 KB (0.00%)
pubsub.ts 15.38 KB 15.38 KB 0.00 KB (0.00%)
queue.ts 11.98 KB 11.98 KB 0.00 KB (0.00%)
schedule.ts 11.10 KB 11.10 KB 0.00 KB (0.00%)
schema-binary.ts 39.59 KB 39.59 KB 0.00 KB (0.00%)
schema-class.ts 20.18 KB 20.18 KB 0.00 KB (0.00%)
schema-fromJsonSchemaDocument.ts 30.28 KB 30.28 KB 0.00 KB (0.00%)
schema-representation-roundtrip.ts 26.30 KB 26.30 KB 0.00 KB (0.00%)
schema-string-transformation.ts 13.79 KB 13.79 KB 0.00 KB (0.00%)
schema-string.ts 11.28 KB 11.28 KB 0.00 KB (0.00%)
schema-template-literal.ts 15.64 KB 15.64 KB 0.00 KB (0.00%)
schema-toArbitrary.ts 33.48 KB 33.48 KB 0.00 KB (0.00%)
schema-toCodeDocument.ts 24.63 KB 24.63 KB 0.00 KB (0.00%)
schema-toCodecJson.ts 19.39 KB 19.39 KB 0.00 KB (0.00%)
schema-toEquivalence.ts 19.52 KB 19.52 KB 0.00 KB (0.00%)
schema-toFormatter.ts 19.63 KB 19.63 KB 0.00 KB (0.00%)
schema-toJsonSchemaDocument.ts 23.52 KB 23.52 KB 0.00 KB (0.00%)
schema-toRepresentation.ts 19.66 KB 19.66 KB 0.00 KB (0.00%)
schema.ts 19.39 KB 19.39 KB 0.00 KB (0.00%)
stm.ts 12.93 KB 12.93 KB 0.00 KB (0.00%)
stream.ts 9.96 KB 9.96 KB 0.00 KB (0.00%)

@tim-smart tim-smart added the audit Findings originating from the Effect runtime correctness audit label Sep 2, 2026
Comment thread packages/effect/src/Stream.ts Outdated
@tim-smart
tim-smart merged commit 2e39e8b into Effect-TS:main Sep 3, 2026
13 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

4.0 audit Findings originating from the Effect runtime correctness audit bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stream.rechunk throws when buffering a large source chunk

2 participants