fix(node): encode string chunks when serializing stream bodies#487
Merged
Conversation
Adds coverage for passing a Web Request with a body, Node Readable streams, and a ReadableStream that yields string chunks (#486). Uses a local httpbin-like echo route so the cases don't depend on an external service.
A ReadableStream that yields string chunks was copied into a Uint8Array via set(), which coerces each character to NaN and writes a zero byte. The body went out with a correct Content-Length but filled with zeros. Normalize each chunk through TextEncoder before concatenating, and apply the same handling to Node Readable streams and other async iterables. Fixes #486
github-actions
Bot
added
t-tooling
Pijukatel
approved these changes
Jun 22, 2026
| chunks.push(toUint8Array(chunk)); | ||
| } | ||
| return { body: typedArray, type: '' }; | ||
| return { body: concatUint8Arrays(chunks), type: '' }; |
Contributor
There was a problem hiding this comment.
Just wondering how the receiver of this stream knows what character set was used to encode the stream
https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Type
Member
Author
There was a problem hiding this comment.
Good point 👍 Encoding the string-based body with UTF-8 is done in undici (Node.js's native fetch implementation) as well (see source). Since we're going for compatibility with this implementation, it's imo a strong enough argument (although I see the issue too now).
Anyway, imo in 2026, it's safe to expect UTF-8 pretty much anywhere, if not specified otherwise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #486.
A request body backed by a
ReadableStreamthat yields string chunks was copied into aUint8Arrayviaset(). Sinceset()coerces each character toNaN→0, the body went out with a correctContent-Lengthbut filled entirely with zero bytes. This surfaced downstream in Crawlee as a POST going out ascontent-length: 58followed by 580x00bytes.