Skip to content

feat/stream large files#2464

Merged
forgetso merged 5 commits intomainfrom
feat/stream-large-files
Mar 25, 2026
Merged

feat/stream large files#2464
forgetso merged 5 commits intomainfrom
feat/stream-large-files

Conversation

@forgetso
Copy link
Copy Markdown
Member

  • Stream large files
  • docs(changeset): Stream large files to avoid loading entire file into RAM

Copilot AI review requested due to automatic review settings March 25, 2026 11:42
@forgetso forgetso enabled auto-merge (squash) March 25, 2026 11:45
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the @prosopo/util caching utilities to stream remote file downloads to disk (instead of buffering the entire response in memory), improving memory usage when working with large files.

Changes:

  • Change fetchWithETag to return a ReadableStream (plus optional contentLength) instead of a full string payload.
  • Update cacheFile / saveFileWithETag to write fetched content to disk via streaming.
  • Update unit tests and add a changeset entry documenting the streaming change.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File Description
packages/util/src/tests/cacheFile.unit.test.ts Updates tests to mock streamed content instead of string content.
packages/util/src/fetchWithEtag.ts Returns a response body stream (and parses content-length) instead of response.text().
packages/util/src/cacheFile.ts Streams writes to the cache file and makes saveFileWithETag async.
.changeset/better-maps-sin.md Adds a changeset entry for the util package release.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@forgetso forgetso disabled auto-merge March 25, 2026 11:48
@forgetso forgetso enabled auto-merge (squash) March 25, 2026 12:29
Copilot AI review requested due to automatic review settings March 25, 2026 12:53
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@forgetso forgetso merged commit 22bfee7 into main Mar 25, 2026
13 checks passed
@forgetso forgetso deleted the feat/stream-large-files branch March 25, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants