fix: Database dumps do not work on large databases#107
Open
misteromb wants to merge 2 commits intoouterbase:mainfrom
Open
fix: Database dumps do not work on large databases#107misteromb wants to merge 2 commits intoouterbase:mainfrom
misteromb wants to merge 2 commits intoouterbase:mainfrom
Conversation
outerbase#59) Replace in-memory string concatenation with ReadableStream-based streaming and batched SELECT queries (LIMIT/OFFSET with batch size of 5000 rows). This prevents memory exhaustion on large databases by: - Using ReadableStream to write chunks incrementally - Fetching row counts per table and iterating in batches - Adding Transfer-Encoding: chunked header for streaming response Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ase dumps (closes outerbase#59) Improve the streaming dump implementation by: - Using an async generator with pull-based ReadableStream for proper backpressure (consumer controls data flow) - Adding breathing intervals (sleep(0) every 100 rows) to yield control back to the event loop and avoid exceeding CF Workers CPU time limits - Extracting dump generation into a reusable async generator function Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #59
Changes
This PR addresses the issue described in #59: Database dumps do not work on large databases
Testing
Automated fix by bounty-hunter agent