[legacy-framework] Maybe fix the stuck at _manifest issue: Increase stream highwatermark #697
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
flybayer
changed the title
dillondotzip
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This may fix this issue: blitz-js/legacy-framework#931
What are the changes and their implications?
I added a test looking for errors where streams corked themselves because of throughput and I found that happened within pumpify. Generally for the stream to be memory efficient we should have it manage to handle a low highwatermark scenario and pump is designed to handle this apparently although it appears as though this is not the case with pumpify for whatever reason so I have increased the highwatermark to a huge number.
We shouldn't necessarily be doing this as it might mean we use far more memory than we should or get into situations where we run out of memory but I have a good feeling it might solve this issue for now and we should investigate whether or not it is possible to lower the value to use a limited amount of memory instead of a huge amount. (Which should be possible if every stream handling thing we use including external libs are behaving themselves)
Checklist