create blob/files backed up by fs when parsing #1640
jimmywarting
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Reproduction
request.blob()
request.formData()
Both of this will allocate memory of what could technically be very large files. As of now they are just stored in the memory.
ArrayBuffer
ArrayBuffer
to make it immutable. at this stage we have allocated 2x the amount of memory for just having both a ArrayBuffer and a cloned one in the Blob.node-fetch/src/body.js
Lines 133 to 140 in 6f72caa
something that would perhaps be better would be to create many smaller chunks of blobs and then create one blob containing all the other blobs:
At least then it will be able to gradually move chunks of data over to blobs and avoid having the need to allocate 1 very large arrayBuffer and GC the chunks earlier
But perhaps something better would be to just dump everything into a temporary file instead.
this is exactly the reason why i created createTemporaryBlob and files in
fetch-blob
So perhaps
.blob()
should just pipe the data to a file and return a blob instead?Some concerns:
might fill up disk space?
not what the developer wants?
createTemporaryBlob
requires FinalizationRegistry - node v14.6Alternative considerations
ppl could just use
createTemporaryBlob
themself instead of using.blob()
Also i don't really know any good reason why you would want to create a blob in the first place... you would just end up moving it to some location anyway so why not just pipe the data to the location of your own choosing?
fsPromises.writeFile('dest', res.body)
but this dose not work when parsing files in
formdata()
Beta Was this translation helpful? Give feedback.
All reactions