Partial pieces are kept in memory leading to high RAM usage

[feross]

We use more RAM than we need to for a large files which are downloading quickly because we keep pieces in memory until they're fully downloaded at which point we write them to disk.

I believe there's a good chance that all the reports of high RAM usage on WebTorrent Desktop are due to this design decision. Example: webtorrent/webtorrent-desktop#1877 Can we explore how we might limit the number of partial pieces in memory or somehow not keep partial pieces in memory at all?

[datnguyencse]

@feross Why don't we use IndexedDB or Cache Store instead of https://github.com/mafintosh/memory-chunk-store
Take a look: https://github.com/datnguyencse/cache-storage-chunk-store

[fivedots]

I'm working on a new web storage API that seems to fit your needs pretty well! It's called NativeIO, and it's currently available behind a flag on Chrome. I suspect that you could avoid keeping partial pieces in memory, by writing them to disk through NativeIO and directly updating them as more data arrives.

We are very interested in getting feedback for it as we approach an origin trial, so please let me know if trying it out sounds interesting to you! I'm happy to have chat or discussed it further in a separate issue.

[feross]

Semi-related to this issue: @jhiesey and I just fixed an issue in block-stream2 that significantly reduced memory usage on https://instant.io, almost by half. See: https://github.com/substack/block-stream2/pull/4

EDIT: Just published a new webtorrent with the latest block-stream2 fix. webtorrent 0.112.4.

[redbrain]

Perhaps integrating a library such as StreamSaver.js into WebTorrent is viable so Blobs don't have to be created.

[ThaUnknown]

that is an unrelated issue to this one, and this is already being worked on with store backpressure

[redbrain]

I'm confused by that last sentence. Which problem is being solved by backpressuring, and do both problems have an issue here?

[github-actions]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?