Chunked loader to reduce memory ceiling

[FL33TW00D]

Your peak memory usage when using WASM is your memory usage for the rest of time, see WebAssembly/design#1397

This means we need to load from IndexedDB in chunks, to reduce peak memory consumption.

[jozefchutka]

Not sure what exactly are the memory issues you face, but when working with large files in browser, Blob-s are very helpful.

To my experience, even when using dozen of GBs of data referenced via Blob, there are no memory peaks observed.

The downside of using Blobs are that:

1. access to the bytes is async
2. the data are readonly, unless you decide to slice the source Blob to create another Blob object (which is a synchronous operation).

I wonder if Blobs are suitable for your case.

[FL33TW00D]

Not sure what exactly are the memory issues you face, but when working with large files in browser, Blob-s are very helpful.

To my experience, even when using dozen of GBs of data referenced via Blob, there are no memory peaks observed.

The downside of using Blobs are that:

1. access to the bytes is async
2. the data are readonly, unless you decide to slice the source Blob to create another Blob object (which is a synchronous operation).

I wonder if Blobs are suitable for your case.

I need to look into this! I'll take a look at Blobs thanks for the reminder.