Efficient large file upload?

[rchrd2]

Hello,

I attempted to upload a large file (approx. 100mb), and my browser became very slow.

I wish I could provide more details, but have large files been considered in this implementation? What is the expectation?

[raucao]

Currently, large file uploads are not something that the library has been optimized for. It would load the whole file in memory and do a single HTTP request, which will likely time out, before it can finish. You can increase that timeout for this reason (as e.g. Sharesome does, because you oftentimes want to upload e.g. larger animated gifs or zip files and whatnot).

As of now, large file uploads are a general hole in the RS spec, as it's currently not really possible to have it in the protocol without prescribing implementation logic, and it's not something that HTTP offers by itself. See spec issues remotestorage/spec#124 and remotestorage/spec#131 as well. Maybe you have an idea for a solution even...

[rchrd2]

Thanks for the response. I'll need to dig into the RemoteStorage client implementation to provide more concrete suggestions, but here's some ideas.

Without solving all the problems, one improvement, might be to circumvent holding the whole file in memory when uploading. Like passing the stream directly to client.storeFile, instead of the raw value. Then this could be passed separately to the remote and cache storages.

I am also curious what performance characteristics of IndexDB is for large files. If there is a hard-limit there, then it's probably worth mentioning in the docs, because this is a constraint in the library.

[raucao]

Without solving all the problems, one improvement, might be to circumvent holding the whole file in memory when uploading. Like passing the stream directly to client.storeFile, instead of the raw value. Then this could be passed separately to the remote and cache storages.

We'd have to switch to Fetch first then, which would break compat with a ton of browsers, and the spec is not even finished yet. Or maybe XMLHttpRequest accepts a stream for sending requests and I don't know about it?

I am also curious what performance characteristics of IndexDB is for large files. If there is a hard-limit there, then it's probably worth mentioning in the docs, because this is a constraint in the library.

Afaik there is no hard limit. But I think it makes sense to explain somewhere, that the library is not optimized for storing large files, but for application data.

Caching large files like that in your browser is not a great idea in the first place, because browser caches can not be trusted with persistence, in the same way your local filesystem can. Both browsers/OSs as well as users can periodically delete their caches, and you wouldn't want to randomly have to re-download your RAW photo library when that happens. There used to be a FileSystem API for this use case (which would also give you URLs, so you could e.g. play videos straight from the sandbox filesystem), but after only having been implemented in Chrome at the time, it has eventually been forgotten and discontinued. I think it's technically still available in Chrome, but no other other browser.

[raucao]

Just an update, because I found this open issue: we're in the process of adding support for resumable/large file uploads: https://community.remotestorage.io/t/resumable-file-uploads/461