crash when dropping a big file

[Fil]

dropping a big file (4GB) on the quick upload feature results in a crash

---

Clarifying tasks added by @lidel:

• dont crash when dropping >128MB in Chrome
• don't break silently in Firefox when Buffer max size is breached

[lidel]

Thank you for reporting this.
I was able to reproduce the problem and it looks bit different depending on which browser is used.

Background

Currently entire file is buffered in memory before being sent to external daemon, which introduces artificial limit on the max upload size.

Current Upload Size Limits

Chrome

This one looks really bad. An upload to an external node over js-ipfs-api breaks between 96MB and 128MB in size. It just kills my Chromium build:

Crashing Chrome is a known upstream bug, tracked in ipfs-inactive/js-ipfs-http-client#654

Firefox

Firefox works fine with 1GB file, but things bigger than 2GB hit language/lib limitations and fail silently with:

RangeError: Attempt to allocate Buffer larger than maximum size: 0x7fffffff bytes

This is something we should at least provide proper error handling for.

Possible Workarounds

While waiting for upstream fixes, we can do some work to address these problems:

• UX: detect uploads bigger than a limit of a current runtime and display feedback for the user instead of failing/crashing
• Investigate streaming versions of ipfs.add

[olizilla]

Streaming versions of ipfs.add need upstream fixes too.

• go-ipfs sometimes closes the stream preemptively ipfs-inactive/js-ipfs-http-client#339
• Passing a pull-stream to ipfs.add is interpreted as a callback ipfs-inactive/ipfs-postmsg-proxy#25
• ipfs.addPullStream() wont pull ipfs-inactive/ipfs-postmsg-proxy#26
• ipfs.files.addReadableStream() won't read streams ipfs-inactive/ipfs-postmsg-proxy#27

[lidel]

I partially addressed this in #483.

Streaming only helps if embedded js-ipfs node is used.
When external node is used js-ipfs-api seems to buffer entire thing in memory which causes the crash (even when streaming API is used). For that case we just refuse uploads over limit until the issue is fixed upstream (ipfs-inactive/js-ipfs-http-client#654).

I was unable to address Firefox error under 2GB as I am working on low-memory laptop, will look at it next week.