Handle large files better in the browser

[feross]

_Moving this issue from webtorrent/instant.io#3 to here, for better visibility._

After lots of work on fixing memory leaks, I was able to seed a 1GB video from Firefox and stream it to Chrome! But there are still issues:

• Can't seed files larger than 500MB in Chrome due to a chromium bug.
• Can't seed or download files larger than 1GB because of a buffer limitation. This limitation comes from V8 and we could probably lift this limit in feross/buffer and improve the situation in Firefox.

Besides fixing the above issues, future work should include using browser storage (#86) to get this data out of memory.

---

Update (5/26/2015):

• Chrome supports seeding and video streaming (i.e. downloading) files up to 1GB now, due to this PR that implements streaming blob reading in the filestream package! Downloading via the download blob link is still limited to 500MB until the chromium bug is fixed.

[kawa-]

I wondered how mega.nz dealing with the problem and found http://stackoverflow.com/questions/15994554/download-files-like-mega-co-nz and http://stackoverflow.com/questions/21019796/how-mega-does-to-save-a-file-to-harddrive-without-requiring-permission . Looks like they are doing every ways (Filesystem API, Adobe Flash, IndexedDB, and so on) in each browsers. How about mega.nz's solution?

[kawa-]

But actually, I tried to upload 183.1MB file to mega.nz by Mac OSX Chrome (42.0.2311.135 (64-bit)), it crashed :(

[kawa-]

[update] MEGA does not solve it with plugin-free. I tried downloading over 1GB file from MEGA by Firefox, MEGA required addon. Looks like there's so far no compatible way to manage large data within a browser.

[jakefb]

Have you seen this https://github.com/eligrey/FileSaver.js/? I'm interested in downloading a blob without displaying a link on the page.

[kawa-]

@jakefb I've not tried FileSaver.js, but they say they have a size limitation. Firefox 800MiB, Chrome 500MiB from README.md.

[feross]

Chrome supports seeding and video streaming (i.e. downloading) files up to 1GB now, due to this PR that implements streaming blob reading in the filestream package! Downloading via the download blob link is still limited to 500MB until the chromium bug is fixed.

[aniquetahir]

@kawa- I can download a 1.8GB file from mega just fine using my Firefox(dev)
https://mega.co.nz/#F!81skTRYK!Ybk7gc4oQfbOM_5P3POzcw

[kawa-]

@aniquetahir Great, also I've successfully downloaded the file of deepin_2014.3_amd64.iso (1.9GB) from your link in my Firefox ESR (31.7.0, yes it's old) without any addons.
MEGA using blob. I wonder if MEGA manages them.

[feross]

WebTorrent should be able to handle downloading a 1.8GB file just fine in Firefox once we lift the 1GB limitation in buffer. I'll look at doing that now.

[feross]

I just released a new version of the browserify buffer that has an increased max size limit of 2GB, instead of 1GB.

I just tested https://instant.io with a 1.5 GB file. Firefox (nightly) can seed and download it with no problems!

Chrome still can't seed files larger than 500GB because of the blob bug, but it can stream the 1.5 GB video/audio file without problems now! The download link doesn't work because of the same blob bug.

The Chrome team is actively working on that bug, so it's just a matter of time. 👍

[alexeisavca]

HTML5 filesystem API?

[feross]

@alexeisavca The filesystem API is Chrome only, and it's non-standard and deprecated (and might be removed from Chrome eventually). I'd rather just wait for chrome to fix the blob bug. I've seen lots of commit activity on it recently – it'll be fixed soon.

[ericwooley]

@alexeisavca I actually attempted a workaround using the filesystem api for a chrome extension, and ran into more problems. The 500mb cap applies to all your blobs, so you can make a buffer splice it into blobs, and write it, but once you have created 500mb of them, you run into the same issue. So what you need to do is release the memory from blobs which are already written, but you can't.

We sort of hacked it using https://developer.mozilla.org/en-US/docs/Web/API/URL/revokeObjectURL, (even though we didn't create an object url out of the blob) and it worked about half the time. The half the time it didn't work, it crashed the browser.

I saw someone claimed some kind of workaround writing from indexdb, but I chose to stop trying to implement half baked solutions and wait for the bug to be fixed.

[qgustavor]

Does anyone tried using Streams API for downloading files in Chrome? It allows download huge files but seems an active service worker is required. A test where a 1 GB random file is generated and download: rawgit link.

[feross]

The Streams API won't help us with generating a Blob to use with a download link, since Blobs still can't be more than 500MB in Chrome. But it's a good thought.

[micovi]

Have you guys looked into what those guys did handling big files? reep.io is their web page and they have the code open source in github here https://github.com/KodeKraftwerk/reepio
Maybe this can help bypassing the google chrome issues.

[qgustavor]

@feross Just don't use blobs: stream the download directly to the disk or temporally store chunks in IndexedDB then download creating in a Service Worker a stream using those stored chunks as source.
@ovidiualx It uses the deprecated FileSystem API, which @feross don't want to use.

[ericwooley]

@qgustavor How do you steam directly to disk?

And can you explain your indexdDB solution more in depth?

[qgustavor]

@ericwooley I wrote directme.ga when Streams API was implemented just on Canary, so maybe the spec changed and it's now broken. I will say how it works (or worked), so the only difference will be instead of using mega as a source just use WebTorrent as a source (both libraries uses Node streams).

Let's download some file, let's say Big Buck Bunny. Instead of downloading chunks (via HTTP or WebRTC) and storing it in a big blob it's used to feed a ReadableSteam, which is downloaded using Service Workers and finally being handled as a normal HTTP download, thus stored in the disk, reducing memory usage and using the disk as the storage.

As ServiceWorkers still can't connect to WebRTC peers IndexedDB is used as a intermediate storage between the download page and the service worker which will create the download stream. It can be also used to allow the WebTorrent download start before the HTTP download.

Edit: seems jimmywarting implemented an library which simplifies a lot this and works even on HTTP: StreamSaver.js (from this comment).

[jimmywarting]

Whooot! @qgustavor I thought i was the only one that had that idea with service worker + fetch
cheers 🍺

I knew this was going to be possible long before cuz i could do it in the main thread with response + stream from an existing api that returned a stream but wasn't able to save it until it was possible to construct it in a service worker

[qgustavor]

@jimmywarting I never trought I was the only only which got this idea, but maybe one of the first which implemented it, as I don't found nothing about it when I created that website, just the spec. I think those who idealized and wrote Streams API got that idea first.
For the second part, sorry, I could not understand your English.

[jimmywarting]

Sorry för my bad English. Yea, they probably thought they needed a way to save large data. But they didn't implement a way create a write stream to the filesystem

Well, I'm using a different communication layer (postmessage) so nothing is stored in the browser

[jimmywarting]

Did put up an simple example of how to do it in the readme

const client = new WebTorrent()
const torrentId = 'magnet:?xt=urn:btih:6a9759bffd5c0af65319979fb7832189f4f3c35d&dn=sintel.mp4&tr=wss%3A%2F%2Ftracker.btorrent.xyz&tr=wss%3A%2F%2Ftracker.fastcast.nz&tr=wss%3A%2F%2Ftracker.openwebtorrent.com&tr=wss%3A%2F%2Ftracker.webtorrent.io&ws=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2Fsintel-1024-surround.mp4'
// Sintel, a free, Creative Commons movie

client.add(torrentId, torrent => {
    // Download the first file

    const file = torrent.files[0]
    const fileStream = streamSaver.createWriteStream(file.name)
    const writeStream = fileStream.getWriter()

    // Unfortunately we have two different stream protocol so we can't pipe.
    file.createReadStream()
        .on('data', data => writeStream.write(data))
        .on('end', () => writeStream.close())
})

[feross]

@jimmywarting Cool module, and including the example is very helpful 👍

[jimmywarting]

Thanks

[feross]

FYI Chrome finally fixed the 500MB blob size limitation! https://bugs.chromium.org/p/chromium/issues/detail?id=375297#c107

This is shipping in tomorrow's Chrome Canary. We can test it out there and see if things work better now!

[guest271314]

@feross @jimmywarting Can webtorrent or StreamSaver.js handle downloading 1GB+ files Download large data stream (> 1Gb) using javascript ?

[kocoten1992]

Firefox 50 able to seed to 4GB (maybe more)
Chrome 57 ~1GB, crash on 1.2GB

[jimmywarting]

@guest271314 StreamSaver is able to save any size. But it has one limitation in that the service worker dies/restart after 10m. jimmywarting/StreamSaver.js#39

So you need to be quick

[andreapaiola]

What is the actual upper limit?

[feross]

Chrome supports super large files now (>2GB). I'm closing this issue.