Move SHA1 hashing to another process/worker

[feross]

Creating SHA1 hashes for large multi-gigabyte files can take up a lot of CPU time.

• In the browser, we should investigate moving this work to a web worker with transferrable objects.
• In node, we should investigate spawning worker processes to do hashing.

[astro]

Transferrable objects worked well for Bitford:

• https://github.com/astro/bitford/blob/master/src/store.js#L616
• https://github.com/astro/bitford/blob/master/src/sha1-worker.js#L18

You're always welcome to copy my code for groundwork. We're implementing file-sharing here anyway. :-)

[astro]

It is not clear to me how to integrate this into the browserify process. Maybe with workerify?

[feross]

Managed to make client.seed much faster by skipping the piece verification in storage.js. It's unnecessary since we're the first seeder of a new file.

Before:

createTorrent.parseInput: 0.598ms
index.js:206 createTorrent: 6654.848ms
index.js:211 add: 186.781ms
index.js:220 tasks (storage.load): 6906.296ms
index.js:224 seed: 13752.465ms

After:

createTorrent.parseInput: 0.626ms
index.js:209 createTorrent: 6656.904ms
index.js:214 add: 419.079ms
index.js:223 tasks (storage.load): 256.499ms
index.js:227 seed: 7337.118ms

Total time for 250MB file almost cut in half. 13s to 7s.

[feross]

With this PR (michaelrhodes/simple-sha1#1) to simple-sha1, the time to prepare a 250MB file is only 2 seconds.

[feross]

Aaaand, using the web crypto API gets us down to <1 second to seed a 250MB file!

PR out: michaelrhodes/simple-sha1#3

We're now officially in the territory of C speeds. We probably can punt on web workers for a while. This is so much faster than it was before (13.7 seconds).

[feross]

Released 0.22.0 using simple-sha1 for MUCH faster perf in the browser.

[Svenskunganka]

@FerrOS This is amazing! Really needed! We should try to push the larger BitTorrent client developers to implement WebRTC as WebTorrent is revolutionary to BitTorrent!

[feross]

@Svenskunganka Yep, we need to apply pressure and/or send them pull requests that add WebRTC support. Do you know any client developers?

[Svenskunganka]

@FerrOS Unfortunately not. As for µTorrent, it seems the only thing we can do there is to make a feature request on their forums...

[alberto-esposito]

we could also try the other way around: create a desktop and mobile app with node.js/Meteor. Like Atom.io did. It's not very difficult.

Being desktop based, I could create a bridge between the standard torrent protocol and webtorrent, using Unix Pipes on linux.

[Svenskunganka]

@servermeta that won't cut it in the long run as people are still going to use µTorrent, Transmission, Vuze and all the other large BitTorrent clients. The only way to get to the large audience is if those clients adds support for WebRTC. I imagine only seedbox users would be interested in using a bridge.

[alberto-esposito]

the idea is to substitute them (exaggerated expectation). We could have the momentum, as webtorrent is firewall proof.

[MzHub]

As for µTorrent, it seems the only thing we can do there is to make a feature request on their forums...

They seem to be quite active with the responses though, so I believe they are reading the requests. A while ago I was already writing a request there, but then I hesitated, since I don't know if that is on the agenda at this point. I thought it might be better to wait for the BEP and other such things that might persuade them, or help with integrating WebRTC (examples?) that are still pending. What do you think?

[Svenskunganka]

@MzHub maybe we can open their eyes with Feross awesome presentation from JSConf Asia? https://www.youtube.com/watch?v=kxHRATfvnlw

It's very educational and really brings forth the weight and importance of WebRTC support!

[cmawhorter]

just passing through but npm webworker-threads might be of interest

[arestov]

did you see?
https://github.com/pgriess/node-webworker

[josephfrazier]

It looks like threads.js (npm install threads) has most of what is needed here:

• "Use different APIs (web worker, node child_process) transparently"
• Transferable Objects

However, it doesn't provide the ability to use require in the browser, but that appears to be an upcoming feature. I wonder if webworkify could help out with that.

[feross]

Closing this. See: #1233 (comment) for reasoning.