Continuous/streaming SHA1 hashing

[astro]

I've had good experience with this in Bitford and would like to work on it. But first I would like to hear your opinion. My arguments for it:

• Shorter and more evenly distributed 100% CPU situations. It will smoothen things much more even with Web Workers, because...
• I just discovered that the 31C3 video torrents come with a piece length of 16 MB. After receiving the last few 16 KB chunks we will be able to deliver the media to the player frontend much faster if there were only a few KB to hash instead of the full piece.

As neither Rusha nor subtle support streaming SHA1 we will have to emulate it in the browser for now. Still, the node.js crypto module can do it well. Rusha could probably be modified in the future. The subtle interface only provides the oneshot method. :-(

[feross]

I agree that this is worth exploring.

One thought, though: After all the SHA-1 perf fixes I made last night, it only takes 1s to take a 250MB file, split it into 1000 pieces, and hash each piece (in browsers with the web crypto API). You can try it at https://instant.io.

Given this, do you still think it's worth optimizing the hash time of a 16MB piece? It could be. But I'd first get a benchmark and see if it's worth it before you spend too much time :) If it's <100ms, we probably don't need to optimize this.

[feross]

The point about more evenly distributing the CPU computation is interesting though, and this might be important for playing back video smoothly (and not freezing the browser).

But based on what I've been able to find online, the WebCrypto API runs off the main thread. So at least in Chrome (and soon Firefox) even hashing large data shouldn't block the event loop.