Indexeddb chunk store in browser

[KayleePop]

Here's the chunk store used https://github.com/KayleePop/idbkv-chunk-store
which is built on this IndexedDB wrapper https://github.com/KayleePop/idb-kv

npm run size shows this branch's bundle to be 85,873 bytes which compared to the 85,252 bytes of master is a .7% increase or 621 bytes larger

 

Here's a deployment of instant.io using this PR's code for demo and smoke testing https://instant-io-idbkv.glitch.me/ (glitch.com is awesome)

Things to test (please share results 😄 ):

• Very large files
• Concurrent downloads of different torrents
• Reload tab to load persisted data from disk
  • After download has finished
  • While download is still going
• Multiple browser tabs (they use the same indexeddb database)
• RAM usage
• Speed compared to https://instant.io
  • Speed of completing download
  • downloading the file to disk
• Streaming
• Different browsers

EDIT: The default-chunk-store is no longer changed in this PR

[KayleePop]

One thing I noticed is that everything breaks in Firefox's private browsing mode (ctrl+shift+P). This is because Firefox simply disables indexeddb in that mode.

Maybe this could be fixed with another layered abstract chunk store (like immediate-chunk-store) that reverts to memory-chunk-store on failure.

[KayleePop]

A potential optimization is to increase or remove the concurrency limit on verification here https://github.com/webtorrent/webtorrent/blob/master/lib/torrent.js#L582 (FILESYSTEM_CONCURRENCY is set to 2)

idb-kv uses batching to improve the performance of indexeddb as each transaction adds a lot of overhead. Increasing the limit would allow more reads to make it into each batch, and it might speed up store verification.

EDIT: I added this. The benefit was so drastic that there's no reason not to. I was seeing more than a 2x speed up of verification.

EDIT2: This would load the entire torrent into memory for verification I think. The fix would be to change this from Infinity to something like a gigabyte of data at a time. (1024 * 1024 * 1024 / torrent.pieceLength)

[DiegoRBaquero]

This had been suggest before, making the default for the browser a different other than memory, but we'd rather offer extensibility than by default fill the disk and maybe never get cleaned. I'm just not convinced

[KayleePop]

What about just the name option and concurrency changes?

[DiegoRBaquero]

Sure, for now, and we can keep the "debate" open

[KayleePop]

I removed the commit that changed the default store.

[DiegoRBaquero]

LGTM