Feature Request: state object

[retrohacker]

This is an attempt to standardize how applications resume a WebTorrent client after a restart.

WebTorrent Desktop is manually doing a subset of this already: webtorrent/webtorrent-desktop#46

This proposal introduces the state object. This object contains the subset of a clients state necessary to create a new client downloading the same torrent files in the same way.

I would be happy to do the legwork of implementing this if it seems like a sane thing to do.

I'd like to make the following API additions:

Client API

client = new WebTorrent([opts])

Create a new WebTorrent instance.

If opts is specified, then the default options (shown below) will be overridden.

{
  dht: Boolean|Object,     // Enable DHT (default=true), or options object for DHT
  maxConns: Number,        // Max number of connections per torrent (default=55)
  nodeId: String|Buffer,   // DHT protocol node ID (default=randomly generated)
  peerId: String|Buffer,   // Wire protocol peer ID (default=randomly generated)
  tracker: Boolean|Object,  // Enable trackers (default=true), or options object for Tracker
  state: Object // Start WebTorrent in the supplied state (default = null)
}

For possible values of opts.dht see the
bittorrent-dht documentation.

For possible values of opts.tracker see the
bittorrent-tracker documentation.

client.getState()

Returns the state of the client.

client.on('state', function(state) {})

Emitted whenever a client's state object is updated.

State API

State is an object that reflects the state of a WebTorrent client. This object can be used to construct a new WebTorrent client.

state.torrents

A list of all torrents being tracked by the client

state.torrents[i].files

All files inside of a torrent. null if we don't yet know the files. This isn't enforced in any way when supplied to the client constructor as an option but is used to set the state of the files in a torrent if they match any of the files in this list.

state.torrents[i].files[i].selected

Is this file being downloaded?

state.torrents[i].files[i].name

The name of the file, as specified by the torrent.

state.torrents[i].files[i].path

The path of the file, as specified by the torrent.

state.torrents[i].pieces

All the pieces inside of a torrent. null if we don't yet know how many pieces there are. This isn't enforced in any way when supplied to the client constructor as an option, but is used to set the state of pieces in a torrent if they match any of the pieces in this list.

state.torrents[i].pieces[i].priority

The priority of the torrent piece.

state.torrents[i].path

The path that the torrent is being downloaded to.

state.torrents[i].torrentId

A torrent file if known, otherwise an infohash.

state.torrents[i].paused

Whether or not a torrent is currently paused.

[feross]

This seems reasonable. It will definitely simplify things in webtorrent-desktop.

A few suggestions:

• There should be a state.torrents[i].infoHash. Torrent info hash if known, otherwise null.
• I would change state.torrents[i].torrentId so that it doesn't promise to only be a torrent file or infohash. It could be anything. (For example, if the user adds an http url to a .torrent file, then state.torrents[i].torrentId should be the url. Because there's no known info hash until the torrent file is downloaded, parsed, etc. Also, once we know the info hash, we can make this field always return a torrent file, for consistency.)
• What's the point of state.torrents[i].pieces? I think we should allow the client to re-verify the files on disk instead of passing this in, since the files could have changed, and we already a modTimes API for skipping re-verification of unchanged files.
• Should we consider putting the "selections" into the state object. These are what determine the byte ranges (i.e. files, parts of files, etc.) and priority that the torrent pieces will get downloaded with. This is tricky because if client1 had called file.createReadStream() but never finished consuming it, do we necessarily want client2 to continue fetching that range with high-priority even though there's no stream consuming it now? Perhaps, only selections due to file.select() and file.deselect() should be preserved? Or perhaps selections due to file.createReadStream() should be filtered out prior to creating the state object?

[retrohacker]

state.torrents[i].pieces and state.torrents[i].files should not be prescriptive, the client will re-verify everything and use the metadata to build a list of files and pieces. Once the client has verified the pieces and has a list of files, it will check the state.torrents[i].pieces array to set the priority of each piece and the state.torrents[i].files array to select/deselect files before starting the download.

I don't know what selections are. It sounds like they are what implement piece priority and file select/deselect behind the scenes?

[feross]

Yes, selections are how ranges of bytes in the torrent are prioritized behind-the-scenes. It's basically an array of objects that specify start and end bytes. You can see the default selection that's added automatically for all torrents here: https://github.com/feross/webtorrent/blob/master/lib/torrent.js#L613-L616

Unless I'm missing something, I don't think state.torrents[i].pieces makes sense because pieces have 3 states: missing, reserved, or downloaded. Reserving happens when we've asked a peer on the network for the piece -- this should not be persisted.

I'm not sure exactly what's right to do here, but I think that storing the full selections array has some big downsides. For one, it doesn't make sense to persist the particular part of the video that the user was watching in WebTorrent Desktop when the app was quit.

As a first pass, what about if we just persist calls to file.select() and file.deselect()?

[feross]

@dcposch Do you have thoughts on the design of this?

[retrohacker]

Ah, understood, then yes pieces makes no sense here.

I'm +1 on only persisting file state for now. Every application will probably have its own way of prioritizing/deprioritizing pieces that need to be downloaded, so it might make sense to leave persisting that logic to the application entirely.

[lbeschastny]

Shouldn't DHT nodes be persisted as well?

According to bittorrent-dht docs:

The DHT nodes, in particular, are useful for persisting the DHT to disk between restarts of a BitTorrent client (as recommended by the spec).