Idea: Stream WebSeed instead of making thousand range request

[jimmywarting]

Instead of making 987 range request to download the Sintel Movie how about making just one request instead and have the WebSeed priorities the first pieces and all other peers starts from the end?

I'm certain that it would be faster then having to make multiple range requests to a server. A lot of packages are wasted (request and response headers takes up extra bytes, lookup time are wasted and all computing power to figure out the next range request to make up time too)

So when a WebSeed collide somewhere in the middle with what the peer are downloading - abort the streaming request

I would recommend using node-fetch and browser's native fetch implementation for this since both are compatible of streaming and aborting a request using AbortController both difference in a streaming API but a whatwg streams can easily be converted to a node stream so BlockStream can used on both of them

This is just a roughly idea of how it would work:

async function download (url, controller) {
  if (!controller) controller = new AbortControl()
  const signal = controller.signal
  const range = `bytes: ${ already_have_or_prioritised }-${ end_of_file_or_until_next_downloaded_piece }`
  const res = await fetch(url, {signal, headers: { range }})
  const stream = res.body.readable ? res.body : toNodeStream(res.body)
  const block = new BlockStream(length)
  
  let pieceIndex = figure_out_start_index() // 0

  stream.pipe(block)

  // maybe do something like this also?
  something.on('prioritise_changed', () => {
    controller.abort()
  })

  for await (let chunk of block) {
    const ok = await sha1(chunk)
    if (ok) {
      downloaded(pieceIndex)
      store.put(chunk)
      pieceIndex++
      if (other_peer_downloading(pieceIndex) || or_already_have_piece(pieceIndex)) {
        controller.abort()
        // maybe start from another range or get next file
      } else {
        // set the next 1-5 piece request are going to be downloaded by this WebSeed so
        // no other peer takes any chunk that this stream are going to download soon
      }
    } else {
      controller.abort() // WebSeed sending use the wrong thing
    }
  }
}

Ofc we would have to dealt with converting this above rough example with async/await and the asyncIterator to es6 somehow and also have a fallback for browser that don't have AbortController + streaming body (firefox has moz-chunked-arraybuffer and abort if we would like to go deep - but i bet they will implement it soon)

This strategy is turned on if there is at least one WebSeed, If there would be 2 or more WebSeed the other stream would start somewhere between the first webseed and the end of what the peer is downloading

[guanzo]

how about making just one request instead

Agreed. My webseed server can serve a 1MB image in 1 second in 1 request. So why does webtorrent instead make 62 requests of 16KB each, which takes like 10 seconds? It doesn't make sense to me. It would be nice if we could somehow specify a different piece length when requesting from the webseed.

[jimmywarting]

Browser support seems quite good now. All latest browser seems to support AbortController, fetch, res.body and ReadableStream

except for firefox that only lacks res.body and ReadableStream

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.