Allow transformation of webseed http request

[guanzo]

What version of WebTorrent?
"webtorrent": "^0.102.4"

What operating system and Node.js version?
OS: Windows
Node.js version: v8.9.1

What browser and version? (if using WebTorrent in the browser)
Browser: Chrome
Version: 69.0.3497.92 (Official Build) (64-bit)

Feature request: Allow some way for the user to transform the webseed http request.

Tangentially related issue: #1193

I'm running into an issue regarding Chrome caching range requests.

Full context here: https://stackoverflow.com/questions/52349984/chrome-cors-requests-are-faster-when-cache-is-disabled

Another explanation of the problem here: https://stackoverflow.com/questions/27513994/chrome-stalls-when-making-multiple-requests-to-same-resource

TL;DR: Chrome reuses the same TCP connection and explicitly waits for each range request to finish before executing the next one because the URL is the same per request, resulting in extremely poor network performance. Enable "Disable cache" to see the performance drastically improve.

One solution that brought the network performance from snail-speed (1s) back to "normal" (250ms) was to add cache-control: no-cache, no-store to webconn.js, like so

var opts = {
      url: url,
      method: 'GET',
      headers: {
        'user-agent': 'WebTorrent/' + VERSION + ' (https://webtorrent.io)',
        range: 'bytes=' + start + '-' + end,
        'cache-control': 'no-cache, no-store' // Added by me
      }
    }

To do this I have to modify code inside node_modules, so it'd be nice if webtorrent provided an API to transform http requests before they're sent.

[jimmywarting]

Would be nicer if you could set the cache control on Request instead of sending a extra header to the server.

I think we should just use fetch instead of simple-get

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[DiegoRBaquero]

Agree on this one with @jimmywarting , it'd be best to use Request cache options. I don't know how if node-fetch could perform everything on Node and fetch on the browser while maintaining the compatibility with browserify. Open to PRs

[jimmywarting]

Using fetch in browser will lower the dependency. Not so much for us since we already use some of the sub-dependencies.

node-fetch and browsers fetch are compatible of doing the same things on both sides

What you don't see in your dependency tree is Nodes core modules that becomes bundled with browserify like require('stream') actually becomes require('readable-stream') but readable-stream is not included in your package.json so you don't exactly know what you will get. and readable-stream requires buffer but that is also not something you will see in the package.json dependency tree b/c browserify takes care of that.

When requiring "simple-get" i think you will include all of this modules

stream-http: "^3.0.0"
builtin-status-codes: "^3.0.0",
inherits: "^2.0.1",
readable-stream: "^3.0.6",
string_decoder: "^1.1.1",
safe-buffer: "^5.1.2"
util-deprecate: "^1.0.1"
xtend: "^4.0.0"
buffer: "^5.2.1"
util-deprecate: "1.0.2"
once: "^1.3.1",
simple-concat: "^1.0.0"
url: "0.11.0"
"punycode": "1.3.2"
"querystring": "0.2.0"

that is something around ~72KB minified (~21KB gzip)

[jimmywarting]

I have actually been working on using fetch instead of simple-get inside webconn.js (I'm not going to make a PR)

class WebConn extends Wire {
  constructor (url, torrent) {
    super()

    this.url = url
    this._torrent = torrent
    this.webPeerId = sha1.sync(url)

    this._init()
  }

  _init () {
    this.setKeepAlive(true)

    this.once('handshake', (infoHash, peerId) => {
      if (this.destroyed) return
      this.handshake(infoHash, this.webPeerId)
      const numPieces = this._torrent.pieces.length
      const bitfield = new BitField(numPieces)
      bitfield.buffer.fill(255)
      for (let i = 0; i < 8; i++) {
        bitfield.set(numPieces + i, false)
      }
      this.bitfield(bitfield)
    })

    this.once('interested', () => {
      this.unchoke()
    })

    this.on('request', (pieceIndex, offset, length, callback) => {
      // debug('request pieceIndex=%d offset=%d length=%d', pieceIndex, offset, length)
      this.httpRequest(pieceIndex, offset, length, callback).catch(console.warn)
    })
  }

  async httpRequest (pieceIndex, offset, length, cb) {
    const pieceOffset = pieceIndex * this._torrent.pieceLength
    const rangeStart = pieceOffset + offset /* offset within whole torrent */
    const rangeEnd = rangeStart + length - 1
    // Web seed URL format:
    // For single-file torrents, make HTTP range requests directly to the web seed URL
    // For multi-file torrents, add the torrent folder and file name to the URL
    const files = this._torrent.files
    let requests
    if (files.length <= 1) {
      requests = [{
        url: this.url,
        start: rangeStart,
        end: rangeEnd
      }]
    } else {
      const requestedFiles = files.filter(file => {
        return file.offset <= rangeEnd && (file.offset + file.length) > rangeStart
      })
      if (requestedFiles.length < 1) {
        return cb(new Error('Could not find file corresponnding to web seed range request'))
      }

      requests = requestedFiles.map(requestedFile => {
        const fileEnd = requestedFile.offset + requestedFile.length - 1
        const url = this.url +
          (this.url[this.url.length - 1] === '/' ? '' : '/') +
          requestedFile.path
        return {
          url,
          fileOffsetInRange: Math.max(requestedFile.offset - rangeStart, 0),
          start: Math.max(rangeStart - requestedFile.offset, 0),
          end: Math.min(fileEnd, rangeEnd - requestedFile.offset)
        }
      })
    }

    const chunks = await Promise.all(requests.map(async request => {
      const url = request.url
      const start = request.start
      const end = request.end

      // debug(
      //   'Requesting url=%s pieceIndex=%d offset=%d length=%d start=%d end=%d',
      //   url, pieceIndex, offset, length, start, end
      // )

      const res = await fetch(url, {
        cache: 'no-store',
        headers: {
          'user-agent': `WebTorrent/${VERSION} (https://webtorrent.io)`,
          range: `bytes=${start}-${end}`
        }
      }).catch(console.error)

      const ab = await res.arrayBuffer().catch(console.error)

      return new Uint8Array(ab)
    }))

    cb(null, concat(chunks))
  }

  destroy () {
    super.destroy()
    this._torrent = null
  }
}

module.exports = WebConn