Enable bitswap to handle more requests

[aschmahmann]

At the moment go-bitswap has fixed worker count limits for areas where we expect resource limits such as the reading from the blockstore, packaging up messages and responding to peers.

We should either make the worker limits configurable from the config file, or allow them to dynamically shrink/grow to meet network demands.

Proposed steps for now are:

• Make each of go-bitswaps worker counts passable as options
• Expose metrics in prometheus for tracking the size of the backlog for each of the workers to help users figure out if they should be increasing their worker counts
• Make a section in the config file for setting internal/unstable parameters of components
  • Proposed: Internal.Bitswap.*
  • Document how a user might use the prometheus metrics to decide to increase their worker counts

[aschmahmann]

We currently have some hacked together branches that cover a bunch of the work here, but they need to be transitioned into something usable in go-ipfs master.

• go-ipfs: 236f890
• go-bitswap: ipfs/go-bitswap@78aabea
• go-peertaskqueue: https://github.com/ipfs/go-peertaskqueue/tree/feat/track-ongoing-jobs (ipfs/go-peertaskqueue@2b501cb)
  • Required to actually land the changes with configurable workers, but requires some testing/benchmarking

Next steps (in order):

• Add config file support for go-bitswap configurability changes (let's do this first)
• Expose metrics for tracking which queues are backed up and some docs related to the config options (i.e. a "how to tune" section)
• Test with the existing peertaskqueue changes (we'll be able to test later)
  • Ideally these would be testground tests, but doing tests on some production nodes and some local nodes to gauge performance should do an ok job as well

[aschmahmann]

closed by #8268