Improve BLS multi-thread worker pool strategy

[dapplion]

Motivation

Some weeks ago we landed a naive version of a worker thread pool to verify BLS signatures. It did the essential well, to help the node breathe and survive in Prater. But some obvious work can be done to improve the current impl.

Description

Extend usage

Use the BLS worker pool for all gossip validation, not only attestations.

Reduce latency cost

Communication between the main thread and workers has significant latency, similar to the cost of a single BLS sig verification. To reduce the impact of this latency in the throughput of validation I've update the pool to allow batching work. It will send packages of multiple groups of signature sets to be verified individually but sending the whole package once. This splits the latency cost between multiple operation. However this strategy only works if all workers are busy so the queue can accumulate work.

Add metrics

By using our own custom Thread Pool instead of threads implementation it's good to add some metrics to track:

• Average signature verification time. Is blst as efficient as expected in production?
• Job wait time. Does the queue structure delay signature verification significantly?
• Job package size. Does the "reduce latency cost" strategy above work? Are work packages big or too small?

[codeclimate]

Code Climate has analyzed commit 9baf22a and detected 0 issues on this pull request.

View more on Code Climate.

[wemeetagain]

🚀