Use an adaptive number of threads in the verification queue

[rphmeier]

No description provided.

[rphmeier]

an ideal solution would probably just balance the rate of queue draining vs the rate of verification per thread and adaptively allocate the correct number of threads based on that.

this one uses some simple heuristics which seem to work decently well (although there's a bit more churn than I'd like) but it could be pretty simple to alter it to something more sophisticated.

[coveralls]

Coverage increased (+0.01%) to 86.452% when pulling 2d28c70 on adaptive_queue_threads into d205c08 on master.

[NikVolf]

please add tests
at least purely descriptive ones for add_verifier, remove_verifier

[coveralls]

Coverage decreased (-0.03%) to 86.411% when pulling abbf3b3 on adaptive_queue_threads into d205c08 on master.

[gavofyork]

still no tests :(

[NikVolf]

this heuristic might oscillate between, say, 4 and 5 threads
and each collect_garbage call would be creating threads or wait for one thread to join

[NikVolf]

can't extra 2 verifiers be added to the 7 verifiers
so that this check later will fail given ::num_cpus::get() == 8?

    if len == ::num_cpus::get() {
            return;
     }

[NikVolf]

ah, disregard this, they are added 1 by 1
still no need to call add_verifier and lock self.verifiers there once maximum is reached

[rphmeier]

@NikVolf I'll add tests once we figure out the right strategy for the adaptive threads. It's true that we don't want to be adding and removing threads all the time. Maybe a specialized thread pool where we allocate min(num_cpus, 8) threads and park/unpark them as necessary. I've found that with the current changes my block queue is almost always fully verified with less work being done.

I also think the ticks_per_adjustment might work better as 12 so it adjusts once every minute.

[rphmeier]

Ready for review again.

Changes:

• allocates the threads up front
• wakes verifiers and puts them to sleep as necessary
• simple scaling algorithm just compares amounts of verified and unverified blocks
• adjustment once every minute

[gavofyork]

tests?

[rphmeier]

tests :)

@NikVolf could you approve the changes on your review now that I've addressed that grumble?