Use spawn_blocking for dry-run requests

[Voxelot]

fixes: #752

Moves dry_run to a tokio::spawn_blocking call

Since the executor is mostly blocking code, issuing too many dry-run requests could impact more critical functions of the node (such as block production) if executed directly on the main worker pool.

Limits the number of concurrent dry_run requests by the CPU count

This is because in the worst case we will be CPU bound. While some txs may be state-access heavy and I/O bound, other txs could be CPU bound by using lots of crypto ops like hashing and ecrecover. To ensure we don't grind the system to a halt with hundreds of CPU intensive blocking threads, semaphore permits are used to rate limit the number of dry_run requests.

Removes async from the executor

Previously the executor methods were marked as async, even though there were no async calls. This posed a risk when calling the executor within spawn_blocking, as spawn_blocking is a sync context and must use block_on to call any futures. Using block_on is risky because it can easily lead to deadlocks when using async code. While block_on likely would've been safe to use in this one case, it could be the source of difficult-to-debug issues if more asynchronous code is added to the executor in the future.

[xgreenx]

The change looks clean. I like the reduction of async in our code=D Now, Executor acts as expected.

We can add a unit test where the semaphore's value is 2. We run 4 dry_run:

• First dry_run runs forever
• Second dry_run immediately executed
• Third dry_run runs forever
• Fourth dry_run is blocked because the limit is reached(test can wait for timeout).

It is easy to achieve with the mocking of the Executor trait and block or not based on the block_height.

But maybe it is overkill, up to you=)