fix: deadlock

[SWvheerden]

Description

This fixes a deadlock in the code.

The RwLock allows a mutex to exist that allows concurrent reads but blocks concurrent writes, this behaves differently depending on the OS.
On Linux systems: The lock favors reads, meaning that as long as there exists a reader lock, it will allow a new reader lock to open. This means that the system can cause starvation of writers.
On Mac/Win the lock has equal ordering, this means that as soon as a writer queues for a lock, all additional readers will be blocked till after the writer has acquired and released its lock.

This behavior can be dangerous if recursive locks are used, as was the case here.

At about the same time, a block was submitted, and a template was constructed for a new miner.
The add_block process requires a write lock, while the block template process requires a read lock.
The template process was first in acquiring a lock on the read, followed shortly by the add_block on the blocking for a write. But the deadlock was caused after the add_block blocked for a write, the block template required an additional read_lock on the calculation of the mmr roots. And thus, the entire block_chain db class is deadlocked.

Fixes: #4668

[sdbondi]

Great find, the rust docs now (not sure since when) have an example of this

https://doc.rust-lang.org/stable/std/sync/struct.RwLock.html

[sdbondi]

Took the liberty of proving/understanding the spurious deadlock, running this on playpen sometimes deadlocks (timeout), but on my local Linux machine it always deadlocks.
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=c8d94e50a665a4c15d13e3b3886703d8

Same behaviour for tokio async locks.

This only happens when obtaining a read lock more than once at the same time from the same thread.

[stringhandler]

Nice find. I feel like there might be more