[Feature request] Use several zcash-msghand threads

[ytrezq]

Using a single thread currently cap syncing at 4Mb per seconds on a Cascade Lake cpu (one of the fastest per thread performance in the world). But if different unrelated messages could be processed in parallel nothing would prevent things going 10 times faster for those who have a optical fibre connection.

This would be especially useful for faster syncing (since the process is cpu bound because of this thread).
Of course, I’m talking with the fast syncing option enabled.

[tromer]

Syncing up seems inherently sequential: the validity of every transaction depends on the latest state after taking into account all prior transactions (within this block and prior ones).

This can be partially parallelized: some parts of the transaction verification don't depend on prior state (e.g., checking signatures). But it may be architecturally difficult to separate the two and orchestrate everything reliably, including correctly handling rollbacks.

[Eirik0]

Yes, @tromer is correct that syncing blocks is an inherently sequential operation. While there may be a way to parallelize this, it is unlikely that this will be included in zcashd in the short-term.

[ytrezq]

@Eirik0 : what about thinking that most of the time the info provided by a node is ok (treat everything as valid) in order to go ahead for the next transactions and roll back if a transaction appears to be finally invalid ?

This would allows several threads while fetching them transactions in the right order but in batch (which requires a batch of transactions to complete in order to start the next one).

[daira]

It's quite a bit more complicated than that. In order to optimistically validate a transaction, you need to know what its transparent inputs are, which requires you to have validated the transaction(s) with corresponding transparent outputs. There are opportunities for parallelism if you could completely redesign how validation is implemented, but realistically we're not going to do that just for t-addresses: we're more likely to remove transparent address support entirely and (if we're funded to do so) use recursive SNARKs to scale the validation of the shielded protocol.

You can handle signature and proof validation optimistically. However, for transactions before the latest known checkpoint, we already assume that signatures and proofs are valid. So this wouldn't help syncing.