Take state writing off the critical path

[Ekleog-NEAR]

Currently, we flush the state to disk as part of the critical path here:

nearcore/runtime/runtime/src/lib.rs

Line 1466 in d7e1232

self.validate_apply_state_update(

However, since in-memory trie, we should not need to write the state to disk before processing the next block. Instead, we could publish the block before writing to disk, and write to disk asynchronously. That should give us a likely 10-50% performance boost. If my recollection serves correctly, we sometimes spend 300ms writing state to disk in production, which would hopefully give us a roughly 30% single-shard performance boost, by moving these 300ms to a background thread.

This is a project we had considered a few times, without being able to act on it due to trie complexity. With in-memory trie, it should become easier to act on now — we already eliminated disk reads, and disk writes should be able to go soon off the critical path too.

[nagisa]

This reads a lot like a duplicate of #11118. Since you're working on it, I don't mind whichever issue you want to keep for tracking this work, but consider closing the other one.

[Ekleog-NEAR]

That's exactly the one, thank you! I was sure there was already an issue open for it but failed to find it in the issue list, will close this one :)