speed up state/block loading #5207
Conversation
When loading blocks and states from db/era, we currently redundantly check their CRC32 - for a state, this costs 50ms of loading time presently (110mb uncompressed size) on a decent laptop. * remove `maxDecompressedDbRecordSize` - not actually used on recent data since we store the framed format - also, we're in luck: we blew past the limit quite some time ago * fix obsolete exception-based error checking * avoid `zeroMem` when reading from era store see status-im/nim-snappy#22 for benchmarks
|
Can you clarify the claim why you find these checks to be redundant? Is it because the code computes |
Sort of - it happens as soon as a slot transition or block application happens - that said, this PR does increase risk slightly: in case of the wrong kind of data corruption, decompression could plausibly create a bogus state that gets read from in some code path where no HTR is called - however, the crc32 check was never really an intended part of the database design, ie a data corruption can happen when loading a public key or any other uncompressed data from disk (which does not pass through snappy) - if we want consistent crc32 checking, this needs to be an explicit feature on top of all database data. Notably, our crc32 implementation is also a ridiculously slow one - >10x faster implementations exist, like https://github.com/google/crc32c - if we go down that path, we should switch to a proper implementation. |
When loading blocks and states from db/era, we currently redundantly check their CRC32 - for a state, this costs 50ms of loading time presently (110mb uncompressed size) on a decent laptop.
maxDecompressedDbRecordSize- not actually used on recent data since we store the framed format - also, we're in luck: we blew past the limit quite some time agozeroMemwhen reading from era storesee status-im/nim-snappy#22 for benchmarks