Race condition between assignment of "latest" tag and persistence of the latest raw block data causing intermittent test failures

[benjamincburns]

If you wrap the "should set storage slot and delete after" evm API test in a for loop such that it executes 100 times, you'll observe that it fails intermittently, due to the RPC method handler for eth_getStorageAt throwing because it can't find a block.

This occurs even though the test isn't specifying the block number in its call to eth_getStorageAt, meaning it's implicitly using the "latest" tag. This implies that blockchain.blocks.getEffectiveBlockNumber is occasionally returning the number for a block that hasn't yet been persisted to the database.

If you refactor the API handler for eth_getStorageAt to avoid the use of blockchain.blocks.getRawByBlockNumber and instead prefer blockchain.blocks.get, the test in question succeeds 100% of the time. While this adequately works around the problem, I don't think that this is a proper fix for the issue, as it seems that the real problem is a lack of consistency between the in-memory and on-disk representations of the block's current head state.

Note that several other RPC method handlers appear to use the same pattern for reading the raw block data, and are therefore likely also affected by this issue.

[cds-amal]

whoa! nice @benjamincburns ! Do you have a test set up for this?

[benjamincburns]

I caught it because I was observing intermittent test failures while running tests locally for another PR.

I didn't really isolate the exact cause of the issue however, so I couldn't write a test that guards against the problem recurring. The best way to detect it that I know of for now is to run the test that I mentioned in the description many times over and over again in a loop.

[MicaiahReid]

Good find, @benjamincburns. I have a PR (#3016) out to fix a similar issue that was caused by block being returned before it was persisted to the database.

I believe that PR should fix this too; I will wrap the test you mentioned in a loop to confirm that failures aren't still happening.