Beacon Node Restarts with New State Mgmt Causes Chain Stall

[rauljordan]

Hi all, currently our prod nodes in our cluster running in the topaz testnet all stalled after the pods restarted due to a bug in the new state management feature:

{"error":"could not get pre state for slot 5696: could not load blocks for hot state using root: end block roots don't match","message":"Failed to validate attestation","prefix":"blockchain","severity":"ERROR"}
{"error":"could not get pre state for slot 5696: could not load blocks for hot state using root: end block roots don't match","message":"Failed to validate attestation","prefix":"blockchain","severity":"ERROR"}
{"error":"could not get pre state for slot 5696: could not load blocks for hot state using root: end block roots don't match","message":"Failed to validate attestation","prefix":"blockchain","severity":"ERROR"}
{"error":"could not get pre state for slot 5696: could not load blocks for hot state using root: end block roots don't match","message":"Failed to validate attestation","prefix":"blockchain","severity":"ERROR"}
{"error":"could not get pre state for slot 5696: could not load blocks for hot state using root: end block roots don't match","message":"Failed to validate attestation","prefix":"blockchain","severity":"ERROR"}

The bug is pending resolution by adding the --disable-new-state-mgmt to our pods as a temporary fix

[terencechain]

Unable to reproduce it via the snapshot DB from cluster. Looking into ways to reproduce this issue. If anyone has any idea, please let me know