Sporadic error: "panic: Value missing for hash"

[derekadams]

Summary of Bug

When a node is started and the backlog of transactions is processed, there is occasionally an error similar to the following that crashes the node and prevents startup:

panic: Value missing for hash 05d82a027b4ee9a59ad06fef9c39032297efc406678255ef31a20723fafbb684 corresponding to nodeKey 6e05d82a027b4ee9a59ad06fef9c39032297efc406678255ef31a20723fafbb684

Running the same sequence of actions to bootstrap the node works the majority of the time, but fails sporadically with this error message. Once the node enters this state, it can not be restarted without deleting the data directory and reloading everything.

This appears to be the same issue as reported by other Cosmos chains:
idena-network/idena-go#893

Version

Reproduced in v1.7.x and v1.7.6

Steps to Reproduce

Clear the data folder from a local node and reload from a quicksync file (successfully reproduced on recent testnet quicksync). The process will generally work fine, but occasionally will fail with the above message. Once it fails, the only way to get out of the state is to delete the data folder and reload from scratch.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[iramiller]

This is being addressed with the IAVL concurrency fixes in 1.8.1