Problems to retrieve epoch data from memory

[EclesioMeloJunior]

Describe the bug

While running a Gossamer node with 2 other substrate nodes after some block height the finalization stops but we keep producing blocks, but after some amount of epochs (in this case 3) I noticed several ERROR logs equal to the one below:

2022-06-28T16:48:26-04:00 EROR block data processing for block with hash 0x35314fd6441257ceff7b1b8519a1ea11ecc6c2590a48c1998d8a9e72f40ea67a failed: failed to get verifier info for block 118: failed to get epoch data for epoch 3: failed to get epoch data from memory: cannot get parent header: Key not found	chain_processor.go:L86	pkg=sync

Possible investigation path

The block hash in the error message was created by a substrate node B and imported by substrate node C but the gossamer node is not aware of the existence of the node

Substrate node B log:

🔖 Pre-sealed block for proposal at 118. Hash now 0x35314fd6441257ceff7b1b8519a1ea11ecc6c2590a48c1998d8a9e72f40ea67a, previously 0x2a109713b91616436bcec2426d3f7af2d8f8ff4572aa0f7798a68d5bb1f115bf.
2022-06-28 16:47:08 ✨ Imported #118 (0x3531…a67a)
2022-06-28 16:47:08 ✨ Imported #118 (0x6273…53de)
2022-06-28 16:47:10 💤 Idle (2 peers), best: #118 (0x3531…a67a), finalized #19 (0x5064…be79), ⬇ 0.5kiB/s ⬆ 0.6kiB/s

Substrate node C log:

2022-06-28 16:47:08 🔖 Pre-sealed block for proposal at 118. Hash now 0x62732b34e5caf14412eb7322471e34270916be3265d79888c93f6954338953de, previously 0xa3d5223191b51ab7753ae090845293fb66c53b42d312f023be67b80c311ed3f2.
2022-06-28 16:47:08 ✨ Imported #118 (0x3531…a67a)
2022-06-28 16:47:08 ✨ Imported #118 (0x6273…53de)
2022-06-28 16:47:11 💤 Idle (2 peers), best: #118 (0x3531…a67a), finalized #19 (0x5064…be79), ⬇ 0.6kiB/s ⬆ 8.2kiB/s

The error in the gossamer node tells us that: we tried:

• Get the epoch data to block #118 (0x3531…a67a) in memory, but the block #118 (0x3531…a67a) does not exist in our blocktree
• As the block is not in the blocktree let's try with the parent hash, then the gossamer node fails returning cannot get parent header: Key not found, telling that it does not know the hash

[kishansagathiya]

epoch data from memory shows up much less now (only in case of empty epochs).

In case, there are empty epochs in the network, it is expected behaviour to node produce blocks anymore, which happens in case of gossamer with epoch not found in memory error.

Take a look at #2680 (comment).

@timwu20 @danforbes closing this as per our discussion.