Forest chain muxer error

[LesnyRumcajs]

Issue summary

I got hit by this several times already. In short, sometimes, in case of an error (e.g., state mismatch) and after throwing a ctrl-c (I am not sure it's 100% needed, though) Forest will start spamming the terminal with:

2023-03-15T12:00:49.755785Z ERROR forest_chain_sync::chain_muxer: Evaluating the network head failed, retrying. Error = P2PEventStreamReceive("receiving on a closed channel")
2023-03-15T12:00:49.755788Z  INFO forest_chain_sync::chain_muxer: Evaluating network head...
2023-03-15T12:00:49.755789Z ERROR forest_chain_sync::chain_muxer: Evaluating the network head failed, retrying. Error = P2PEventStreamReceive("receiving on a closed channel")
2023-03-15T12:00:49.755792Z  INFO forest_chain_sync::chain_muxer: Evaluating network head...
2023-03-15T12:00:49.755793Z ERROR forest_chain_sync::chain_muxer: Evaluating the network head failed, retrying. Error = P2PEventStreamReceive("receiving on a closed channel")
2023-03-15T12:00:49.755796Z  INFO forest_chain_sync::chain_muxer: Evaluating network head...
2023-03-15T12:00:49.755798Z ERROR forest_chain_sync::chain_muxer: Evaluating the network head failed, retrying. Error = P2PEventStreamReceive("receiving on a closed channel")
2023-03-15T12:00:49.755802Z  INFO forest_chain_sync::chain_muxer: Evaluating network head...
2023-03-15T12:00:49.755804Z ERROR forest_chain_sync::chain_muxer: Evaluating the network head failed, retrying. Error = P2PEventStreamReceive("receiving on a closed channel")
2023-03-15T12:00:49.755806Z  INFO forest_chain_sync::chain_muxer: Evaluating network head...
2023-03-15T12:00:49.755808Z ERROR forest_chain_sync::chain_muxer: Evaluating the network head failed, retrying. Error = P2PEventStreamReceive("receiving on a closed channel")
2023-03-15T12:00:49.755810Z  INFO forest_chain_sync::chain_muxer: Evaluating network head...
2023-03-15T12:00:49.755812Z ERROR forest_chain_sync::chain_muxer: Evaluating the network head failed, retrying. Error = P2PEventStreamReceive("receiving on a closed channel")
2023-03-15T12:00:49.755815Z  INFO forest_chain_sync::chain_muxer: Evaluating network head...
2023-03-15T12:00:49.755817Z ERROR forest_chain_sync::chain_muxer: Evaluating the network head failed, retrying. Error = P2PEventStreamReceive("receiving on a closed channel")
2023-03-15T12:00:49.755819Z  INFO forest_chain_sync::chain_muxer: Evaluating network head...
2023-03-15T12:00:49.755821Z ERROR forest_chain_sync::chain_muxer: Evaluating the network head failed, retrying. Error = P2PEventStreamReceive("receiving on a closed channel")

It is impossible to kill it with a regular ctrl-c; one has to open another tab and pkill -9 forest. To make it even worse, this uses a significant amount of CPU (probably coming from endless logs flooding the terminal without sleep). It's hard to reproduce, but it happened quite a few times when I sent SIGINT at the wrong moment.

We need to handle it better, but I have no clear solution. The simple idea is to shut down Forest after some amount of errors in a given interval.

Other information and links

[elmattic]

Could you give more details on how to reproduce this error (at which moment do you hit ctrl-c)?

Maybe the ChainMuxer future needs special case when shutting down the Tokio runtime.

[LesnyRumcajs]

Not really. As I mentioned, it's difficult to reproduce. To no surprise, it happens during sync. I often encountered it when analysing state mismatches while pressing ctrl-c just after the state mismatch error.

[elmattic]

@LesnyRumcajs But can it be reproduced on calibnet?

[LesnyRumcajs]

Yeah, it was happening on calibnet on my machine. The logs I posted are from mainnet from the sync-check.