Forest stuck on downloading headers

[LesnyRumcajs]

Describe the bug

Forest sometimes gets stuck on downloading block headers.

To reproduce

It happens from time to time, likely due to network conditions and peer selection (or not?). After a restart it usually recovers. No clear way to reproduce without further investigation.

Log output

2024-06-20T14:49:09.455644Z  INFO forest_filecoin::daemon: Starting Forest daemon, version 0.19.0+git.a4fe792
2024-06-20T14:49:09.455784Z  WARN forest_filecoin::daemon: Forest has encryption disabled
2024-06-20T14:49:09.455877Z  INFO forest_filecoin::daemon: Admin token: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXSwiZXhwIjoxNzI0MDc4OTQ5fQ.2mMWKCjauicZBmxYUExdRkzBjm43IQQlHI_L_aFQhcA
2024-06-20T14:49:09.456279Z  INFO forest_filecoin::db::migration::db_migration: No database migration required
2024-06-20T14:49:09.622853Z  INFO forest_filecoin::daemon::bundle: Loading actor bundle from /var/tmp/forest_actor_bundle.car.zst set by FOREST_ACTOR_BUNDLE_PATH environment variable
2024-06-20T14:49:10.118055Z  INFO forest_filecoin::genesis: Initialized genesis: bafy2bzacecyaggy24wol5ruvs6qm73gjibs2l2iyhcqmvi7r7a4ph7zx3yqd4
2024-06-20T14:49:10.118127Z  INFO forest_filecoin::daemon: Prometheus server started at 0.0.0.0:6116
2024-06-20T14:49:10.118648Z  INFO forest_filecoin::daemon: Using network :: calibnet
2024-06-20T14:49:10.118987Z  INFO forest_filecoin::libp2p::behaviour: libp2p Forest version: 0.19.0+git.a4fe792
2024-06-20T14:49:10.119651Z  INFO libp2p_swarm: local_peer_id=12D3KooWAAc8fL1KRjwTEdWV2KJ9KarV51f4f3u7nQ1smJvy8XDT
2024-06-20T14:49:10.119691Z  INFO forest_filecoin::libp2p::service: p2p network peer id: 12D3KooWAAc8fL1KRjwTEdWV2KJ9KarV51f4f3u7nQ1smJvy8XDT
2024-06-20T14:49:10.120405Z  INFO forest_filecoin::libp2p::service: p2p peer is now listening on: /ip4/127.0.0.1/tcp/42433
2024-06-20T14:49:10.120495Z  INFO forest_filecoin::daemon: JSON-RPC endpoint will listen at 127.0.0.1:2345
2024-06-20T14:49:10.122254Z  INFO forest_filecoin::chain_sync::chain_muxer: Evaluating network head...
2024-06-20T14:49:10.122315Z  INFO forest_filecoin::chain_sync::chain_muxer: local head is behind the network, local_epoch: 1718845, now_epoch: 1718952
2024-06-20T14:49:10.132399Z  INFO forest_filecoin::rpc: Ready for RPC connections
2024-06-20T14:49:10.494453Z  INFO forest_filecoin::libp2p::service: Running libp2p service
2024-06-20T14:49:10.540783Z  INFO forest_filecoin::chain_sync::chain_muxer: Local node is behind the network, starting BOOTSTRAP from LOCAL_HEAD = 1718845 -> NETWORK_HEAD = 1718952
2024-06-20T14:49:15.712508Z  INFO forest::progress: Downloading headers 105, 20 items/s, elapsed time: 5s
2024-06-20T14:49:20.121647Z  INFO forest_filecoin::libp2p::discovery: UPnP GatewayNotFound
2024-06-20T14:49:20.803622Z  INFO forest::progress: Downloading headers 105, 0 items/s, elapsed time: 10s
2024-06-20T14:49:26.029280Z  INFO forest::progress: Downloading headers 105, 0 items/s, elapsed time: 15s
2024-06-20T14:49:31.501326Z  INFO forest::progress: Downloading headers 105, 0 items/s, elapsed time: 20s
2024-06-20T14:49:36.746777Z  INFO forest::progress: Downloading headers 105, 0 items/s, elapsed time: 26s
2024-06-20T14:49:41.757951Z  INFO forest::progress: Downloading headers 105, 0 items/s, elapsed time: 31s
2024-06-20T14:49:47.089244Z  INFO forest::progress: Downloading headers 105, 0 items/s, elapsed time: 36s
2024-06-20T14:49:52.288682Z  INFO forest::progress: Downloading headers 105, 0 items/s, elapsed time: 41s
...
...
...
2024-06-20T16:50:19.531622Z  INFO forest::progress: Downloading headers 105, 0 items/s, elapsed time: 2h 1m 8s
2024-06-20T16:50:25.063933Z  INFO forest::progress: Downloading headers 105, 0 items/s, elapsed time: 2h 1m 14s
2024-06-20T16:50:30.213260Z  INFO forest::progress: Downloading headers 105, 0 items/s, elapsed time: 2h 1m 19s
2024-06-20T16:50:35.462762Z  INFO forest::progress: Downloading headers 105, 0 items/s, elapsed time: 2h 1m 24s

Expected behaviour

Forest doesn't get stuck. It either fails or (more preferred) recovers on its own without manual intervention or external scripting.

Screenshots

Environment (please complete the following information):

• Branch/commit 0.19.0+git.a4fe792
• Hardware Forest sync check runner

Other information and links

It was not happening ~1 month ago (at least I didn't notice it), so it may be a regression. That, or the network has new, exciting conditions Forest needs to adapt to.