Full sync gets stuck at sender recovery

[Bobface]

Describe the bug

When syncing a new full node, reth gets stuck at the SenderRecovery stage with eta=unknown. Restarting the node fixes the issue, and the sync continues.

Steps to reproduce

Unknown

Node logs

reth  | 2024-04-15T06:23:16.703143Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:23:41.702380Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:24:06.703271Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:24:31.703192Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:24:56.702814Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:25:21.702480Z  INFO Status connected_peers=131 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:25:46.703413Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:26:11.703089Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:26:36.702623Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:27:01.703219Z  INFO Status connected_peers=129 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:27:26.702958Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:27:51.702323Z  INFO Status connected_peers=129 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:28:16.702480Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:28:41.703083Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:29:06.702519Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:29:31.702753Z  INFO Status connected_peers=132 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:29:56.702744Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown                                                                                                                                                 reth  | 2024-04-15T06:30:21.702373Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:30:46.702788Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:31:11.702614Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:31:36.702810Z  INFO Status connected_peers=131 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:32:01.702512Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:32:26.702711Z  INFO Status connected_peers=131 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:32:51.702458Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:33:16.702721Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:33:41.703337Z  INFO Status connected_peers=131 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown                                                                                                                                                 
reth  | 2024-04-15T06:34:06.703346Z  INFO Status connected_peers=131 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:34:31.703217Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:34:56.702537Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:35:21.702752Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:35:46.703200Z  INFO Status connected_peers=131 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:36:11.703062Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:36:36.703065Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:37:01.702981Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:37:26.702389Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown
reth  | 2024-04-15T06:37:51.702461Z  INFO Status connected_peers=131 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown                                                                                                                                                 reth  | 2024-04-15T06:38:16.702795Z  INFO Status connected_peers=130 freelist=8 stage=SenderRecovery checkpoint=12993785 target=19640279 stage_progress=53.18% stage_eta=unknown

Platform(s)

Linux (x86)

What version/commit are you on?

(using the official docker image)
reth Version: 0.2.0-beta.5
Commit SHA: 54f75cd
Build Timestamp: 2024-04-03T16:25:25.825621855Z
Build Features: jemalloc
Build Profile: maxperf

What database version are you on?

Current database version: 2
Local database version: 2

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

None

If you've built Reth from source, provide the full command you used

Using the official docker image

Code of Conduct

• I agree to follow the Code of Conduct

[Bobface]

Update: This seems to happen also at later stages. Again, a restart fixes the issue temporarily.

reth  | 2024-04-15T12:54:46.584592Z  INFO Status connected_peers=131 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T12:55:11.584436Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T12:55:36.584542Z  INFO Status connected_peers=128 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T12:56:01.584587Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T12:56:26.584109Z  INFO Status connected_peers=129 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T12:56:51.583943Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T12:57:16.584323Z  INFO Status connected_peers=127 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T12:57:41.584288Z  INFO Status connected_peers=131 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T12:58:06.583877Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T12:58:31.584025Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T12:58:56.583910Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T12:59:21.583912Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T12:59:46.584502Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:00:11.583870Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:00:36.584548Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:01:01.584107Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:01:26.583813Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:01:51.584715Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:02:16.584092Z  INFO Status connected_peers=126 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:02:41.584501Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:03:06.583887Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:03:31.584228Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:03:56.584346Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:04:21.584003Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:04:46.584256Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:05:11.584557Z  INFO Status connected_peers=127 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:05:36.584542Z  INFO Status connected_peers=129 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:06:01.584841Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:06:26.583828Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:06:51.584108Z  INFO Status connected_peers=129 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:07:16.584650Z  INFO Status connected_peers=131 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:07:41.584262Z  INFO Status connected_peers=128 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:08:06.584276Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:08:31.584175Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:08:56.584151Z  INFO Status connected_peers=129 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:09:21.584543Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:09:46.584808Z  INFO Status connected_peers=131 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:10:11.584030Z  INFO Status connected_peers=127 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:10:36.584651Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:11:01.584812Z  INFO Status connected_peers=126 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:11:26.584696Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:11:51.584274Z  INFO Status connected_peers=131 freelist=8 stage=Execution checkpoint=0 target=19640279
reth  | 2024-04-15T13:12:16.584697Z  INFO Status connected_peers=130 freelist=8 stage=Execution checkpoint=0 target=19640279

[sungpilpaek]

Same here, I have to keep restarting at every interval to make a progress. I also dropped the entire db and did the full sync.

[sungpilpaek]

Sounds obvious, however downgrading to v0.2.0-beta.4 works successfully.

[github-actions]

This issue is stale because it has been open for 21 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale.