Beacon client stalls during init-sync to sync transition

[torfbolt]

Sometimes the transition from init-sync to normal sync seems to silently fail. I saw this particularly when the client went into init-sync mode to resync after falling behind:

time="2020-01-18 18:32:50" level=info msg="Processing block 63154/63164 - estimated time remaining 20s" blocksPerSecond=0.5 peers="1/10" prefix=initial-sync
time="2020-01-18 18:32:50" level=info msg="Executing state transition on block" prefix=forkchoice root=0x2e721142... slot=63154
time="2020-01-18 18:32:52" level=info msg="Finished applying state transition" attestations=5 deposits=0 prefix=blockchain slot=63154
time="2020-01-18 18:32:52" level=info msg="Resync attempt complete" prefix=initial-sync slot=63154 synced=true
time="2020-01-18 18:33:13" level=info msg="Peer disconnected" active=9 peer=16Uiu2HAmEhpnUJhpdzpiduRRrRCkpQAGZsUq3qJmWpvbdzFUWdeJ prefix=p2p
time="2020-01-18 18:36:15" level=info msg="Peer connected" active=10 peer=16Uiu2HAkvSZyPFfSonVPiF9EEPKKv65v6QcPC3sBgPVRxURWX2u3 prefix=p2p
time="2020-01-18 18:40:22" level=info msg="Peer disconnected" active=9 peer=16Uiu2HAkwSeLfRR7m7zxpF3qxDwGK8ghHizJDHDd2e6j8arwsNG1 prefix=p2p
time="2020-01-18 18:48:47" level=info msg="Peer connected" active=10 peer=16Uiu2HAmRMqPe1nR1yAwUEpBKzUh8ABHzcMBSRnnK1z4fkuARu8e prefix=p2p
time="2020-01-18 18:48:47" level=info msg="Peer disconnected" active=9 peer=16Uiu2HAmRMqPe1nR1yAwUEpBKzUh8ABHzcMBSRnnK1z4fkuARu8e prefix=p2p
time="2020-01-18 18:49:18" level=info msg="Fallen behind peers; reverting to initial sync to catch up" currentEpoch=1976 peersEpoch=1974 prefix=sync syncedEpoch=1973
time="2020-01-18 18:51:10" level=info msg="Peer connected" active=10 peer=16Uiu2HAmEhpnUJhpdzpiduRRrRCkpQAGZsUq3qJmWpvbdzFUWdeJ prefix=p2p
time="2020-01-18 18:58:33" level=info msg="Peer connected" active=11 peer=16Uiu2HAmRMqPe1nR1yAwUEpBKzUh8ABHzcMBSRnnK1z4fkuARu8e prefix=p2p
time="2020-01-18 18:59:28" level=info msg="Peer disconnected" active=9 peer=16Uiu2HAmEhpnUJhpdzpiduRRrRCkpQAGZsUq3qJmWpvbdzFUWdeJ prefix=p2p

It looks like the client tries to resync again, but no block processing occurs any more, only some p2p log messages. After a restart the client usually syncs ok again.

[torfbolt]

Possibly related to #4551 ?

[mcdee]

#4551 became a bit too unfocused, #4570 should address this issue.

[torfbolt]

I just noticed that the beacon client uses 3.8 of my 4G RAM, which would explain random stalling. Is it supposed to use that much or is there a memory leak/issue?

[rauljordan]

Hey @torfbolt we just resolved this thanks @nisdas in #4811, could you try again?

[prestonvanloon]

FYI #4811 was reverted in #4825