goerli after cancun upgrade: prysm 4.2.1 and geth 1.13.11 don't work together anymore

[c0deright]

Describe the bug

We missed the Upgrade on January 17th and didn't upgrade our clients in time.

So we wanted to start from scratch, upgraded prysm to 4.2.1 and geth to 1.13.11 and removed each others databases.

Both prysm and geth only do log errors and not sync anything at all.

Has this worked before in a previous version?

worked until january 17th 2024 with prysm 4.1.1 and geth 1.13.5

🔬 Minimal Reproduction

prysm systemd service file

[Unit]
Description=Ethereum Beacon Chain (prysm) -- goerli1
After=network.target

[Service]
User=eth
Group=eth

Type=simple
ExecStart=/home/eth/prysm-client --datadir=/data/goerli1/prysm --grpc-gateway-port 3501 --rpc-port 4001 --p2p-tcp-port 13001 --p2p-udp-port 12001 --monitoring-port 8081 --execution-endpoint=http://localhost:8552 --jwt-secret=/data/goerli1/jwtsecret --block-batch-limit 1024 --goerli --genesis-state=/data/goerli1/prysm/genesis.ssz --checkpoint-sync-url https://goerli-sync.invis.tools/
StandardOutput=append:/data/goerli1/prysm/log.txt

Restart=always
PrivateTmp=true
TimeoutStopSec=3600s
TimeoutStartSec=10s
StartLimitInterval=120s
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

geth systemd service file

[Unit]
Description=Ethereum Go daemon (geth) -- goerli1
After=network.target

[Service]
User=eth
Group=eth

Type=simple
ExecStart=/usr/bin/geth --datadir /data/goerli1/geth --port 30304 --discovery.port 30304 --http --http.addr 0.0.0.0 --http.port 8546 --rpc.enabledeprecatedpersonal --http.api "engine,debug,personal,eth,net,web3" --http.vhosts='*' --allow-insecure-unlock --maxpeers 30 --verbosity 3 --authrpc.jwtsecret /data/goerli1/jwtsecret --authrpc.port 8552 --cache 2048 --goerli
StandardOutput=append:/data/goerli1/geth/log.txt
KillSignal=SIGINT

Restart=always
PrivateTmp=true
TimeoutStopSec=3600s
TimeoutStartSec=10s
StartLimitInterval=120s
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

Error

prysm errors

time="2024-01-31 14:33:59" level=info msg="Finished reading JWT secret from /data/goerli1/jwtsecret"
time="2024-01-31 14:33:59" level=info msg="using checkpoint sync url https://goerli-sync.invis.tools/ for value in --genesis-beacon-api-url flag"
time="2024-01-31 14:33:59" level=warning msg="Running on the Prater Testnet" prefix=flags
time="2024-01-31 14:33:59" level=warning msg="In order to receive transaction fees from proposing blocks, you must provide flag --suggested-fee-recipient with a valid ethereum address when starting your beacon node. Please see our documentation for more information on this requirement (https://docs.prylabs.network/docs/execution-node/fee-recipient)." prefix=node
time="2024-01-31 14:33:59" level=info msg="Checking DB" database-path="/data/goerli1/prysm/beaconchaindata" prefix=node
time="2024-01-31 14:33:59" level=info msg="Opening Bolt DB at /data/goerli1/prysm/beaconchaindata/beaconchain.db" prefix=db
time="2024-01-31 14:34:02" level=info msg="detected supported config in remote finalized state, name=prater, fork=deneb"
time="2024-01-31 14:34:03" level=info msg="Downloaded checkpoint sync state and block." block_root=0x114f5d95990c0b45134475c4fcfbaef8a2af218b78a27f439feb1d7fabde4188 block_slot=7509280 state_root=0xc64447eb7f054a40256f75e8affeabdcfe2068cfd6a6ecd097b0905d40fc50e5 state_slot=7509280
time="2024-01-31 14:34:03" level=info msg="detected supported config for state & block version, config name=prater, fork name=deneb" prefix=db
time="2024-01-31 14:34:03" level=info msg="saving checkpoint block to db, w/ root=0x114f5d95990c0b45134475c4fcfbaef8a2af218b78a27f439feb1d7fabde4188" prefix=db
time="2024-01-31 14:34:03" level=info msg="calling SaveState w/ blockRoot=114f5d95990c0b45134475c4fcfbaef8a2af218b78a27f439feb1d7fabde4188" prefix=db
time="2024-01-31 14:34:04" level=info msg="Deposit contract: 0xff50ed3d0ec03ac01d4c79aad74928bff48a7b2b" prefix=node
time="2024-01-31 14:34:04" level=info msg="Running node with peer id of 16Uiu2HAkyWb9XPx4GvCjzFEE6tHC7fEbe5FGUuKiZ26KCSjoMFZ1 " prefix=p2p
time="2024-01-31 14:34:04" level=info msg="gRPC server listening on port" address="127.0.0.1:4001" prefix=rpc
time="2024-01-31 14:34:04" level=warning msg="You are using an insecure gRPC server. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc" prefix=rpc
time="2024-01-31 14:34:04" level=info msg="Starting beacon node" prefix=node version="Prysm/v4.2.1/59b310a2216c57fcf67ea0fdec739dad07aeec8b. Built at: 2024-01-30 16:26:21+00:00"
time="2024-01-31 14:34:04" level=info msg="Waiting for state to be initialized" prefix=initial-sync
time="2024-01-31 14:34:04" level=info msg="Blockchain data already exists in DB, initializing..." prefix=blockchain
time="2024-01-31 14:34:04" level=info msg="Exiting backfill service; not enabled."
time="2024-01-31 14:34:04" level=info msg="Starting gRPC gateway" address="127.0.0.1:3501" prefix=gateway
time="2024-01-31 14:34:05" level=info msg="Received state initialized event" prefix=initial-sync
time="2024-01-31 14:34:05" level=info msg="Starting initial chain sync..." prefix=initial-sync
time="2024-01-31 14:34:05" level=info msg="Waiting for enough suitable peers before syncing" prefix=initial-sync required=3 suitable=0
time="2024-01-31 14:34:05" level=info msg="Started discovery v5" ENR="enr:-MK4QOjKTyTdTBt_Ttp82hvJNiBi4eL3JQ40XaJUjky_um5AAWNm4mV1w7Tzopp_Oo52YjWmxULklJMOyihAlWjeoVeGAY1fugzKh2F0dG5ldHOIAAAAAAAAAACEZXRoMpCnXczyBAAQIP__________gmlkgnY0gmlwhAoAMYyJc2VjcDI1NmsxoQI8sTAdIzw8ghk8O3PhlpimeOFprLFlb5VEAVYaWB_nOIhzeW5jbmV0cwCDdGNwgjLJg3VkcIIu4Q" prefix=p2p
time="2024-01-31 14:34:05" level=info msg="Node started p2p server" multiAddr="/ip4/10.0.49.140/tcp/13001/p2p/16Uiu2HAkyWb9XPx4GvCjzFEE6tHC7fEbe5FGUuKiZ26KCSjoMFZ1" prefix=p2p
time="2024-01-31 14:34:10" level=info msg="Waiting for enough suitable peers before syncing" prefix=initial-sync required=3 suitable=0
time="2024-01-31 14:34:15" level=info msg="Waiting for enough suitable peers before syncing" prefix=initial-sync required=3 suitable=0
time="2024-01-31 14:34:19" level=info msg="Connected to new endpoint: http://localhost:8552" prefix=execution
time="2024-01-31 14:35:05" level=info msg="Peer summary" activePeers=4 inbound=0 outbound=4 prefix=p2p

[..]

time="2024-01-31 16:11:12" level=info msg="Peer summary" activePeers=11 inbound=0 outbound=11 prefix=p2p
time="2024-01-31 16:12:12" level=info msg="Peer summary" activePeers=11 inbound=0 outbound=11 prefix=p2p
time="2024-01-31 16:12:43" level=error msg="Unable to process past deposit contract logs, perhaps your execution client is not fully synced" error="processPastLogs: no contract code at given address" prefix=execution
time="2024-01-31 16:13:12" level=info msg="Peer summary" activePeers=12 inbound=0 outbound=12 prefix=p2p
time="2024-01-31 16:14:12" level=info msg="Peer summary" activePeers=9 inbound=0 outbound=9 prefix=p2p
time="2024-01-31 16:14:58" level=error msg="Unable to process past deposit contract logs, perhaps your execution client is not fully synced" error="processPastLogs: no contract code at given address" prefix=execution
time="2024-01-31 16:15:12" level=info msg="Peer summary" activePeers=8 inbound=0 outbound=8 prefix=p2p
time="2024-01-31 16:16:12" level=info msg="Peer summary" activePeers=10 inbound=0 outbound=10 prefix=p2p
time="2024-01-31 16:17:12" level=info msg="Peer summary" activePeers=14 inbound=0 outbound=14 prefix=p2p
time="2024-01-31 16:17:13" level=error msg="Unable to process past deposit contract logs, perhaps your execution client is not fully synced" error="processPastLogs: no contract code at given address" prefix=execution
time="2024-01-31 16:18:12" level=info msg="Peer summary" activePeers=10 inbound=0 outbound=10 prefix=p2p
time="2024-01-31 16:19:12" level=info msg="Peer summary" activePeers=12 inbound=0 outbound=12 prefix=p2p
time="2024-01-31 16:19:28" level=error msg="Unable to process past deposit contract logs, perhaps your execution client is not fully synced" error="processPastLogs: no contract code at given address" prefix=execution
time="2024-01-31 16:20:12" level=info msg="Peer summary" activePeers=12 inbound=0 outbound=11 prefix=p2p
time="2024-01-31 16:21:12" level=info msg="Peer summary" activePeers=11 inbound=0 outbound=11 prefix=p2p
time="2024-01-31 16:21:43" level=error msg="Unable to process past deposit contract logs, perhaps your execution client is not fully synced" error="processPastLogs: no contract code at given address" prefix=execution
time="2024-01-31 16:22:12" level=info msg="Peer summary" activePeers=12 inbound=0 outbound=12 prefix=p2p
time="2024-01-31 16:23:12" level=info msg="Peer summary" activePeers=14 inbound=0 outbound=14 prefix=p2p
time="2024-01-31 16:23:58" level=error msg="Unable to process past deposit contract logs, perhaps your execution client is not fully synced" error="processPastLogs: no contract code at given address" prefix=execution

geth errors

INFO [01-31|14:06:54.575] "Starting Geth on Görli testnet..."
INFO [01-31|14:06:54.580] Maximum peer count                       ETH=30 total=30
INFO [01-31|14:06:54.581] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [01-31|14:06:54.589] Set global gas cap                       cap=50,000,000
INFO [01-31|14:06:54.589] Initializing the KZG library             backend=gokzg
INFO [01-31|14:06:54.622] Allocated trie memory caches             clean=307.00MiB dirty=512.00MiB
INFO [01-31|14:06:54.623] Defaulting to pebble as the backing database
INFO [01-31|14:06:54.623] Allocated cache and file handles         database=/data/goerli1/geth/geth/chaindata cache=1024.00MiB handles=262,144
INFO [01-31|14:06:54.650] Opened ancient database                  database=/data/goerli1/geth/geth/chaindata/ancient/chain readonly=false
INFO [01-31|14:06:54.650] State schema set to default              scheme=hash
INFO [01-31|14:06:54.650] Initialising Ethereum protocol           network=5 dbversion=<nil>
INFO [01-31|14:06:54.652] Writing custom genesis block
INFO [01-31|14:06:54.663] Persisted trie from memory database      nodes=361 size=51.44KiB time=2.698237ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [01-31|14:06:54.678]
INFO [01-31|14:06:54.678] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [01-31|14:06:54.678] Chain ID:  5 (goerli)
INFO [01-31|14:06:54.678] Consensus: Beacon (proof-of-stake), merged from Clique (proof-of-authority)
INFO [01-31|14:06:54.678]
INFO [01-31|14:06:54.678] Pre-Merge hard forks (block based):
INFO [01-31|14:06:54.678]  - Homestead:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [01-31|14:06:54.678]  - Tangerine Whistle (EIP 150): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [01-31|14:06:54.678]  - Spurious Dragon/1 (EIP 155): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [01-31|14:06:54.678]  - Spurious Dragon/2 (EIP 158): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [01-31|14:06:54.678]  - Byzantium:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [01-31|14:06:54.678]  - Constantinople:              #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [01-31|14:06:54.678]  - Petersburg:                  #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [01-31|14:06:54.678]  - Istanbul:                    #1561651  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [01-31|14:06:54.678]  - Berlin:                      #4460644  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [01-31|14:06:54.678]  - London:                      #5062605  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [01-31|14:06:54.678]
INFO [01-31|14:06:54.678] Merge configured:
INFO [01-31|14:06:54.678]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [01-31|14:06:54.678]  - Network known to be merged: true
INFO [01-31|14:06:54.678]  - Total terminal difficulty:  10790000
INFO [01-31|14:06:54.678]
INFO [01-31|14:06:54.678] Post-Merge hard forks (timestamp based):
INFO [01-31|14:06:54.678]  - Shanghai:                    @1678832736 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [01-31|14:06:54.679]  - Cancun:                      @1705473120
INFO [01-31|14:06:54.679]
INFO [01-31|14:06:54.679] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [01-31|14:06:54.679]
INFO [01-31|14:06:54.679] Loaded most recent local block           number=0 hash=bf7e33..b88c1a td=1 age=5y3w5d
WARN [01-31|14:06:54.679] Failed to load snapshot                  err="missing or corrupted snapshot"
INFO [01-31|14:06:54.682] Rebuilding state snapshot
INFO [01-31|14:06:54.683] Initialized transaction indexer          range="last 2350000 blocks"
INFO [01-31|14:06:54.683] Resuming state snapshot generation       root=5d6cde..8b3008 accounts=0 slots=0 storage=0.00B dangling=0 elapsed=1.752ms
INFO [01-31|14:06:54.683] Regenerated local transaction journal    transactions=0 accounts=0
INFO [01-31|14:06:54.687] Generated state snapshot                 accounts=260 slots=0 storage=9.69KiB dangling=0 elapsed=5.797ms
INFO [01-31|14:06:54.699] Enabled snap sync                        head=0 hash=bf7e33..b88c1a
INFO [01-31|14:06:54.699] Chain post-merge, sync via beacon client
INFO [01-31|14:06:54.699] Gasprice oracle is ignoring threshold set threshold=2
INFO [01-31|14:06:54.701] Stored checkpoint snapshot to disk       number=0 hash=bf7e33..b88c1a
WARN [01-31|14:06:54.703] Engine API enabled                       protocol=eth
INFO [01-31|14:06:54.703] Starting peer-to-peer node               instance=Geth/v1.13.11-stable-8f7eb9cc/linux-amd64/go1.21.6
INFO [01-31|14:06:54.713] New local node record                    seq=1,706,706,414,712 id=d1d4bd4a31ed63b2 ip=127.0.0.1 udp=30304 tcp=30304
WARN [01-31|14:06:54.713] Deprecated personal namespace activated
INFO [01-31|14:06:54.713] Started P2P networking                   self=enode://8f6ba80110d14e68f17c126bf26d5f8635b0729c74c73a763e58bf3aed89282aeeb1572d58c5ee01c5b8dd87ee727a8cd8beebf84bed47cc287d3fb2ad7f1c33@127.0.0.1:30304
INFO [01-31|14:06:54.714] IPC endpoint opened                      url=/data/goerli1/geth/geth.ipc
INFO [01-31|14:06:54.715] Loaded JWT secret file                   path=/data/goerli1/jwtsecret crc32=0xf7043bac
INFO [01-31|14:06:54.715] HTTP server started                      endpoint=[::]:8546 auth=false prefix= cors= vhosts=*
INFO [01-31|14:06:54.715] WebSocket enabled                        url=ws://127.0.0.1:8552
INFO [01-31|14:06:54.715] HTTP server started                      endpoint=127.0.0.1:8552 auth=true  prefix= cors=localhost vhosts=localhost
INFO [01-31|14:07:05.238] Looking for peers                        peercount=0 tried=18 static=0
INFO [01-31|14:07:15.398] Looking for peers                        peercount=0 tried=33 static=0
INFO [01-31|14:07:25.426] Looking for peers                        peercount=0 tried=20 static=0
WARN [01-31|14:07:29.704] Post-merge network, but no beacon client seen. Please launch one to follow the chain!

[..]

INFO [01-31|14:10:17.403] Looking for peers                        peercount=2 tried=37 static=0
INFO [01-31|14:10:27.607] Looking for peers                        peercount=2 tried=42 static=0
INFO [01-31|14:10:37.670] Looking for peers                        peercount=2 tried=43 static=0
INFO [01-31|14:10:47.670] Looking for peers                        peercount=2 tried=32 static=0
INFO [01-31|14:10:57.679] Looking for peers                        peercount=2 tried=20 static=0
INFO [01-31|14:11:07.854] Looking for peers                        peercount=2 tried=56 static=0
INFO [01-31|14:11:17.863] Looking for peers                        peercount=2 tried=37 static=0
INFO [01-31|14:11:28.003] Looking for peers                        peercount=2 tried=41 static=0
INFO [01-31|14:11:38.189] Looking for peers                        peercount=2 tried=41 static=0
INFO [01-31|14:11:48.213] Looking for peers                        peercount=2 tried=42 static=0
WARN [01-31|14:12:29.737] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
WARN [01-31|14:17:29.766] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
WARN [01-31|14:22:29.801] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
WARN [01-31|14:27:29.833] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
WARN [01-31|14:32:29.865] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
WARN [01-31|14:37:29.897] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
WARN [01-31|14:42:29.930] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
WARN [01-31|14:47:29.964] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
WARN [01-31|14:52:29.996] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
WARN [01-31|14:57:30.029] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
WARN [01-31|15:02:30.058] Post-merge network, but no beacon client seen. Please launch one to follow the chain!

Platform(s)

Linux (x86)

What version of Prysm are you running? (Which release)

Prysm/v4.2.1/59b310a2216c57fcf67ea0fdec739dad07aeec8b.

Anything else relevant (validator index / public key)?

No validator.

[ArturDolzan]

Same issue on Holesky

[kasey]

I see that you have your --block-batch-limit=1024. If you enable debug logs I think you'll see that peers are rejecting your RPC requests because they are too large. Could you try removing this flag to sync using the default value?

cc @c0deright @ArturDolzan

[nisdas]

Hey this should be resolved with this: https://github.com/prysmaticlabs/prysm/releases/tag/v4.2.2-rc.0

If this goes well, it will be our next release.

[ArturDolzan]

I was setting 512 for --block-batch-limit. I was able to sync the beacon again after decreasing it to 128 only. Same case on Holesky and Sepolia for me

[c0deright]

Indeed, removing the --block-batch-limit option entirely made prysm sync on goerli. That's very strange as it did work in the past and at least on mainnet it was way faster (speaking in blocks per second) than the default value of --block-batch-limit.

[ArturDolzan]

Will it be an issue for mainnet as well? My mainnet beacons are set block-batch-limit as 512

[nisdas]

I will prioritize getting this in:
#13502

along with a follow-up PR to cap the batch limit. After Deneb the network batch limit is fixed at 128, which is why it has been triggered for your setups.

[nisdas]

This is resolved with v5