failed to notify engine driver of L1 head change

[vataops]

I'm running and managing an archive node for Base. Recently, the block height synced up, but suddenly it's falling way behind. And it's not just one node, but three of them in the same state.

I keep seeing this log message:

mainnet-base-geth-1  | INFO [07-08|04:03:27.553] Looking for peers                        peercount=0 tried=187 static=0
mainnet-base-node-1  | t=2024-07-08T04:03:32+0000 lvl=warn msg="failed to notify engine driver of L1 head change" err="context deadline exceeded"

mainnet-base-node-1  | t=2024-07-08T04:08:15+0000 lvl=info msg="connected to peer" peer=16Uiu2HAmRLX5wnescX2mE62ZZwZULkeMiPCXF5xs7UDLKYSpT8hD addr=/ip4/174.138.187.2/tcp/56604
mainnet-base-node-1  | t=2024-07-08T04:08:15+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAmRLX5wnescX2mE62ZZwZULkeMiPCXF5xs7UDLKYSpT8hD
mainnet-base-node-1  | t=2024-07-08T04:08:15+0000 lvl=info msg="disconnected from peer" peer=16Uiu2HAmRLX5wnescX2mE62ZZwZULkeMiPCXF5xs7UDLKYSpT8hD addr=/ip4/174.138.187.2/tcp/56604
mainnet-base-node-1  | t=2024-07-08T04:08:17+0000 lvl=info msg="connected to peer" peer=16Uiu2HAm6xpJr9JHjdTMX3k6Qyz5Bf1cSGzYfCFETWj61TJKAJWj addr=/ip4/65.109.18.95/tcp/50560
mainnet-base-node-1  | t=2024-07-08T04:08:17+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAm6xpJr9JHjdTMX3k6Qyz5Bf1cSGzYfCFETWj61TJKAJWj

Here's the configuration:

RETH_CHAIN=base
RETH_SEQUENCER_HTTP=https://mainnet-sequencer.base.org

#OP_GETH_AUTHRPC_PORT=30303
OP_GETH_GENESIS_FILE_PATH=mainnet/genesis-l2.json
OP_GETH_SEQUENCER_HTTP=https://mainnet-sequencer.base.org
# [optional] used to enable geth stats:
# OP_GETH_ETH_STATS=nodename:secret@host:port

# [required] replace with your preferred L1 (Ethereum, not Base) node RPC URL:
OP_NODE_L1_ETH_RPC={{ }}

# [required] replace with your preferred L1 CL beacon endpoint:
OP_NODE_L1_BEACON={{ }}

# auth secret used by op-geth engine API:
OP_NODE_L2_ENGINE_AUTH_RAW={{ }}

OP_NODE_L2_ENGINE_AUTH=/tmp/engine-auth-jwt
OP_NODE_L2_ENGINE_RPC=http://base-geth:8551
OP_NODE_LOG_LEVEL=info
OP_NODE_METRICS_ADDR=0.0.0.0
OP_NODE_METRICS_ENABLED=true
OP_NODE_METRICS_PORT=7300
OP_NODE_NETWORK=base-mainnet
OP_NODE_P2P_AGENT=base
OP_NODE_P2P_BOOTNODES=enr:-J24QNz9lbrKbN4iSmmjtnr7SjUMk4zB7f1krHZcTZx-JRKZd0kA2gjufUROD6T3sOWDVDnFJRvqBBo62zuF-hYCohOGAYiOoEyEgmlkgnY0gmlwhAPniryHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQKNVFlCxh_B-716tTs-h1vMzZkSs1FTu_OYTNjgufplG4N0Y3CCJAaDdWRwgiQG,enr:-J24QH-f1wt99sfpHy4c0QJM-NfmsIfmlLAMMcgZCUEgKG_BBYFc6FwYgaMJMQN5dsRBJApIok0jFn-9CS842lGpLmqGAYiOoDRAgmlkgnY0gmlwhLhIgb2Hb3BzdGFja4OFQgCJc2VjcDI1NmsxoQJ9FTIv8B9myn1MWaC_2lJ-sMoeCDkusCsk4BYHjjCq04N0Y3CCJAaDdWRwgiQG,enr:-J24QDXyyxvQYsd0yfsN0cRr1lZ1N11zGTplMNlW4xNEc7LkPXh0NAJ9iSOVdRO95GPYAIc6xmyoCCG6_0JxdL3a0zaGAYiOoAjFgmlkgnY0gmlwhAPckbGHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQJwoS7tzwxqXSyFL7g0JM-KWVbgvjfB8JA__T7yY_cYboN0Y3CCJAaDdWRwgiQG,enr:-J24QHmGyBwUZXIcsGYMaUqGGSl4CFdx9Tozu-vQCn5bHIQbR7On7dZbU61vYvfrJr30t0iahSqhc64J46MnUO2JvQaGAYiOoCKKgmlkgnY0gmlwhAPnCzSHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQINc4fSijfbNIiGhcgvwjsjxVFJHUstK9L1T8OTKUjgloN0Y3CCJAaDdWRwgiQG,enr:-J24QG3ypT4xSu0gjb5PABCmVxZqBjVw9ca7pvsI8jl4KATYAnxBmfkaIuEqy9sKvDHKuNCsy57WwK9wTt2aQgcaDDyGAYiOoGAXgmlkgnY0gmlwhDbGmZaHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQIeAK_--tcLEiu7HvoUlbV52MspE0uCocsx1f_rYvRenIN0Y3CCJAaDdWRwgiQG
OP_NODE_P2P_LISTEN_IP=0.0.0.0
OP_NODE_P2P_LISTEN_TCP_PORT=9222
OP_NODE_P2P_LISTEN_UDP_PORT=9222
OP_NODE_RPC_ADDR=0.0.0.0
OP_NODE_RPC_PORT=8545
OP_NODE_SNAPSHOT_LOG=/tmp/op-node-snapshot-log
OP_NODE_VERIFIER_L1_CONFS=4
OP_NODE_ROLLUP_LOAD_PROTOCOL_VERSIONS=true
OP_NODE_BETA_EXTRA_NETWORKS=true
OP_NODE_ROLLUP_CONFIG=mainnet/rollup.json