Erigon mainnet stuck at erigon | [INFO] [06-21|07:01:47.459] [p2p] GoodPeers eth66=43

[hritique]

System information

Erigon version: ./erigon --version

erigon version 2022.06.6-alpha-77240899

OS & Version: Windows/Linux/OSX

Linux - Ubuntu 20.04

Commit hash :

7724089985bf9cb8bb7d1bde92892256a6af859f

Expected behaviour

The erigon node should sync properly and server RPC queries

Actual behaviour

The node gets stuck and stops syncing. The rpcdaemon also stops serving queries like eth_blockNumber.
It starts to work again after a restart and then stops again after a while.

Steps to reproduce the behaviour

We use docker to run the processes. Below is our docker-compose config.

version: '3.4'
services:
  erigon:
    image: <IMAGE>
    container_name: erigon
    stop_grace_period: 1m
    user: <USER>
    ports:
      - '30303:30303'
      - '6060:6060'
    volumes:
      - <DATADIR>:/.local/share/erigon'
    command:
      - erigon
      - --metrics
      - --metrics.addr
      - '0.0.0.0'
      - --metrics.port
      - '6060'
      - --private.api.addr
      - '0.0.0.0:9090'
    ulimits:
       nofile:
         soft: 500000
         hard: 500000

  rpcdaemon:
    image: <IMAGE>
    container_name: rpcdaemon
    stop_grace_period: 1m
    user: <USER>
    ports:
      - '8545:8545'
    volumes:
      - '<DATADIR>:/.local/share/erigon'
    command:
      - rpcdaemon
      - --ws
      - --http.api
      - 'eth,debug,net,trace,web3,txpool,erigon'
      - --http.addr
      - '0.0.0.0'
      - --http.vhosts
      - '*'
      - --private.api.addr
      - 'erigon:9090'

Backtrace

Below are the logs which continue forever until we restart

erigon       | [INFO] [06-21|05:41:49.278] [1/16 Headers] Waiting for headers...    from=15000744
erigon       | [INFO] [06-21|05:41:50.751] [txpool] stat                            block=15000744 pending=618 baseFee=30000 queued=30000 alloc=3.5GB sys=8.6GB
erigon       | [INFO] [06-21|05:41:53.007] [1/16 Headers] Processed                 highest inserted=15000745 age=5s
erigon       | [INFO] [06-21|05:41:53.070] [6/16 Execution] Completed on            block=15000745
erigon       | [INFO] [06-21|05:41:53.165] Timings (slower than 50ms)               Headers=3.728s Execution=50ms
erigon       | [INFO] [06-21|05:41:53.165] Tables                                   PlainState=67.9GB AccountChangeSet=153.6GB StorageChangeSet=224.6GB BlockTransaction=9.8GB TransactionLog=406.6GB FreeList=58.8MB ReclaimableSpace=58.8GB
erigon       | [INFO] [06-21|05:41:53.214] Commit cycle                             in=48.506307ms
erigon       | [INFO] [06-21|05:43:47.458] [p2p] GoodPeers                          eth66=43
erigon       | [INFO] [06-21|05:45:47.459] [p2p] GoodPeers                          eth66=43
erigon       | [INFO] [06-21|05:47:47.458] [p2p] GoodPeers                          eth66=43
erigon       | [INFO] [06-21|05:49:47.459] [p2p] GoodPeers                          eth66=46
erigon       | [INFO] [06-21|05:51:47.458] [p2p] GoodPeers                          eth66=45
erigon       | [INFO] [06-21|05:53:47.459] [p2p] GoodPeers                          eth66=45
erigon       | [INFO] [06-21|05:55:47.459] [p2p] GoodPeers                          eth66=46
erigon       | [INFO] [06-21|05:57:47.458] [p2p] GoodPeers                          eth66=45
erigon       | [INFO] [06-21|05:59:47.459] [p2p] GoodPeers                          eth66=46
erigon       | [INFO] [06-21|06:01:47.458] [p2p] GoodPeers                          eth66=45
erigon       | [INFO] [06-21|06:03:47.458] [p2p] GoodPeers                          eth66=44
erigon       | [INFO] [06-21|06:05:47.458] [p2p] GoodPeers                          eth66=44
erigon       | [INFO] [06-21|06:07:47.459] [p2p] GoodPeers                          eth66=44
erigon       | [INFO] [06-21|06:09:47.458] [p2p] GoodPeers                          eth66=45
erigon       | [INFO] [06-21|06:11:47.458] [p2p] GoodPeers                          eth66=43
erigon       | [INFO] [06-21|06:13:47.459] [p2p] GoodPeers                          eth66=43
erigon       | [INFO] [06-21|06:15:47.458] [p2p] GoodPeers                          eth66=43
erigon       | [INFO] [06-21|06:17:47.459] [p2p] GoodPeers                          eth66=43
erigon       | [INFO] [06-21|06:19:47.458] [p2p] GoodPeers                          eth66=43
erigon       | [INFO] [06-21|06:21:47.459] [p2p] GoodPeers                          eth66=43
erigon       | [INFO] [06-21|06:23:47.458] [p2p] GoodPeers                          eth66=43
erigon       | [INFO] [06-21|06:25:47.459] [p2p] GoodPeers                          eth66=42
erigon       | [INFO] [06-21|06:27:47.458] [p2p] GoodPeers                          eth66=43
erigon       | [INFO] [06-21|06:29:47.459] [p2p] GoodPeers                          eth66=42
erigon       | [INFO] [06-21|06:31:47.458] [p2p] GoodPeers                          eth66=43
erigon       | [INFO] [06-21|06:33:47.459] [p2p] GoodPeers                          eth66=44
erigon       | [INFO] [06-21|06:35:47.459] [p2p] GoodPeers                          eth66=44
erigon       | [INFO] [06-21|06:37:47.459] [p2p] GoodPeers                          eth66=44
erigon       | [INFO] [06-21|06:39:47.458] [p2p] GoodPeers                          eth66=44
erigon       | [INFO] [06-21|06:41:47.458] [p2p] GoodPeers                          eth66=44

[mandrigin]

I see the same behaviour every now and then. So far no luck reproducing it properly. It looks like it behaves better on the current devel branch. Trying to find steps to reproduce and fix.

[mandrigin]

@hritique I will close this one in favor of #4543, looks like the same issue to me.