Senders recovery MdbxKV error

[admincheg]

System information

Erigon version: 2.40.0-dev-2af7ec56 (builded from 1.0.2 tag)

OS & Version: Linux (Docker)

Commit hash: 3da15fc

Erigon Command (with flags/config):

command:
  - "--datadir"
  - "/srv/bsc/data/"
  - "--ethash.dagdir"
  - "/srv/bsc/data/"
  - "--http"
  - "--http.port"
  - "8545"
  - "--http.addr"
  - "0.0.0.0"
  - "--http.vhosts"
  - "*"
  - "--http.corsdomain"
  - "bsc"
  - "--http.api"
  - "eth,net,web3,debug,admin,personal"
  - "--authrpc.addr"
  - "0.0.0.0"
  - "--authrpc.port"
  - "8551"
  - "--authrpc.vhosts"
  - "*"
  - "--authrpc.jwtsecret"
  - "/srv/bsc/data/jwt.hex"
  - "--metrics"
  - "--metrics.addr"
  - "0.0.0.0"
  - "--metrics.port"
  - "6061"
  - "--ws"
  - "--maxpeers"
  - "200"
  - "--chain"
  - "bsc"
  - "--p2p.protocol"
  - "66"
  - "--bodies.cache"
  - "214748364800"
  - "--batchSize"
  - "4096M"
  - "--db.pagesize"
  - "16k"

Concensus Layer: None (internal)

Concensus Layer Command (with flags/config): None

Chain/Network: bsc

Expected behaviour

Pass synchronization

Actual behaviour

Fail somewhere in MdbxKV at Senders stage.

Steps to reproduce the behaviour

1. Start syncing
2. Try a lot of advises from internet for pass Headers syncing stage (unwind + some additional arguments)
3. Wait
4. Got stuck at Senders syncing stage

Backtrace

[INFO] [04-30|07:13:26.479] [6/15 Senders] Flushed buffer file       name=/srv/bsc/data/temp/erigon-sortable-buf-1490780490
[INFO] [04-30|07:13:50.951] [6/15 Senders] Recovery                  block_number=25226906 ch=0/10000
[INFO] [04-30|07:14:12.188] [txpool] stat                            pending=1 baseFee=0 queued=91 alloc=4.4GB sys=6.0GB
[INFO] [04-30|07:14:20.951] [6/15 Senders] Recovery                  block_number=25278648 ch=0/10000
[INFO] [04-30|07:14:31.752] [6/15 Senders] Flushed buffer file       name=/srv/bsc/data/temp/erigon-sortable-buf-206732764
[INFO] [04-30|07:14:50.951] [6/15 Senders] Recovery                  block_number=25333132 ch=0/10000
[INFO] [04-30|07:15:11.399] [p2p] GoodPeers                          eth66=59
[INFO] [04-30|07:15:12.188] [txpool] stat                            pending=1 baseFee=0 queued=95 alloc=5.2GB sys=6.0GB
[INFO] [04-30|07:15:20.951] [6/15 Senders] Recovery                  block_number=25389859 ch=0/10000
[INFO] [04-30|07:15:36.363] [6/15 Senders] Flushed buffer file       name=/srv/bsc/data/temp/erigon-sortable-buf-2527265756
[INFO] [04-30|07:15:50.951] [6/15 Senders] Recovery                  block_number=25446385 ch=0/10000
[INFO] [04-30|07:16:12.188] [txpool] stat                            pending=1 baseFee=0 queued=99 alloc=3.5GB sys=6.0GB
[EROR] [04-30|07:16:14.945] failed ReadTransactionByHash             hash=0x6fec7acf1d28abbe42b1a51307f3036637c0fc5f7b13b2749f68ef19b9a93965 block=25487600 err="failed MdbxKV cursor.Next(): mdbx_cursor_get: MDBX_PAGE_NOTFOUND: Requested page not found"
[WARN] [04-30|07:16:14.945] [6/15 Senders] ReadCanonicalBodyWithTransactions can't find block num=25487600 hash=0x6fec7acf1d28abbe42b1a51307f3036637c0fc5f7b13b2749f68ef19b9a93965
[EROR] [04-30|07:16:15.307] Staged Sync                              err="[6/15 Senders] failed MdbxKV cursor.Next(): mdbx_cursor_get: MDBX_BAD_TXN: Transaction is not valid for requested operation, e.g. had errored and be must aborted, has a child, or is invalid"
[INFO] [04-30|07:16:16.062] [6/15 Senders] Started                   from=22999999 to=27793614
[INFO] [04-30|07:16:46.063] [6/15 Senders] Recovery                  block_number=23073075 ch=10000/10000
[INFO] [04-30|07:16:59.611] [6/15 Senders] Flushed buffer file       name=/srv/bsc/data/temp/erigon-sortable-buf-2354223747
[INFO] [04-30|07:17:11.399] [p2p] GoodPeers                          eth66=60
[INFO] [04-30|07:17:12.189] [txpool] stat                            pending=1 baseFee=0 queued=105 alloc=4.7GB sys=8.4GB
[INFO] [04-30|07:17:16.062] [6/15 Senders] Recovery                  block_number=23145546 ch=9863/10000
[INFO] [04-30|07:17:41.839] [6/15 Senders] Flushed buffer file       name=/srv/bsc/data/temp/erigon-sortable-buf-1626203942
[INFO] [04-30|07:17:46.062] [6/15 Senders] Recovery                  block_number=23223562 ch=9983/10000

Additionally

I already checked out all my disks, raid array, memory and all what I can to do with hardware, platform seems alive. Dmesg and syslog doesn't reports about any issue. Also I tried to use snapshot - but actually it's broken and I did not decompress it.

[qinglin89]

Can you try with the latest snapshot ?

[setunapo]

not reproduced with latest release, close right now.
could reopen if still have this issue.

[admincheg]

Interesting fact. After node upgrading from v1.0.2 to v1.0.8 we pass 5th step and now stuck at the 6th :D
It's still same error:

[INFO] [05-25|18:51:27.143] [txpool] stat pending=1 baseFee=0 queued=624 alloc=7.0GB sys=17.2GB
[INFO] [05-25|18:51:49.421] [6/15 Senders] Recovery block_number=25393788 ch=9978/10000
[INFO] [05-25|18:51:58.290] [6/15 Senders] Flushed buffer file name=/srv/bsc/data/temp/erigon-sortable-buf-1262132006
[EROR] [05-25|18:52:19.306] failed ReadTransactionByHash hash=0x001b4635907ba90ae2664608717da22923ea79d74758c488e05d3848eee2fba7 block=25487589 err="failed MdbxKV cursor.Next(): mdbx_cursor_get: MDBX_PAGE_NOTFOUND: Requested page not found"
[WARN] [05-25|18:52:19.306] [6/15 Senders] ReadCanonicalBodyWithTransactions can't find block num=25487589 hash=0x001b4635907ba90ae2664608717da22923ea79d74758c488e05d3848eee2fba7
[INFO] [05-25|18:52:19.421] [6/15 Senders] Recovery block_number=25477845 ch=9687/10000
[EROR] [05-25|18:52:19.936] Staged Sync err="[6/15 Senders] failed MdbxKV cursor.Next(): mdbx_cursor_get: MDBX_BAD_TXN: Transaction is not valid for requested operation, e.g. had errored and be must aborted, has a child, or is invalid"
[INFO] [05-25|18:52:20.702] [6/15 Senders] Started from=22999999 to=28523878
[INFO] [05-25|18:52:27.151] [txpool] stat pending=1 baseFee=0 queued=636 alloc=9.0GB sys=17.2GB
[INFO] [05-25|18:52:50.709] [6/15 Senders] Recovery block_number=23072715 ch=9978/10000
[INFO] [05-25|18:53:04.344] [6/15 Senders] Flushed buffer file name=/srv/bsc/data/temp/erigon-sortable-buf-2696851228
[INFO] [05-25|18:53:20.702] [6/15 Senders] Recovery block_number=23144795 ch=9972/10000
[INFO] [05-25|18:53:26.839] [p2p] GoodPeers eth66=49