Node fails during sync. with writer-errors

[ifavo]

OS Version:

CentOS 7

3.10.0-1160.71.1.el7.x86_64 #1 SMP Tue Jun 28 15:37:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Commit hash : (if develop)

• 61788ddef7058e3966cea8e9067a22bb0360fbd1
• one commit before 2.0.2

Expected behavior

Continuous synchronizing

Actual behavior

INFO[03-24|06:28:49] optimizer started                        pkg=optimizer
INFO[03-24|06:28:49] waiting for synchronization...           pkg=node
INFO[03-24|06:28:54] imported blocks (23)                     pkg=node txs=78 mgas=4.919 et=127.770ms|1.291s mgas/s=30.761 id=[#10330188…1402722d]
INFO[03-24|06:28:56] imported blocks (56)                     pkg=node txs=106 mgas=6.465 et=151.734ms|1.914s mgas/s=34.644 id=[#10330244…ab2ab4df]
WARN[03-24|06:28:56] optimizer interrupted                    pkg=optimizer error="prune tries: dump trie nodes: leveldb/table: Writer: keys are not in increasing order: \"\\x03chain.txi\\x004θ\\x92\\xf2f\\xa4J\\u007f\\xa1\\vW\\xff\\f8ۛ}\\xc9bp\\x91\\x98\\xfc\\xefOR\\x01\\xe6/\\xa0\\x01\\x94\\xbf\\xf2\\x85\\x00\\x00\\x00\", \"\\x03chain.heads\\x00\\x9d\\x03٤\\xe3\\xa91$\\xe6\\xa4r\\x0f\\x1cC\\xcfh\\xee\\xc4\\x1d\\x14\\x95\\xf4\\xf8\\xdc/\\xb4\\xa0\\xc1Y\\x9b.\\x01\\xe6\\xc3\\xe2\\x9b\\x00\\x00\\x00\""
EROR[03-24|06:28:56] failed to process block                  pkg=node err="commit state: state: leveldb/table: Writer: keys are not in increasing order: \"\\x03chain.txi\\x004θ\\x92\\xf2f\\xa4J\\u007f\\xa1\\vW\\xff\\f8ۛ}\\xc9bp\\x91\\x98\\xfc\\xefOR\\x01\\xe6/\\xa0\\x01\\x94\\xbf\\xf2\\x85\\x00\\x00\\x00\", \"\\x03chain.heads\\x00\\x9d\\x03٤\\xe3\\xa91$\\xe6\\xa4r\\x0f\\x1cC\\xcfh\\xee\\xc4\\x1d\\x14\\x95\\xf4\\xf8\\xdc/\\xb4\\xa0\\xc1Y\\x9b.\\x01\\xe6\\xc3\\xe2\\x9b\\x00\\x00\\x00\""
EROR[03-24|06:28:58] failed to process block                  pkg=node err="commit state: state: leveldb/table: Writer: keys are not in increasing order: \"\\x03chain.txi\\x004θ\\x92\\xf2f\\xa4J\\u007f\\xa1\\vW\\xff\\f8ۛ}\\xc9bp\\x91\\x98\\xfc\\xefOR\\x01\\xe6/\\xa0\\x01\\x94\\xbf\\xf2\\x85\\x00\\x00\\x00\", \"\\x03chain.heads\\x00\\x9d\\x03٤\\xe3\\xa91$\\xe6\\xa4r\\x0f\\x1cC\\xcfh\\xee\\xc4\\x1d\\x14\\x95\\xf4\\xf8\\xdc/\\xb4\\xa0\\xc1Y\\x9b.\\x01\\xe6\\xc3\\xe2\\x9b\\x00\\x00\\x00\""
EROR[03-24|06:29:00] failed to process block

stopping and restarting continues after the error:

INFO[03-24|06:30:06] optimizer started                        pkg=optimizer
INFO[03-24|06:30:06] waiting for synchronization...           pkg=node
INFO[03-24|06:30:12] imported blocks (15)                     pkg=node txs=92 mgas=4.597 et=195.133ms|2.382s mgas/s=21.799 id=[#10330269…bcff4c2a]
INFO[03-24|06:30:14] imported blocks (24)                     pkg=node txs=32 mgas=1.533 et=109.387ms|1.913s mgas/s=12.798 id=[#10330293…9422d330]
INFO[03-24|06:30:16] imported blocks (36)                     pkg=node txs=51 mgas=2.359 et=130.270ms|1.992s mgas/s=16.037 id=[#10330329…c50ce758]

Steps to reproduce the behavior

no reproduction instructions available:

• clone release to server
• make
• ./bin/thor --network main --data-dir /data/main --api-addr localhost:18669 --p2p-port 21235 --api-cors '*'
• did a few stops & starts during sync which were all graceful (CTRL+C)

[kgapos]

This is a known issue in LevelDB that is also present in Ethereum. Are you sure all stops were graceful? A single SIGKILL or a power outage at the wrong time will lead to this corruption.

[ifavo]

I've read that thread too, did not suggest a real solution.
I deleted the data directory and started a new sync, it is past the same point now.

Thanks!

[qianbin]

@ifavo Was the node ever abnormally killed, before the error?

[ifavo]

@ifavo Was the node ever abnormally killed, before the error?

Not that I was aware of, it is a fresh one that was running for ~48hrs, but as always with these things, I can not be a 100% certain. I am however certain that it was running for ~24hrs without interruption before the error.

I am syncing in a screen at the moment and stopped with CTRL+C at beginning a few times, because I wanted to test a few parameters (bootnode, peers, cache, etc.).