Invalid Blocks after node restart (crash?) v4.0.0

[Stonygan]

Summary

Tonight 2 independent nodes crashed on the same block (#3470797) and after the restart both reported the valid block #3470790 (hash: 71d037fbaa1e7d04254dfc49fb2816f91766c50b639d47f0f60c451397f76c0b) as invalid and jumped back to #3470789.
The nodes then remained stuck on this block until one of the two nodes mined a new block with its own chain. #3470790 with hash e477d9c4ca899408fc8073f686c7e9c9aa8e7444a3a9e381947ab2c7ab1f9c3f, which the other node then also jumped to. The node recognizes that there is a longer chain, but does not jump to it even several hours later.
A reconsiderblock of the invalid block does not produce an error and no output in the defi-cli, but is still displayed as invalid in the debug.log.
An invalidateblock of the incorrectly created block marks it as invalid, a reconsiderblock on the valid block that was executed before the invalidate without error output then reports:

~/.defi/defi-cli reconsiderblock 71d037fbaa1e7d04254dfc49fb2816f91766c50b639d47f0f60c451397f76c0b
error code: -20
error message:
 (code 16)

The node can no longer switch to the correct chain.
We have the same behavior with a normal restart of the node with defi-cli stop. This is also the reason why the snapshots (all those created after the hard fork) no longer work, as they always recognize the following block as invalid.

Maybe others can also observe this behavior, these are the first findings that we noticed in the first few days after the hardfork on the mainnet. Anyone who can contribute further details, please feel free to add them.

Steps to Reproduce

Stop the node and start again after few minutes.

Environment

[Please fill all of the following or NA if not applicable]

• Node Version: v4.0.0 and v4.0.1
• Block height on bug if applicable: see above
• TX or TX type on bug if applicable: see above
• OS with version: Debian 11/12/ubuntu 22
• Any other relevant environment info:

Notecrash in the debug.log:

2023-11-19T04:41:34Z UpdateTip: new best=9fb3beb9ed5f4fcba08097b241f3bc6a6eb9d28cd42260e98c02f4c89b2b77b0 height=3470797 version=0x20000000 log2_work=87.942014 tx=27192192 date='2023-11-19T04
:41:47Z' progress=1.000000 cache=0.0MiB(39txo)
...............................................................................................................................................................................................
...............................................................................................................................................................................................
...............................................................................................................................................................................................
...............................................................................................................................................................................................
.........

[prasannavl]

Could you please help understand which snapshot was this node started from?

[Stonygan]

No Snapshot, inPlace-Upgrade before Hardfork.

[Stonygan]

Another Server today, crash:

2023-11-20T17:02:51Z UpdateTip: new best=e158b5dc5f86ff0df21f48e8bb2faa952328014e4ca77100c5a3b0cf2316bd44 height=3475739
.
.
.
2023-11-20T17:06:37Z UpdateTip: new best=cf085f4304f8765ea42c87b288a2c6c956b61f608281cf572fe8b3158d4dd220 height=3475749
2023-11-20T17:07:51Z Unable to open file /home/xxx/.defi/data/blocks/blk00075.dat
2023-11-20T17:07:51Z ERROR: ReadRawBlockFromDisk: OpenBlockFile failed for FlatFilePos(nFile=75, nPos=114244036)

Noderestart, rollback to Block #3475738:

2023-11-20T17:08:35Z InvalidChainFound: invalid block=e158b5dc5f86ff0df21f48e8bb2faa952328014e4ca77100c5a3b0cf2316bd44 height=3475739

~/.defi/defi-cli reconsiderblock e158b5dc5f86ff0df21f48e8bb2faa952328014e4ca77100c5a3b0cf2316bd44
error code: -20
error message:
 (code 16)

2023-11-20T17:08:35Z ERROR: ConnectBlock: Incorrect EVM block hash in coinbase output

Restart of the Node end up with inconsistent EVM-DB <-> DVM-DB Blockchaindata?

[Stonygan]

Hi all, have another Server today aftera planned restart.
before restart:
2023-11-23T06:58:44Z UpdateTip: new best=5e26d53505f85446bfedc3305ec6b384602263dc71fc0e8b95af70f3ef212cd6 height=3483244
After restart:

2023-11-23T07:03:32Z Loaded best chain: hashBestChain=5e26d53505f85446bfedc3305ec6b384602263dc71fc0e8b95af70f3ef212cd6 height=3483244
2023-11-23T07:03:37Z ERROR: ConnectBlock: ApplyCustomTx on 20cc8d090a108e33e459e30951d8f90890279529a6207fd04e8582513b3759f3 failed with EvmTx: evm tx failed to queue [exe
cute_tx] nonce check failed. Account nonce 3, signed_tx nonce 2

2023-11-23T07:03:37Z InvalidChainFound: invalid block=1921321ad7a10f91ce634f799c3753573395760c9de9338d0ae740831c474a9a  height=3483245

Not possible without Snapshot to geht this Node back to the chain.

~/.defi/defi-cli reconsiderblock 1921321ad7a10f91ce634f799c3753573395760c9de9338d0ae740831c474a9a
error code: -20
error message:
bad-custom-tx (code 68)

[neuriGH]

Have you activated pruning on your nodes? As written in #2704, I think the problem could have something to do with this.

[prasannavl]

Closing outdated / stale issues.