S5 fails state-hash when primed at 65436418

[matejmode]

When priming S5 carmen up till 51M blocks the state-hash is correct. When priming last block of previous patch it fails.
xapi150 https://jenkins.fantom.network/job/Aida/job/CustomBuild/job/s5-priming-tests/10/

This issue also occured on xapi310:
https://jenkins.fantom.network/job/Aida/job/AidaDb/job/pre-release-mainnet/20/

On xapi310 I started zero-last block without priming to confirm the database is correct.
https://jenkins.fantom.network/job/Aida/job/AidaDb/job/pre-release-mainnet/21/

[wsodsong]

The last known block where priming works correctly is 51359673.

Error at when priming to block 65436417:

 2023/12/09 01:31:36 �[32mNOTICE�[0m: Total elapsed time: 30h57m21s; last block 65436418; total transaction rate ~0.00 Tx/s, ~0.00 MGas/s
 record-replay: CloseSubstateDB
 unexpected hash for Live block 65436418
 wanted 0x114017bca8cfb3bd9bd3a216bea4ff420a739da9e55a7fbf31853e783fa7259c
    got 0x5810cdb26b017de4065cdb16922e40cf871d20b2a44d5a0849870d9589f0d374

Command to reproduce the error.

build/aida-vm-sdb substate --vm-impl lfvm --aida-db=/var/opera/Aida/mainnet-data/aida-db --update-buffer-size 4000 --db-tmp=/var/opera/Aida/dbtmpjenkins --validate-state-hash --db-impl carmen --db-variant go-file --carmen-schema 5 --archive --archive-variant s5 --validate-tx --cpu-profile cpu-profile.dat --memory-profile mem-profile.dat --memory-breakdown --continue-on-failure 65436418 65436418

[matejmode]

Confirming that the database is correct. Because ithout priming it succeeds: https://jenkins.fantom.network/job/Aida/job/AidaDb/job/pre-release-mainnet/21/

[matejmode]

Priming keeps failing after 55M with develop and most recent tosca and carmen.
https://jenkins.fantom.network/job/Aida/job/CustomBuild/job/s5-priming-tests/16/

[matejmode]

Found out that the block 54535894 has incorrect ressurection order generated in deletedDb generation has to be fixed.