Devel data migration leads to panic & hash mismatch

[geomad]

Erigon version: v2021.10.03-1257-gd7d698f56-dirty (devel)
Commit hash : d7d698f
Machine specs: 64 GB ram, AMD 3900x, 4TB Nvme gen4 ssd
OS & Version: Linux Debian

Erigon was running fine (with --snap.stop) after having recovered from #4323
Updating to latest devel triggered the data migration, which when finished lead to panic.

.
.
.
[INFO] [06-08|01:30:18.564] TruncateBlocks                           block=14087218
[INFO] [06-08|01:30:38.564] TruncateBlocks                           block=14077462
[INFO] [06-08|01:30:58.559] TruncateBlocks                           block=14066988
[INFO] [06-08|01:31:18.560] TruncateBlocks                           block=14056050
[INFO] [06-08|01:31:38.559] TruncateBlocks                           block=14045141
[INFO] [06-08|01:31:58.562] TruncateBlocks                           block=14034157
[INFO] [06-08|01:32:18.562] TruncateBlocks                           block=14024588
[INFO] [06-08|01:32:38.565] TruncateBlocks                           block=14014725
[INFO] [06-08|01:32:58.564] TruncateBlocks                           block=14005111
[INFO] [06-08|01:34:28.311] Applied migration                        name=reset_blocks_3
[INFO] [06-08|01:34:28.314] Updated DB schema to                     version=6.0.0
[INFO] [06-08|01:34:29.416] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, DAO Support: true, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Terminal Total Difficulty: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
[INFO] [06-08|01:34:29.417] Initialising Ethereum protocol           network=1
[INFO] [06-08|01:34:29.417] Using snapshots                          on=true
[INFO] [06-08|01:34:29.417] Effective                                prune_flags= snapshot_flags="--snapshots=true --snap.stop=true"
[INFO] [06-08|01:34:32.545] Disk storage enabled for ethash DAGs     dir=/erigon/erigon-ethash count=2
[INFO] [06-08|01:34:32.548] Starting private RPC server              on=127.0.0.1:9090
[INFO] [06-08|01:34:32.548] new subscription to logs established
[INFO] [06-08|01:34:32.558] Started P2P networking                   version=66 self=<enode> name=erigon/v2022.99.99-dev-d7d698f5/linux-amd64/go1.18.2
[INFO] [06-08|01:34:32.559] [Snapshots] Fetching torrent files metadata
[INFO] [06-08|01:34:34.120] [Snapshots] Stat                         blocks=13999999 segments=14499999 indices=13999999 alloc=2.1Gb sys=2.4Gb
[INFO] [06-08|01:34:54.120] [1/16 Headers] Writing total difficulty index for snapshots block_num=3191850
[INFO] [06-08|01:35:14.121] [1/16 Headers] Writing total difficulty index for snapshots block_num=5719442
[INFO] [06-08|01:35:34.121] [1/16 Headers] Writing total difficulty index for snapshots block_num=8118262
[INFO] [06-08|01:35:57.155] [1/16 Headers] Writing total difficulty index for snapshots block_num=11184812
[INFO] [06-08|01:36:14.120] [1/16 Headers] Writing total difficulty index for snapshots block_num=13972821
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x112a449]
goroutine 49198 [running]:
github.com/ledgerwatch/erigon-lib/compress.(*Getter).nextPos(0xc0e7786330?, 0x98?)
        github.com/ledgerwatch/erigon-lib@v0.0.0-20220605213234-a77e6425eb24/compress/decompress.go:312 +0x29
github.com/ledgerwatch/erigon-lib/compress.(*Getter).Next(0xc080d8de70, {0xc080d8cdf0, 0x0, 0x1000})
        github.com/ledgerwatch/erigon-lib@v0.0.0-20220605213234-a77e6425eb24/compress/decompress.go:389 +0x59
github.com/ledgerwatch/erigon/turbo/snapshotsync.forEachAsync.func1.1()
        github.com/ledgerwatch/erigon/turbo/snapshotsync/block_snapshots.go:1599 +0x16f
github.com/ledgerwatch/erigon-lib/compress.(*Decompressor).WithReadAhead(0xc053d77080, 0xc109d88768)
        github.com/ledgerwatch/erigon-lib@v0.0.0-20220605213234-a77e6425eb24/compress/decompress.go:287 +0xa2
github.com/ledgerwatch/erigon/turbo/snapshotsync.forEachAsync.func1()
        github.com/ledgerwatch/erigon/turbo/snapshotsync/block_snapshots.go:1594 +0xb4
created by github.com/ledgerwatch/erigon/turbo/snapshotsync.forEachAsync
        github.com/ledgerwatch/erigon/turbo/snapshotsync/block_snapshots.go:1592 +0xc5

./build/bin/downloader torrent_hashes --verify --datadir=<datadir> found hash missmatches

INFO[06-08|01:50:41.222] [Snapshots] Verify                       Progress=82.95%
INFO[06-08|01:50:46.220] [Snapshots] Verify                       Progress=84.72%
INFO[06-08|01:50:51.221] [Snapshots] Verify                       Progress=86.38%
INFO[06-08|01:50:56.223] [Snapshots] Verify                       Progress=87.87%
INFO[06-08|01:51:01.220] [Snapshots] Verify                       Progress=89.08%
INFO[06-08|01:51:06.221] [Snapshots] Verify                       Progress=90.51%
EROR[06-08|01:51:11.166] [Snapshots] Verify hash mismatch         at piece=0 file=v1-014000-014500-bodies.seg
EROR[06-08|01:51:11.170] [Snapshots] Verify hash mismatch         at piece=1 file=v1-014000-014500-bodies.seg
EROR[06-08|01:51:11.172] [Snapshots] Verify hash mismatch         at piece=2 file=v1-014000-014500-bodies.seg
EROR[06-08|01:51:11.174] [Snapshots] Verify hash mismatch         at piece=3 file=v1-014000-014500-bodies.seg
EROR[06-08|01:51:11.177] [Snapshots] Verify hash mismatch         at piece=4 file=v1-014000-014500-bodies.seg
EROR[06-08|01:51:11.179] [Snapshots] Verify hash mismatch         at piece=5 file=v1-014000-014500-bodies.seg
EROR[06-08|01:51:11.181] [Snapshots] Verify hash mismatch         at piece=6 file=v1-014000-014500-bodies.seg
EROR[06-08|01:51:11.183] [Snapshots] Verify hash mismatch         at piece=7 file=v1-014000-014500-bodies.seg
EROR[06-08|01:51:11.185] [Snapshots] Verify hash mismatch         at piece=8 file=v1-014000-014500-bodies.seg
EROR[06-08|01:51:11.188] [Snapshots] Verify hash mismatch         at piece=0 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.190] [Snapshots] Verify hash mismatch         at piece=1 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.192] [Snapshots] Verify hash mismatch         at piece=2 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.194] [Snapshots] Verify hash mismatch         at piece=3 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.197] [Snapshots] Verify hash mismatch         at piece=4 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.199] [Snapshots] Verify hash mismatch         at piece=5 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.201] [Snapshots] Verify hash mismatch         at piece=6 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.204] [Snapshots] Verify hash mismatch         at piece=7 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.206] [Snapshots] Verify hash mismatch         at piece=8 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.208] [Snapshots] Verify hash mismatch         at piece=9 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.210] [Snapshots] Verify hash mismatch         at piece=10 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.213] [Snapshots] Verify hash mismatch         at piece=11 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.215] [Snapshots] Verify hash mismatch         at piece=12 file=v1-014000-014500-headers.seg
EROR[06-08|01:51:11.217] [Snapshots] Verify hash mismatch         at piece=13 file=v1-014000-014500-headers.seg
.
.
.
EROR[06-08|01:51:32.204] [Snapshots] Verify hash mismatch         at piece=8668 file=v1-014000-014500-transactions.seg
EROR[06-08|01:51:32.206] [Snapshots] Verify hash mismatch         at piece=8669 file=v1-014000-014500-transactions.seg
EROR[06-08|01:51:32.209] [Snapshots] Verify hash mismatch         at piece=8670 file=v1-014000-014500-transactions.seg
EROR[06-08|01:51:32.211] [Snapshots] Verify hash mismatch         at piece=8671 file=v1-014000-014500-transactions.seg
EROR[06-08|01:51:32.213] [Snapshots] Verify hash mismatch         at piece=8672 file=v1-014000-014500-transactions.seg
EROR[06-08|01:51:32.215] [Snapshots] Verify hash mismatch         at piece=8673 file=v1-014000-014500-transactions.seg
EROR[06-08|01:51:32.220] [Snapshots] Verify hash mismatch         at piece=8674 file=v1-014000-014500-transactions.seg
EROR[06-08|01:51:32.224] [Snapshots] Verify hash mismatch         at piece=8675 file=v1-014000-014500-transactions.seg
EROR[06-08|01:51:32.228] [Snapshots] Verify hash mismatch         at piece=8676 file=v1-014000-014500-transactions.seg
EROR[06-08|01:51:32.231] [Snapshots] Verify hash mismatch         at piece=8677 file=v1-014000-014500-transactions.seg
Error: not all files are valid

[geomad]

Went ahead and

rm <datadir>/snapshots/v1-014000*
./build/bin/integration stage_headers --reset --datadir=<datadir>
./build/bin/downloader torrent_hashes --verify --datadir=<datadir>

Looks like its running fine now but maybe the issue should be investigated since it breaks erigon on update.

[AskAlexSharov]

tnx will check

[AskAlexSharov]

fixed by #4404