"Head state missing, repairing chain" I lost many blocks

[MANOJKUMARCH]

Hi there,

I have run geth as private chain. We were doing a load test and injected 10K transactions into the chain. The chain was able to handle the load for 3 times, but fourth time the chain is crashed. And when I restarted the chain, it is rewound to old state that is 2 months old state. As a result I lost close to 600000+ blocks.

Is there a way I can get the blocks back.

System information

Geth version: 1.9.9-stable
OS & Version: Linux - Ubuntu 18.0.4
Commit hash : (if develop)

Expected behaviour

Geth to repair the chain and start from the latest state

Actual behaviour

Geth has rewound the chain to old state with is 2 months old

WARN [04-30|08:52:03.578] Head state missing, repairing chain      number=3231085 hash=143d04…20a4e5
INFO [04-30|08:52:47.776] Rewound blockchain to past state         number=2687939 hash=f07aee…c540c6
INFO [04-30|08:52:47.778] Loaded most recent local header          number=3231085 hash=143d04…20a4e5 td=2132445952434 age=1h8m43s
INFO [04-30|08:52:47.778] Loaded most recent local full block      number=2687939 hash=f07aee…c540c6 td=1676807286165 age=2mo2w5d
INFO [04-30|08:52:47.778] Loaded most recent local fast block      number=3231085 hash=143d04…20a4e5 td=2132445952434 age=1h8m43s

WARN [04-30|08:53:24.246] Truncating freezer table                 database=/home/ctadmindev/CT91_Dev/geth/chaindata/ancient table=headers items=3141085 limit=2687940
WARN [04-30|08:53:24.261] Truncating freezer table                 database=/home/ctadmindev/CT91_Dev/geth/chaindata/ancient table=hashes  items=3141085 limit=2687940
WARN [04-30|08:53:24.263] Truncating freezer table                 database=/home/ctadmindev/CT91_Dev/geth/chaindata/ancient table=bodies  items=3141085 limit=2687940
WARN [04-30|08:53:24.264] Truncating freezer table                 database=/home/ctadmindev/CT91_Dev/geth/chaindata/ancient table=receipts items=3141085 limit=2687940
WARN [04-30|08:53:24.265] Truncating freezer table                 database=/home/ctadmindev/CT91_Dev/geth/chaindata/ancient table=diffs    items=3141085 limit=2687940
INFO [04-30|08:53:24.265] Rewind ancient data                      number=2687939
**WARN [04-30|08:53:24.265] Truncate ancient chain                   from=3231085 to=2687939**

I have looked into issue # 19124, and tried to export the chain. But it did not work, as the export failed at block number 2687940 saying it is not found.

Any chance or way to get the blocks? Also would like mention that this is a private cchain with no other nodes in network.

Is this a bug or expected behaviour?

Please help if there is a way to get the lost blocks.

Regards,
Manoj

[holiman]

It's obviously not an "expceted behaviour" for geth to crash, but when geth does crash, it is expected behaviour that a roll-back can occur. Geth does in-memory pruning of state, and only writes out data every once in a while. If geth crashes, it may well have to discard a whole lot of blocks to go back to that point in time where it most recently stored the state to disk.

Short answer: no, you cannot get the blocks back.
If you want more reliability in the future, consider running more nodes, and/or disable pruning -- a.k.a run archive mode.

Hope I've answered your questions, and since I don't see anything more actionable about this particular ticket, I'll close it.

[jasper22]

Is there any option to save state more frequently ??

I guess I crash into similar issue with official Geth client: Geth keep truncating ancient chain + rewind blockchain ? Why ? 😥 in "didn't touch anything" scenario.

10 hours of syncing is out of window because of unknown error :( it always come back to first block. I'm really frustrated here.....

[holiman]

Is there any option to save state more frequently ??

Yes. gcmode=archive means 'save state at every block'. As I remember it, there was a way to set the flush-time to an arbitrary time, however,

TrieTimeLimit       time.Duration // Time limit after which to flush the current in-memory trie to disk

Afaict this is not configurable via CLI or toml. @karalabe am I misremembering?