eth: make traceChain avoid OOM on long-running tracing

[holiman]

This PR changes long-running chain tracing, so that it at some points releases the memory trie db, and switch over to a fresh disk-backed trie.

[rjl493456442]

If the accumulated in memory nodes exceed the limit, shouldn't we flush them into disk and then retrieve a disk-based statedb?
Although I'm not sure if we really want to push junks. Maybe we just want to check if there is a complete state in disk?

[holiman]

Although I'm not sure if we really want to push junks. Maybe we just want to check if there is a complete state in disk?

That's basically what it does. We don't want to push junk, but we start telling the backend "hey, if you happen to find the state on disk, can you please use that instead of using this accumulated junk?"

[rjl493456442]

Gotcha. It can be helpful in the current codebase since we periodically push the state.

[holiman]

Finally got around to testing this on a node..

> debug.getAccessibleState(0,"latest")
13401259
> debug.getAccessibleState(13401260,"latest")
13414706

So if tracing starts at 13401259, then it will have to chug through 13414706-13401259=13447 blocks before the memory is cleared. Should be doable.

INFO [11-02|17:40:22.208] Tracing chain segment                    start=13,401,259 end=13,537,532 current=13,414,693 transactions=2,545,920 elapsed=4h3m22.508541811s
INFO [11-02|17:40:30.543] Tracing chain segment                    start=13,401,259 end=13,537,532 current=13,414,700 transactions=2,547,143 elapsed=4h3m30.843300651s
INFO [11-02|17:40:35.864] Deep froze chain segment                 blocks=3  elapsed=36.277ms           number=13,448,639 hash=c57e2d..edd170
INFO [11-02|17:40:39.146] Tracing chain segment                    start=13,401,259 end=13,537,532 current=13,414,706 transactions=2,548,254 elapsed=4h3m39.446252969s
INFO [11-02|17:40:39.578] Found disk backend for state trie 

Yay!

INFO [11-02|17:40:41.366] Downloader queue stats                   receiptTasks=0 blockTasks=0 itemSize=69.51KiB throttle=3772
WARN [11-02|17:40:43.371] Tracing failed                           hash=dff138..a8826e block=13,414,708 err="tracing failed: nonce too high: address 0xAe45a8240147E6179ec7c9f92c5A18F9a97B3fCA, tx: 680930 state: 680929"
WARN [11-02|17:40:46.235] Chain tracing failed                     start=13,401,259 end=13,537,532 transactions=2,548,566 elapsed=4h3m46.535467752s  err="processing block 13414708 failed: could not apply tx 71 [0xdff1382a7bd005a5e54425f65e937b8d477dfd31c21a78e71a5781f873a8826e]: nonce too high: address 0xAe45a8240147E6179ec7c9f92c5A18F9a97B3fCA, tx: 680930 state: 680929"

Nooo!

---

Memory increase during those 13k block tracing:

[holiman]

Yup, now this works.

The drop there is when it finds a disk trie:

INFO [11-02|17:40:46.488] Imported new chain segment               blocks=1  txs=481  mgas=29.983  elapsed=464.753ms          mgasps=64.514  number=13,538,640 hash=f11200..e24c65 dirty=689.33MiB
INFO [11-02|17:40:46.490] Unindexed transactions                   blocks=1  txs=97   tail=13,538,541 elapsed=1.890ms
INFO [11-02|17:41:06.013] Imported new chain segment               blocks=1  txs=498  mgas=29.541  elapsed=231.347ms          mgasps=127.693 number=13,538,641 hash=489220..d81a9f dirty=690.82MiB
INFO [11-02|17:41:06.017] Unindexed transactions                   blocks=1  txs=172  tail=13,538,542 elapsed=3.511ms
...
INFO [11-02|22:47:21.454] Tracing chain segment                    start=13,401,259 end=13,538,940 current=13,414,701 transactions=2,547,632 elapsed=3h59m24.619355258s
INFO [11-02|22:47:32.152] Tracing chain segment                    start=13,401,259 end=13,538,940 current=13,414,706 transactions=2,548,254 elapsed=3h59m35.316871285s
INFO [11-02|22:47:32.155] Found disk backend for state trie 
INFO [11-02|22:47:40.320] Tracing chain segment                    start=13,401,259 end=13,538,940 current=13,414,710 transactions=2,548,816 elapsed=3h59m43.485377022s
INFO [11-02|22:47:44.500] Switching to prefer-disk mode for tracing size=11.78MiB
INFO [11-02|22:47:49.864] Tracing chain segment                    start=13,401,259 end=13,538,940 current=13,414,717 transactions=2,550,574 elapsed=3h59m53.028526974s

After the mem drop, it continued fine

[holiman]

Rebased, lightly squashed, fixed up. PTAL

[holiman]

Has been chugging along the entire night (started a few minutes after midnight ), and switches over from time to time:

[s1na]

LGTM. Also tested on a node