v1.10.23 dev chain: "missing trie node" on debug_getModifiedAccountsByNumber

[mohoff]

System information

Geth version: 1.10.24-unstable (but it's 1.10.23)
OS & Version: Linux

Expected behaviour

Return null or list of addresses for any existing block range when calling debug_getModifiedAccountsByNumber

Actual behaviour

In a local merge test setup, the call works for empty blocks in [0,12]. Then on block 13, two deposit stake tx get successfully mined (using https://github.com/protolambda/merge-genesis-tools as part of the genesis allocs), followed by more txs in blocks 14+. The deposit tx are confirmed working and valid, the merge kicks in with a beacon node and validator. Also txs in blocks 14+ get mined successfully. Only when debug_getModifiedAccountsByNumber runs on a block range including or after block 13, things break.

Symptoms of debug_getModifiedAccountsByNumber:

curl -X POST --data '{"jsonrpc":"2.0","method":"debug_getModifiedAccountsByNumber","params":[1,12],"id":1}' localhost:8545 -H 'Content-Type: application/json'
{"jsonrpc":"2.0","id":1,"result":null}.   --> expected

 curl -X POST --data '{"jsonrpc":"2.0","method":"debug_getModifiedAccountsByNumber","params":[1,13],"id":1}' localhost:8545 -H 'Content-Type: application/json'
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"missing trie node 9c8844fac2b8b2bcb75bf3861312f13b00593e7727945948b5cc44be6962ec06 (path ) \u003cnil\u003e"}}

curl -X POST --data '{"jsonrpc":"2.0","method":"debug_getModifiedAccountsByNumber","params":[1,100],"id":1}' localhost:8545 -H 'Content-Type: application/json'
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"no preimage found for hash 1143df8268b94bd6292fdd7c9b8af39a79f764cfc03ae006844446bc91203927"}}

Geth logs around block 13:

INFO [08-30|16:01:32.001] Commit new sealing work                  number=12 sealhash=f74805..ab47c7 uncles=0 txs=0 gas=0 fees=0 elapsed="398.641µs"
INFO [08-30|16:01:32.229] Setting new local account                address=0x8104A480F40cD0BF439e50711D8d7704108f41d9
INFO [08-30|16:01:32.230] Submitted transaction                    hash=0xee11294f6fdd179c2259afdd55c440ed00b5a995b3182a08a50a61117a40e880 from=0x8104A480F40cD0BF439e50711D8d7704108f41d9 nonce=0 recipient=0x4242424242424242424242424242424242424242 value=32,000,000,000,000,000,000
INFO [08-30|16:01:32.255] Submitted transaction                    hash=0x39c14ea137ef5e87615f7b844062b6f851dd8401d0a976a30d1d0e5a4b3f8cc8 from=0x8104A480F40cD0BF439e50711D8d7704108f41d9 nonce=1 recipient=0x4242424242424242424242424242424242424242 value=32,000,000,000,000,000,000
INFO [08-30|16:01:34.002] Successfully sealed new block            number=12 sealhash=f74805..ab47c7 hash=b6afce..a4824a elapsed=2.001s
INFO [08-30|16:01:34.002] 🔗 block reached canonical chain          number=5  hash=6bd6b6..b72031
INFO [08-30|16:01:34.002] 🔨 mined potential block                  number=12 hash=b6afce..a4824a
INFO [08-30|16:01:34.003] Commit new sealing work                  number=13 sealhash=73f8a2..60ae14 uncles=0 txs=0 gas=0 fees=0 elapsed="338.612µs"
INFO [08-30|16:01:34.004] Commit new sealing work                  number=13 sealhash=4bafed..a790f7 uncles=0 txs=2 gas=155,556 fees=0.00038889 elapsed=1.542ms
INFO [08-30|16:01:36.001] Successfully sealed new block            number=13 sealhash=4bafed..a790f7 hash=f3009b..9d205a elapsed=1.996s
INFO [08-30|16:01:36.001] 🔗 block reached canonical chain          number=6  hash=6f061e..ffd41e

// snip

WARN [08-30|16:34:11.338] Served debug_getModifiedAccountsByNumber conn=172.26.0.6:51492 reqid=1661877251336 duration="149.72µs"  err="no preimage found for hash 1143df8268b94bd6292fdd7c9b8af39a79f764cfc03ae006844446bc91203927"

Sounds similar to #25613

[holiman]

The v1.10.23 and thus .24-unstable contained a flaw which could make trie nodes disapppear. Could you please try to repro this on latest master?

Also, how many blocks have you imported? If you have imported . >128 blocks, then the in-memory pruning comes into play. We need more information about the state of the blockchain when you made the various calls, to know what is happening.

[mohoff]

@holiman The issue persists on latest master fb75f11 and latest release v1.10.25

[holiman]

Please provide logs. The context is important here, because geth is doing in-mempry garbage collection of old stateroots. So if you do getModifiedAccountsByNumber on a number which is in the range "eligible for garbage collection", then the error is expected and correct. The logs should show the latest imported block and the number that's being used in the call.

As for no preimage found for hash, you should use --cache.preimages=true. It might not solve all cases, but should help.

[mohoff]

@holiman My bad, --cache.preimages=true did the trick. The issue from my last post was due to PoS defaults introduced in v1.10.25 (and thus latest master).

Is --cache.preimages=true recommended for public chains too? I found a oneliner description in https://geth.ethereum.org/docs/interface/command-line-options but feels like I need to understand geth internals better to make sense of it. Quick explanation and consequences would be appreciated. Thanks!

[MariusVanDerWijden]

I opened an issue to add more documentation, will close this for now