Disconnect / Reconnect test

[gmaxwell]

As noted at (#2839 (comment))
there should exist test infrastructure that disconnects and reconnects the
whole chain (mainnet and testnet).

(Would have detected the OP_RETURN disconnection failure bug.)

[laanwj]

This would be easier now that there are invalidateblock and reconsiderblock RPCs, These are used in some of the RPC tests, but there are no tests that use the testnet or mainnet chain.

[glozow]

Maybe the regular reorgs on signet now cover this? @0xB10C?

[0xB10C]

Maybe the regular reorgs on signet now cover this? @0xB10C?

I don't think so. The idea was to keep the automatic reorgs smallish (somewhere between 1 to 12 blocks) for now. Also, I don't think SigNet is the right place to test this. I'm not sure what the OP_RETURN disconnection failure bug was, but I don't think there are enough transactions on SigNet to catch something like this.

However, I think a script that invalidateblock blocks one by one starting from the tip while recording hash(dumptxoutset) or gettxoutsetinfo with coinstatsindex enabled - and then, once close to genesis, reconsiderblock for each block and compares the hash of the UTXO set, should be doable. I'm not sure if that's what was meant in #2839 (comment) with a table with the proper utxo state for every height.

@0xB10C I think this is the issue referred as OP_RETURN disconnection failure bug: #3156

[0xB10C]

I've just found out that there is the verifychain RPC and the -checklevel and -checkblocks arguments which with level 4 does a block reconnect test. I'm running bitcoin-cli verifychain 4 0 to verify all blocks.

[0xB10C]

This RPC times out after 15min. Verification seems to continue.

$ time bitcoin-cli verifychain 4 0
error: timeout on transient error: Could not connect to the server 127.0.0.1:8332 (error code 0 - "timeout reached")

Make sure the bitcoind server is running and that you are connecting to the correct RPC port.

real    15m0.104s
user    0m0.001s
sys     0m0.002s

[willcl-ark]

@0xB10C I have just tested your approach (of running verifychain) on signet. It does complete, eventually, and records progress in debug.log

I also think this approach addresses the request in OP. However, if we want to test this against mainnet I don't really see a particularly viable approach in terms of CI, without something convoluted like having a host running a pruned node that runs verifychain 4 (tip_height-1) on each new block (to disconnect and reconnect), and re-compiling and restarting on each new commit (or tag).

Perhaps part of the maintainer release process could include running verifychain 4 0 before a release candidate is tagged?

[0xB10C]

I also think this approach addresses the request in OP. However, if we want to test this against mainnet I don't really see a particularly viable approach in terms of CI, without something convoluted like having a host running a pruned node that runs verifychain 4 (tip_height-1) on each new block (to disconnect and reconnect), and re-compiling and restarting on each new commit (or tag).

I think you would want to cover the whole chain and not only new blocks. If a bug happens to be introduced, it could break on something that's in the chain already, not only on new blocks. I agree that running this on mainnet in some sort of CI for each commit is probably not possible and overkill. Periodically on master, maybe once a week could be enough.

Also like the idea of having it run once before a release. However, I don't think it should be the job of a maintainer. This only adds more work and costs to a job that very few want to do. For me, it took 14h on a beefy machine with 48GB of dbcache. However, I think it could be part of the reoccurring release-candidate testing issue and other contributors could pick it up and comment that this was part of their testing.

fwiw: We'd still need to up the current max dbcache of 16GB to maybe 64GB, otherwise verifychain 4 0 will OOM. See #24851 (comment)

[willcl-ark]

fwiw: We'd still need to up the current max dbcache of 16GB to maybe 64GB, otherwise verifychain 4 0 will OOM. See #24851 (comment)

Interesting, and good to know. Thanks!

[maflcko]

I don't think it is interesting to run this on signet, given that there are no "interesting" transactions (only standard ones), see #26439 (comment). Maybe further discussion can be done in the https://github.com/bitcoin-core/qa-assets repo, which also hosts a heavy CI task to fuzz under valgrind. Though, that one times out, so maybe there can be a general discussion about heavy QA tasks?

Closing this one for now.