[testnet] genesis sync issue when overrides are not applied before reward cycle 413

[wileyj]

Running a genesis sync using the latest tagged release (2.4.0.0.4) results in a stalled sync at block 106978 .
adding the overrides from the forum (https://forum.stacks.org/t/stacks-2-4-and-testnet-reorg/15027) enables the sync to proceed past this reward cycle, but it then stalls again at block 107574, reward cycle 416.

Since this is a burden that new node operators may not be aware of when genesis syncing testnet, i propose that these overrides are hardcoded so the binary applies them automatically when genesis syncing testnet.

[wileyj]

config used to reach the stall:

[node]
working_dir = "/stacks-blockchain/data"
rpc_bind = "0.0.0.0:20443"
p2p_bind = "0.0.0.0:20444"
bootstrap_node = "029266faff4c8e0ca4f934f34996a96af481df94a89b0c9bd515f3536a95682ddc@seed.testnet.hiro.so:30444"
wait_time_for_microblocks = 10000

[burnchain]
chain = "bitcoin"
mode = "xenon"
peer_host = "bitcoind.testnet.stacks.co"
username = "blockstack"
password = "blockstacksystem"
rpc_port = 18332
peer_port = 18333

[[ustx_balance]]
address = "ST2QKZ4FKHAH1NQKYKYAYZPY440FEPK7GZ1R5HBP2"
amount = 10000000000000000

[[ustx_balance]]
address = "ST319CF5WV77KYR1H3GT0GZ7B8Q4AQPY42ETP1VPF"
amount = 10000000000000000

[[ustx_balance]]
address = "ST221Z6TDTC5E0BYR2V624Q2ST6R0Q71T78WTAX6H"
amount = 10000000000000000

[[ustx_balance]]
address = "ST2TFVBMRPS5SSNP98DQKQ5JNB2B6NZM91C4K3P7B"
amount = 10000000000000000

and some logs when the loop starts:

INFO [1709663780.672345] [src/chainstate/burn/db/sortdb.rs:4817] [chains-coordinator-0.0.0.0:20443] ACCEPTED(2441337) leader block commit 061adbf43c8f2ace74188eba291b34eb5bf18e4c68976e897dda63e362ce0386 at 2441337,7, apparent_sender: mxVFsFW5N4mu1HPkxPttorvocvzeZ7KZyk
INFO [1709663780.922500] [src/chainstate/coordinator/mod.rs:1190] [chains-coordinator-0.0.0.0:20443] Sortition anchor block affirmation map `nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnpppppnnnnnnnnnnnnnnnnnnnnnnnpppppppppppppppnnnnnnnnnnnnnnnnnnnnnnnppppppppppnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnppppppppnnnnnnnnnnnnnnnnnnnnnnnppnppnnnnnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnnnppppppnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnnpppppppnnnnnnnnnnnnnnnnnnnnnnnnnnpnnnnnnnnnnnnnnnnnnnnnnnnnpppnppppppppppppppnnppppnpaaaapaaaa` and/or Stacks affirmation map `nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnpppppnnnnnnnnnnnnnnnnnnnnnnnpppppppppppppppnnnnnnnnnnnnnnnnnnnnnnnppppppppppnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnppppppppnnnnnnnnnnnnnnnnnnnnnnnppnppnnnnnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnnnppppppnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnnpppppppnnnnnnnnnnnnnnnnnnnnnnnnnnpnnnnnnnnnnnnnnnnnnnnnnnnnpppnppppppppppppppnnppppnpaaaa` is no longer compatible with heaviest affirmation map nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnpppppnnnnnnnnnnnnnnnnnnnnnnnpppppppppppppppnnnnnnnnnnnnnnnnnnnnnnnppppppppppnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnppppppppnnnnnnnnnnnnnnnnnnnnnnnppnppnnnnnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnnnppppppnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnnpppppppnnnnnnnnnnnnnnnnnnnnnnnnnnpnnnnnnnnnnnnnnnnnnnnnnnnnpppnppppppppppppppnnppppnpaaaapppppp in reward cycles 416-423
INFO [1709663780.925575] [src/chainstate/coordinator/mod.rs:1361] [chains-coordinator-0.0.0.0:20443] Re-playing sortitions starting within reward cycle 417 burn height 2437851
INFO [1709663783.235788] [src/chainstate/coordinator/mod.rs:605] [chains-coordinator-0.0.0.0:20443] Anchor block selected for cycle 417: 970a63d4bd9038a5ae136cefca35ccde58950329/a5e655ff2de5bb7ec98c224ce8523d55554d4ecbe949b8fcadb5c94e3ddf9af5 (txid 726d75ad2d35befec81cacb68fc027b37e83fa74420f8d71a887788dce3374ef)
INFO [1709663783.235908] [src/chainstate/coordinator/mod.rs:1981] [chains-coordinator-0.0.0.0:20443] Anchor block a5e655ff2de5bb7ec98c224ce8523d55554d4ecbe949b8fcadb5c94e3ddf9af5 (txid 726d75ad2d35befec81cacb68fc027b37e83fa74420f8d71a887788dce3374ef) for reward cycle 416 is affirmed by the network (nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnpppppnnnnnnnnnnnnnnnnnnnnnnnpppppppppppppppnnnnnnnnnnnnnnnnnnnnnnnppppppppppnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnppppppppnnnnnnnnnnnnnnnnnnnnnnnppnppnnnnnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnnnppppppnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnnpppppppnnnnnnnnnnnnnnnnnnnnnnnnnnpnnnnnnnnnnnnnnnnnnnnnnnnnpppnppppppppppppppnnppppnpaaaappppppan), but must be downloaded
INFO [1709663783.235950] [src/chainstate/coordinator/mod.rs:2116] [chains-coordinator-0.0.0.0:20443] Burnchain block processing will continue in spite of missing affirmed anchor block a5e655ff2de5bb7ec98c224ce8523d55554d4ecbe949b8fcadb5c94e3ddf9af5
INFO [1709663783.237276] [src/chainstate/burn/db/sortdb.rs:3630] [chains-coordinator-0.0.0.0:20443] Begin reward-cycle sortition with absent anchor block=Some((a5e655ff2de5bb7ec98c224ce8523d55554d4ecbe949b8fcadb5c94e3ddf9af5, 726d75ad2d35befec81cacb68fc027b37e83fa74420f8d71a887788dce3374ef))
INFO [1709663783.237434] [src/chainstate/coordinator/mod.rs:2051] [chains-coordinator-0.0.0.0:20443] Revalidate already-processed snapshot 33ebf1c048cabef6ae1f26d4054de9ee5e1eb242ab09b568f9f78aa951efe596 height 2437851 to have canonical tip eb8f940f935a07f4c63cebd287add8efbf8bb198/632d7f94d927b0641410a5a4d18c193039f7d62a8a7d348cfc5380b1e489d731 height 107574

INFO [1709664573.694636] [src/chainstate/coordinator/mod.rs:1190] [chains-coordinator-0.0.0.0:20443] Sortition anchor block affirmation map `nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnpppppnnnnnnnnnnnnnnnnnnnnnnnpppppppppppppppnnnnnnnnnnnnnnnnnnnnnnnppppppppppnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnppppppppnnnnnnnnnnnnnnnnnnnnnnnppnppnnnnnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnnnppppppnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnnpppppppnnnnnnnnnnnnnnnnnnnnnnnnnnpnnnnnnnnnnnnnnnnnnnnnnnnnpppnppppppppppppppnnppppnpaaaapaaaaaan` and/or Stacks affirmation map `nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnpppppnnnnnnnnnnnnnnnnnnnnnnnpppppppppppppppnnnnnnnnnnnnnnnnnnnnnnnppppppppppnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnppppppppnnnnnnnnnnnnnnnnnnnnnnnppnppnnnnnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnnnppppppnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnnpppppppnnnnnnnnnnnnnnnnnnnnnnnnnnpnnnnnnnnnnnnnnnnnnnnnnnnnpppnppppppppppppppnnppppnpaaaa` is no longer compatible with heaviest affirmation map nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnpppppnnnnnnnnnnnnnnnnnnnnnnnpppppppppppppppnnnnnnnnnnnnnnnnnnnnnnnppppppppppnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnppppppppnnnnnnnnnnnnnnnnnnnnnnnppnppnnnnnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnnnppppppnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnnpppppppnnnnnnnnnnnnnnnnnnnnnnnnnnpnnnnnnnnnnnnnnnnnnnnnnnnnpppnppppppppppppppnnppppnpaaaapppppp in reward cycles 416-428

[wileyj]

reproducing is simple, i.e.:

docker run -d \
    --name stacks \
    -p 30443:20443 \
    -p 30444:20443 \
    -v /stacks-blockchain/data:/stacks-blockchain/data \
    -v /stacks-blockchain/testnet.toml:/src/stacks-blockchain/Config.toml \
blockstack/stacks-blockchain:2.4.0.0.4 \
sh -c "/bin/stacks-node start --config /src/stacks-blockchain/Config.toml"

and let it run for roughly 20 hours without adding the overrides mentioned in the forum

[wileyj]

@8marz8 it should be simple-ish to add these as default for testnet/xenon:

./stacks-core/stackslib/src/burnchains/db.rs, function set_override_affirmation_map - https://github.com/stacks-network/stacks-core/blob/next/stackslib/src/burnchains/db.rs#L940

with the following overrides when booting a testnet instance:

413|nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnpppppnnnnnnnnnnnnnnnnnnnnnnnpppppppppppppppnnnnnnnnnnnnnnnnnnnnnnnppppppppppnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnppppppppnnnnnnnnnnnnnnnnnnnnnnnppnppnnnnnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnnnppppppnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnnpppppppnnnnnnnnnnnnnnnnnnnnnnnnnnpnnnnnnnnnnnnnnnnnnnnnnnnnpppnppppppppppppppnnppppnpa
414|nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnpppppnnnnnnnnnnnnnnnnnnnnnnnpppppppppppppppnnnnnnnnnnnnnnnnnnnnnnnppppppppppnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnppppppppnnnnnnnnnnnnnnnnnnnnnnnppnppnnnnnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnnnppppppnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnnpppppppnnnnnnnnnnnnnnnnnnnnnnnnnnpnnnnnnnnnnnnnnnnnnnnnnnnnpppnppppppppppppppnnppppnpaa
415|nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnpppppnnnnnnnnnnnnnnnnnnnnnnnpppppppppppppppnnnnnnnnnnnnnnnnnnnnnnnppppppppppnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnppppppppnnnnnnnnnnnnnnnnnnnnnnnppnppnnnnnnnnnnnnnnnnnnnnnnnppppnnnnnnnnnnnnnnnnnnnnnnnnnppppppnnnnnnnnnnnnnnnnnnnnnnnnnppnnnnnnnnnnnnnnnnnnnnnnnnnpppppppnnnnnnnnnnnnnnnnnnnnnnnnnnpnnnnnnnnnnnnnnnnnnnnnnnnnpppnppppppppppppppnnppppnpaaa

i would look at ./testnet/stacks-node/src/config.rs: https://github.com/stacks-network/stacks-core/blob/next/testnet/stacks-node/src/config.rs