Stacks 1.0 -> 2.0 automated balance & lockup validation

[zone117x]

Closes #2128

This is a dockerfile + js script that validates migrated STX balances and lockup schedules between Stacks 1.0 and Stacks 2.0.

The following steps are performed:

1. Checkout and install Stacks 1.0.
2. Run a Stacks 1.0 fast-sync to get caught up to the latest chain state (as of the latest hosted snapshot).
3. Trigger a fast-sync-dump similar to how it will be triggered from the name threshold.
4. Perform the chainstate export step from the fast-sync-dump.
5. Checkout the Stacks 2.0 source, and copy over the newly exported chainstate.txt, and build.
6. Query the Stacks 1.0 db for 1000 address balances, and ~1000 lockup schedules.
7. Spin up both a Stacks 1.0 and Stacks 2.0 node, and validate the address balances match using the account RPC endpoints:
   • Stacks 1.0: /v1/accounts/{address}/STACKS/balance
   • Stacks 2.0: /v2/accounts/{address-in-testnet-format}
8. Validate lockup schedules in Stacks 2.0 match the samples dumped from the Stacks 1.0, using a contract map lookup:
   • /v2/map_entry/ST000000000000000000002AMW42H/lockup/lockups

TODO

• Add a "extra addresses" file that can be manually populated, for example the mapped placeholder addresses should be added to the tests.

Notes

• Addresses are converted into testnet format for querying the Stacks 2.0 node. The test would be more robust if Stacks 2.0 had a "mainnet mocknet" mode.

Running

This audit can be ran locally by just running the included script: ./migration-verfication/run.sh.
Or run the docker build:

cd migration-verifcation
DOCKER_BUILDKIT=1 docker build .

Docker needs around 60GB of disk space allocated. It can take a long time to finish, upwards of an hour -- every step is resource intensive: compiling, exporting, importing, and validating.

@CharlieC3 @wileyj would it be useful to have a custom gh action trigger to run this step? For example we could trigger the action when the "final" automated PR is opened.

[CharlieC3]

Docker needs around 60GB of disk space allocated. It can take a long time to finish, upwards of an hour -- every step is resource intensive: compiling, exporting, importing, and validating.

would it be useful to have a custom gh action trigger to run this step? For example we could trigger the action when the "final" automated PR is opened.

If it weren't for the 60GB of space required I would say we can try it on a default GH runner, but they only have ~14 GB of space available on them.

If we really want to use Github Actions for this, we would have more luck creating our own GH runners with enough space on them. But at that point I'm not really sure the extra automation and time required to automate and test this is worth it.

It might be better to just modify our local docker configs to give containers ~100GB of space + ample CPU/memory and run this script locally, then post our results as comments in the PR. Plus, this is only something we'll have to do once, so additional automation on top of what's already automated might not be needed here.

Thoughts?

Only a couple changes noted, but nothing critical! Thank you for this!

[CharlieC3]

Not sure if everyone has realpath installed by default. You should be able to get the same results with:

Suggested change

	cd "$(dirname "$(realpath "$0")")"
	cd "$(dirname "${BASH_SOURCE[0]}")"

[zone117x]

Good catch, fixed!

[zone117x]

Replaced this script with a more useful README file now that a docker ARG is required.

[CharlieC3]

Will have to remember to update this branch to the latest available come migration day. Any chance we can add an ARG which would require someone to pass in the V2 branch they intend to test with?

[zone117x]

Done! Added usage instructions to a readme file

[zone117x]

Note: this PR now includes validating token lockup schedules, which also appear to be working as expected.

[CharlieC3]

[kantai]

LGTM!

Changes that would be nice to have is a way to parameterize the number of addresses, and rather than ordering by amount, order by RAND()

[blockstack-devops]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.