Add command to diff states

[turbolent]

Add a new command, diff-states, to the util, which allows diffing two states. The states may be either given in the form of a snapshot directory / state commitment, or a payloads file. The command generates a report of differences in the states on either a low "raw" level, or on a high Cadence-level.

This essentially exposes the existing state loading and Cadence value diffing functionality of the migration.
While at it, optimize how the traces (paths in the object structure) are generated.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 51.60681% with 256 lines in your changes are missing coverage. Please review.

Project coverage is 55.80%. Comparing base (7ebb82b) to head (095669b).

Files	Patch %	Lines
cmd/util/cmd/diff-states/cmd.go	58.99%	114 Missing and 16 partials ⚠️
cmd/util/ledger/migrations/cadence_value_diff.go	37.93%	101 Missing and 7 partials ⚠️
...util/ledger/migrations/cadence_values_migration.go	0.00%	10 Missing ⚠️
cmd/util/cmd/extract-payloads-by-address/cmd.go	0.00%	3 Missing ⚠️
cmd/util/ledger/util/registers/registers.go	0.00%	3 Missing ⚠️
ledger/complete/ledger.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5970      +/-   ##
==========================================
- Coverage   55.83%   55.80%   -0.04%     
==========================================
  Files        1129     1131       +2     
  Lines       89297    89758     +461     
==========================================
+ Hits        49858    50085     +227     
- Misses      34686    34895     +209     
- Partials     4753     4778      +25     

Flag	Coverage Δ
unittests	55.80% <51.60%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[turbolent]

@fxamacker Any idea why I'd get errors like failed to preload old atree registers: decoding error: data has invalid flag 0x80, want 0x8 or 0xb when running the Cadence value diffing?

Getting this when running the command on cadence-migration-test (flow-benchmark, us-west1-b), with mount -t ext4 -o discard,defaults /dev/disk/by-id/google-cadence-migration-test-temp /var/temp:

$ util diff-states \
    --state-1 /var/temp/migrationtestnet1-may8-inlined-preview20 \
    --state-2 /var/temp/migrationtestnet1-may8-inlined-preview23 \
    --state-commitment-1 5f0ebd52ea07b8a39b5f76d1e981ec92702663dc4ec82a2f7beafba6263d5a8c \
    --state-commitment-2 fd8f5c9d6ef2b13acd0c806284901fad2f510e812d8b8c391913b03143c2cd56 \
    --output-directory /var/temp \
    --raw=false \
    --chain flow-testnet

It's only reported for some of the accounts:

$ cat state-diff_1716412624.json | jq -c '.[] | select(.Kind == "error_diff_failed")' | wc -l
25146

[fxamacker]

@fxamacker Any idea why I'd get errors like failed to preload old atree registers: decoding error: data has invalid flag 0x80, want 0x8 or 0xb when running the Cadence value diffing?

@turbolent The error message "data has invalid flag" is only present in feature/stable-cadence branch (no atree inlining). Is the input data to the program atree inlined or not inlined?

[turbolent]

@fxamacker Oh, right, good catch! I need to build the binary on top of the atree inlining branch 👍

[fxamacker]

Nice! I focused on reviewing non-test code and left some comments.

[fxamacker]

Nice! Thanks for updating PR! I left a few more comments for your consideration.

[fxamacker]

Maybe use AddressToRegisterOwner to be consistent for global registers.

Suggested change

	owner := string(address[:])
	owner := AddressToRegisterOwner(address)

[turbolent]

I changed it in 9a2541a, but is it even required? We call this in MigrateAccount, which is only called for accounts, and not for global registers?

[turbolent]

@onflow/cadence PTAL

[turbolent]

To address the last part of #3366, an atree failure to iterate over a non-migrated key, I added some granular error recovery code to the Cadence value diffing in 095669b. There isn't much we can do about the problem, other than handling it gracefully when we encounter it and continue

[AlexHentschel]

Took a brief pass over the PR. There aren't really any algorithmic changes on the trie, so no concerns from my end. Though, don't have a lot of context on the other details.