UIP-6: App Version Safeguard

[cronokirby]

Describe your changes

This implements UIP 6, creating an "app version safeguard", to try and prevent the wrong version of PD from being started against existing state, or running the wrong version of PD.

This code should be immediately implementable as a non-breaking point-release, which should immediately provide a safeguard against forgetting to upgrade to the next major version of the software before the next migration.
This should happen because nodes running the point release will start writing, to non-consensus state, the app version they have. The current migration in 0.80 will refuse to run unless this app version key is empty, or exactly the previous version, preventing forgetting to upgrade to the next version pre migration. Furthermore, the next migration should do the same, but with the next app version, so that it will not allow skipping the previous migration.

For testing, I think we should:

• test that we can take an existing node, and then start it with the new code, stop it, and then start it again
• test that trying to run pd migrate fails because of the newly added safeguard.

Issue ticket number and link

Closes #4793.

Implements penumbra-zone/UIPs#10.

Checklist before requesting a review

• I have added guiding text to explain how a reviewer should test these changes.

• If this code contains consensus-breaking changes, I have added the "consensus-breaking" label. Otherwise, I declare my belief that there are not consensus-breaking changes, for the following reason:

  This code is sensitive to potentially breaking consensus, but the testing plan above should confirm that it is safe for a point release. If it is not safe for a point release, the code should be amended so that it is.

[conorsch]

The smoke tests are failing, and I can replicate the failure locally: pd is falling down immediately with

thread 'tokio-runtime-worker' panicked at crates/core/app/src/server/consensus.rs:71:26:
init_chain must succeed: database already initialized

which is good ol' #1884. It looks like writing to the storage on a fresh node rightly triggers initialization, which causes pd to error out.

[cronokirby]

Is there any way to write to non verifiable storage without triggering this?

[conorsch]

To review, I prepared a local devnet based on current main (without the safeguard logic), ran it for a while, then switched the pd binary to one built from this branch and bounced the service. Then I submitted an upgrade proposal and confirmed it halted the chain.

When I attempted pd migrate via the pre-upgrade version, I encountered an error:

Error: failed to upgrade state

Caused by:
    app version mismatch:
      expected 7 (penumbra v0.79.x)
      found 8 (penumbra v0.79.x)
    this migration should be run with penumbra v0.80.x instead

which shows an off-by-one in the "found" version; simple fixed already pushed. Will continue with local testing and follow up with more feedback.

[conorsch]

Tested locally by drafting a no-op migration, bumping the APP_VERSION, and confirming behavior. Tacked on a few changes fussing with the err msgs reported, so that the tool is maximally informative to node operators. Very pleased with the behavior, thanks for all the careful thought put into this, @cronokirby!