Release `2551` distribution

[jpraynaud]

Why

In order to deploy new features, improvements and bug fixes, we need to release a new distribution.

Note

The previous distribution has been created with #2727.

What

Create and release a new distribution 2551

Important

• Rotate ancillary signer service account keys inrelease-mainnet
• Upgrade to Cardano node 10.6.1:?
  • pre-release-preview
  • release-preprod
  • release-mainnet
• 🔥 Warmup Cardano nodes for 10.6.1

Distribution +1:

Note

• TBD

How

Phase 1: Prepare distribution

• Refresh NPM token for publication
• Run manual successful restoration of Cardano database + restart of the Cardano node:
  • Cardano DB v1
    • with ancillary
    • without ancillary
  • Cardano DB v2
    • with ancillary
    • without ancillary
• Run a backward compatibility test in the CI on the unstable distribution (with the latest built e2e test)
• Run an all green aggregator stress test in the CI
• Run an all green client multi-platform test in the CI
• Prepare a PR to upgrade repository dependencies (with runbook)
• Reset the CI caches just before merging the PR to upgrade repository dependencies
• Merge the PR to upgrade repository dependencies
• Check that all minor versions of all crates have been bumped for breaking change (if needed)
  • Bump mithril-common to 0.6
  • Bump mithril-stm to 0.6
• Check if versions of repository linked crates are correct for publication to crates.io(mithril-common, mithril-client)
• Check if Cardano minimum supported versions in networks.json must be updated following the runbook
• Create a draft PR for rotating the documentation
• Create a PR for rotating the CHANGELOG.md (set the distribution version)
• Merge the PR for rotating the CHANGELOG.md
• Warmup Cardano node 10.6.1 on pre-release-preview
• Warmup Cardano node 10.6.1 on release-preprod
• Warmup Cardano node 10.6.1 on release-mainnet

Phase 2: Rollout pre-release distribution

• Update the configuration of pre-release-preview:
  • Upgrade the Cardano node version to 10.6.1 in pre-release-preview
• Tag and deploy a 2551.0-pre pre-release distribution on the pre-release-preview network
• Fill highlights, distributions compatibility and release notes sections in the pre-release page on GitHub
• Switch Cardano database with warmed-up node in pre-release-preview
• Make sure pre-release-preview is working as expected
• Make sure Grafana dashboards are fetching data for pre-release-preview
• Make sure libraries version next have been published on npm registry
• Run an all green client multi-platform test in the CI
• Run a Cardano transaction prover benchmark:
  • Store results in the issue
  • Make sure that there is no performance drop since last release
• Announce the pre-release on discord channel
• Announce any breaking change in:
  • #cardano-scaling slack channel for pre-release
  • #mithril-lace slack channel for pre-release
  • #blockfrost-scaling slack channel for pre-release
  • #mithril-catalyst slack channel for pre-release
  • #mithril-acropolis slack channel for pre-release
  • Cardano Foundation IBC bridge team
• Wait for at least 2 signers running the new version (could be more if era change is tested) over at least 2 epochs

Phase 3: Rollout release distribution

• Tag and deploy a 2551.0 release distribution on the pre-release-preview network
• Fill highlights, distributions compatibility and release notes sections in the release page on GitHub
• Make sure pre-release-preview is working as expected
• Update networks compatibility table in release notes
• Create a new key for the ancillary signer service account of release-mainnet
• Configure release-mainnet environment with new service account credentials
• Update the configuration of release-preprod:
  • Upgrade the Cardano node version to 10.6.1 in release-preprod
• Update the configuration of release-mainnet:
  • Upgrade the Cardano node version to 10.6.1 in release-mainnet
• Release the 2551.0
• Switch Cardano database with warmed-up node in release-mainnet
• Make sure release-mainnet is working as expected
• Switch Cardano database with warmed-up node in release-preprod
• Make sure release-preprod is working as expected
• Make sure libraries have been published on crates.io
• Make sure libraries version latest have been published on npm registry
• Make sure Grafana dashboards are fetching data for release-mainnet
• Make sure Grafana dashboards are fetching data for release-preprod
• Destroy the previous key for the ancillary signer service account of release-mainnet
• Check that KMS credentials are working with the creation of the next Cardano database snapshot
• Merge the PR for rotating the documentation
• Merge a PR for rotating the CHANGELOG.md:
  • Updating the release date of the distribution
  • Updating the crate versions table
• Announce the release on discord channel (with breaking-change, upcoming features activation, ...)
• Announce any breaking/new feature change in:
  • #cardano-scaling slack channel for release
  • #mithril-lace slack channel for release
  • #blockfrost-scaling slack channel for release
  • #mithril-catalyst slack channel for release
  • #mithril-acropolis slack channel for release
  • Cardano Foundation IBC bridge team
• Update Hydra tutorial if client breaking change for database restoration
• Dev blog post:
  • Distribution 2551
  • Update the Switching to Cardano database incremental certification dev blog post

[jpraynaud]

Cardano node warmup

Select the Mithril network

pre-release-preview

export NETWORK=preview
export AGGREGATOR_ENDPOINT=https://aggregator.pre-release-preview.api.mithril.network/aggregator
export GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/pre-release-preview/genesis.vkey)

release-preprod

export NETWORK=preprod
export AGGREGATOR_ENDPOINT=https://aggregator.release-preprod.api.mithril.network/aggregator
export GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/release-preprod/genesis.vkey)

release-mainnet

export NETWORK=mainnet
export AGGREGATOR_ENDPOINT=https://aggregator.release-mainnet.api.mithril.network/aggregator
export GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/release-mainnet/genesis.vkey)

Phase 0: Verify enough disk space is available in data disk

At least 50% of available space on data disk

df -h

Phase 1: Warmup a temporary Cardano node from a Mithril snapshota

Create temp dir

rm -rf /home/curry/data/temp
mkdir -p /home/curry/data/temp
cd /home/curry/data/temp

Download Mithril client

curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/input-output-hk/mithril/refs/heads/main/mithril-install.sh | sh -s -- -c mithril-client -d latest -p $(pwd)

Download latest snapshot

./mithril-client cardano-db download latest

Restore the Cardano node

docker run -d --name cardano-node-warmup -v cardano-node-ipc:/ipc -v cardano-node-data:/data --mount type=bind,source="/home/curry/data/temp/db",target=/data/db/ -e NETWORK=${NETWORK} ghcr.io/intersectmbo/cardano-node:10.6.1
docker logs -f --tail 100 cardano-node-warmup

Phase 2: Switch Cardano database from the main node with the warmed-up one

Copy Cardano database to aggregator

docker stop cardano-node-warmup
docker stop cardano-node-aggregator
mv /home/curry/data/$NETWORK/mithril-aggregator/cardano/db /home/curry/data/$NETWORK/mithril-aggregator/cardano/db.bak
mv /home/curry/data/temp/db /home/curry/data/$NETWORK/mithril-aggregator/cardano/db
sudo chown -R curry /home/curry/data/$NETWORK/mithril-aggregator/cardano/db
sudo chgrp -R curry /home/curry/data/$NETWORK/mithril-aggregator/cardano/db
docker start cardano-node-aggregator
docker logs -f --tail 100 cardano-node-aggregator

Cleanup directory (when node is back up)

docker rm cardano-node-warmup
rm -rf /home/curry/data/$NETWORK/mithril-aggregator/cardano/db.bak
cd /home/curry/
rm -rf /home/curry/data/temp