ci: add a backward compatibility checks to the end-to-end runner #2816

Alenar · 2025-11-26T12:47:59Z

Content

This PR enhances the end to end runner by adding two mechanisms:

Test scenarios can now differs based on the nodes version, allowing to keep compatibility with older nodes by removing/adding parameters (i.e. --backend to the cli), checks, or scenario steps.
The end to end runner will now exit immediately if it detects that nodes have incompatible versions either between each other or with the runner itself (with a dedicated exit code: 3)

Compatibility check details

Backward-compatibilty: Supported changes

mithril-client:
- check of the cardano-db snapshot list --epoch latest-5 is skip on versions that don't have the --epoch flag (below 0.12.34 )
- for version below 0.12.11 use cardano-db-v2 download instead of cardano-db download --backend v2
- for version below 0.11.14 remove not yet implemented --include-ancillary flag
- for version below 0.11.13 remove not yet implemented --origin-tag flag
mithril-aggregator:
- since 0.7.94 only the leader aggregator must be restarted when updating protocol parameters, if below this version all aggregators are restarted

Incompatible versions check details

mithril-client:
- versions below 0.11.14 (2517.0) are not compatible with aggregator 0.7.31 and up (same distribution): it introduces the split of ancillary files from the cardano db archive and there's was a bug that make the download verification fails because without ancillaries the incomplete immutable trio is missing.
mithril-aggregator:
- version below 0.7.91 (next distribution) are not compatible with signer 0.2.277 and up (next distribution): those signers needs the new /protocol-configuration/{epoch} route
- version below 0.7.55 (2524.0) are not compatible with cardano node 10.4.1 and up: this cardano node version switch the ledger to the UTxO-HD format which was not supported before
mithril-signer:
versions below 0.2.221 (2450.0) are not compatible with newer distributions: there was a bug that makes signers fails if a unknown signed entity type discriminant was propagated in the network (triggered by the addition the CardanoDatabase signed entity)

Other changes

backward compatibility CI workflow:
- don't fail but exit immediately with a success code if an incompatibility was detected and report the node as incompatible in the summary: this filter out known incompatibilities from the workflow results, allowing easier detection of unwanted breaking change
- fix: missing cardano-node version in summary
Nightly workflow: make failing backward compatibility workflow send the alert email if it fails, this was not doable previously as it was regularly failing because of known incompatibilities

Pre-submit checklist

Branch
- Tests are provided (if possible)
- Crates versions are updated (if relevant)
- CHANGELOG file is updated (if relevant)
- Commit sequence broadly makes sense
- Key commits have useful messages
PR
- All check jobs of the CI have succeeded
- Self-reviewed the diff
- Useful pull request description
- Reviewer requested
Documentation
- Update README file (if relevant)
- No new TODOs introduced

Comments

Issue(s)

Closes #2781, Closes #2760

So the version can be used later to enable/disable/adapt tests scenarios

This allow to check two kind of incompatibilities: - of the runner itself with some older nodes - of two nodes between each others

…ode version Validated directly by clap. This change is needed in order to use the cardano node version in the compatibility checker.

github-actions · 2025-11-26T12:56:20Z

Test Results

4 files ± 0 168 suites ±0 23m 3s ⏱️ -51s
2 222 tests + 8 2 222 ✅ + 8 0 💤 ±0 0 ❌ ±0
6 929 runs +24 6 929 ✅ +24 0 💤 ±0 0 ❌ ±0

Results for commit 1d65b16. ± Comparison against base commit 2943337.

♻️ This comment has been updated with latest results.

Alenar · 2025-11-26T15:31:40Z

mithril-test-lab/mithril-end-to-end/backward-compatibility.md

+List of breaking changes that are NOT supported by the end-to-end tests, running a node with those versions against
+`unstable` will result in an error and an exit code of `3`.


Suggested change

List of breaking changes that are NOT supported by the end-to-end tests, running a node with those versions against

`unstable` will result in an error and an exit code of `3`.

List of breaking changes that are NOT supported by the end-to-end tests, running the tests with nodes versions

violating at least one of the case below will result in an error and an exit code of `3`.

Alenar · 2025-11-26T15:51:49Z

mithril-test-lab/mithril-end-to-end/src/utils/compatibility_checker.rs

+            } => {
+                write!(
+                    f,
+                    "{other_node_name} version `{other_node_min_compatible_version}` is incompatible with {name} with a version below `{min_compatible_version}`{context}",


Suggested change

"{other_node_name} version `{other_node_min_compatible_version}` is incompatible with {name} with a version below `{min_compatible_version}`{context}",

"{other_node_name} starting version `{other_node_min_compatible_version}` is incompatible with {name} with a version below `{min_compatible_version}`{context}",

* mithril-end-to-end from `0.4.108` to `0.4.109`

jpraynaud

LGTM 🔥

jpraynaud · 2025-11-26T15:59:43Z

mithril-test-lab/mithril-end-to-end/backward-compatibility.md

+There are multiple ways to do this:
+
+- disabling part of the scenarios
+- adding or removing arguments to the binaries (i.e. `--backend` flag for the client-cli)


Suggested change

- adding or removing arguments to the binaries (i.e. `--backend` flag for the client-cli)

- adding or removing arguments to the binaries (e.g. `--backend` flag for the client-cli)

jpraynaud · 2025-11-26T16:00:30Z

mithril-test-lab/mithril-end-to-end/backward-compatibility.md

+> Node versions and distribution versions below are the ones that first included the changes, has node versions evolve
+> faster than distribution versions the actual node version included in the distribution may be higher.


Suggested change

> Node versions and distribution versions below are the ones that first included the changes, has node versions evolve

> faster than distribution versions the actual node version included in the distribution may be higher.

> Node versions and distribution versions below are the ones that first included the changes, as node versions evolve

> faster than distribution versions the actual node version included in the distribution may be higher.

jpraynaud · 2025-11-26T16:00:50Z

mithril-test-lab/mithril-end-to-end/backward-compatibility.md

+> Node versions and distribution versions below are the ones that first included the changes, has node versions evolve
+> faster than distribution versions the actual node version included in the distribution may be higher.
+>
+> i.e. `--origin-tag` flag for the client-cli was introduced in version `0.11.13`, the first distribution that included


Suggested change

> i.e. `--origin-tag` flag for the client-cli was introduced in version `0.11.13`, the first distribution that included

> e.g. `--origin-tag` flag for the client-cli was introduced in version `0.11.13`, the first distribution that included

jpraynaud · 2025-11-26T16:01:41Z

mithril-test-lab/mithril-end-to-end/backward-compatibility.md

+
+---
+
+## Not supported changes


Suggested change

## Not supported changes

## Unsupported changes

jpraynaud · 2025-11-26T16:04:06Z

mithril-test-lab/mithril-end-to-end/backward-compatibility.md

+
+### Mithril client
+
+- since `0.12.34` (2543.0): test of new `--epoch` filter to `cardano-db list` (disabled on lower versions)


This will make the lists a bit more readable:

Suggested change

- since `0.12.34` (2543.0): test of new `--epoch` filter to `cardano-db list` (disabled on lower versions)

- **since `0.12.34` (2543.0)**: test of new `--epoch` filter to `cardano-db list` (disabled on lower versions)

Alenar added 12 commits November 26, 2025 13:47

doc(e2e): document e2e back-compatibility in a markdown

990c59c

feat(e2e): parse mithril command version

e0faaf3

So the version can be used later to enable/disable/adapt tests scenarios

feat(e2e): add backcompat for identified cases we want to support

d418db9

fix(e2e): always enable --unstable flag when running client-cli

db0deb0

feat(e2e): add compatibility checker to define and check nodes compats

0fe5efe

This allow to check two kind of incompatibilities: - of the runner itself with some older nodes - of two nodes between each others

chore(e2e): define the incompat rules in the checker

44a8c38

ci(backcompat): fix missing cardano node version in summary

a583cf0

refactor(e2e): use a semver version instead of a string for cardano n…

5ef75f3

…ode version Validated directly by clap. This change is needed in order to use the cardano node version in the compatibility checker.

feat(e2e): exist with a specific code (3) when a compat error is raised

cd0818a

feat(e2e): add compat check when starting the runner

9e325a2

ci(backcompat): success with warning when incompatible nodes are used

e9b5f16

feat(e2e): failfast by checking compat before running devnet

e741890

Alenar requested a review from jpraynaud November 26, 2025 12:47

Alenar self-assigned this Nov 26, 2025

Alenar added CI/CD 🏭 CI/CD testing 🔁 Something related to tests labels Nov 26, 2025

ci(nightly): include backcompat in failure notifications

0b46ba4

Alenar commented Nov 26, 2025

View reviewed changes

Alenar added 2 commits November 26, 2025 16:55

chore(nix): add backtraces to test results

bf3f64b

chore: upgrade crate versions

1d65b16

* mithril-end-to-end from `0.4.108` to `0.4.109`

Alenar force-pushed the djo/2781/e2e-backcompat branch from 5a7b8b8 to 1d65b16 Compare November 26, 2025 15:55

Alenar temporarily deployed to testing-preview November 26, 2025 16:06 — with GitHub Actions Inactive

jpraynaud approved these changes Nov 26, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: add a backward compatibility checks to the end-to-end runner #2816

ci: add a backward compatibility checks to the end-to-end runner #2816

Uh oh!

Alenar commented Nov 26, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 26, 2025 •

edited

Loading

Uh oh!

Alenar Nov 26, 2025

Uh oh!

Alenar Nov 26, 2025

Uh oh!

jpraynaud left a comment

Uh oh!

jpraynaud Nov 26, 2025

Uh oh!

jpraynaud Nov 26, 2025

Uh oh!

jpraynaud Nov 26, 2025

Uh oh!

jpraynaud Nov 26, 2025

Uh oh!

jpraynaud Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		List of breaking changes that are NOT supported by the end-to-end tests, running a node with those versions against
		`unstable` will result in an error and an exit code of `3`.

	"{other_node_name} version `{other_node_min_compatible_version}` is incompatible with {name} with a version below `{min_compatible_version}`{context}",
	"{other_node_name} starting version `{other_node_min_compatible_version}` is incompatible with {name} with a version below `{min_compatible_version}`{context}",

	- adding or removing arguments to the binaries (i.e. `--backend` flag for the client-cli)
	- adding or removing arguments to the binaries (e.g. `--backend` flag for the client-cli)

		> Node versions and distribution versions below are the ones that first included the changes, has node versions evolve
		> faster than distribution versions the actual node version included in the distribution may be higher.

	> i.e. `--origin-tag` flag for the client-cli was introduced in version `0.11.13`, the first distribution that included
	> e.g. `--origin-tag` flag for the client-cli was introduced in version `0.11.13`, the first distribution that included


		### Mithril client

		- since `0.12.34` (2543.0): test of new `--epoch` filter to `cardano-db list` (disabled on lower versions)

	- since `0.12.34` (2543.0): test of new `--epoch` filter to `cardano-db list` (disabled on lower versions)
	- since `0.12.34` (2543.0): test of new `--epoch` filter to `cardano-db list` (disabled on lower versions)

ci: add a backward compatibility checks to the end-to-end runner #2816

Are you sure you want to change the base?

ci: add a backward compatibility checks to the end-to-end runner #2816

Uh oh!

Conversation

Alenar commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Content

Compatibility check details

Backward-compatibilty: Supported changes

Incompatible versions check details

Other changes

Pre-submit checklist

Comments

Issue(s)

Uh oh!

github-actions bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Alenar Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Alenar Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

jpraynaud left a comment

Choose a reason for hiding this comment

Uh oh!

jpraynaud Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

jpraynaud Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

jpraynaud Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

jpraynaud Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

jpraynaud Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Alenar commented Nov 26, 2025 •

edited

Loading

github-actions bot commented Nov 26, 2025 •

edited

Loading