Fix end to end test flakiness in CI #1486
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jpraynaud
force-pushed
the
jpraynaud/1147-fix-e2e-flakiness
branch
12 times, most recently
from
22a54fd
to
474c8fa
Compare
jpraynaud
temporarily deployed
to
testing-preview
GitHub Actions
Inactive
jpraynaud
force-pushed
the
jpraynaud/1147-fix-e2e-flakiness
branch
10 times, most recently
from
1f329d5
to
b9f63a1
Compare
jpraynaud
force-pushed
the
jpraynaud/1147-fix-e2e-flakiness
branch
5 times, most recently
from
e5ba96a
to
5a75559
Compare
Bootstrap protocol parameters on first launch only. Attempt to fix temporal effects at startup (fast epochs) resulting in error: '`mithril-aggregator genesis bootstrap` exited with code: 1'
The discrepancy can occur not only at startup and not only on the 'serve' command.
Tail last 40 lines instead of 20 lines.
This was responsible a source of flakiness in the e2e test. This lead to Cardano nodes running on different divergent forks, and created panic on era reader which did not retrieve any marker.
This was responsible for flakiness in the network with low values of slot length and epoch length (such as those used in the E2E test). It was responsible for era markers readers retrieving no values and panicking.
This avoids flakiness due to 'InsufficientPeers' errors from the P2P network.
Use instead of the default value, to avoid 'InsufficientPeers' errors in the P2P network.
Attempt to avoid not reaching the quorum in some rare cases.
Use the security parameter value ('10') to make sure that transaction is synchornized on all Cardano nodes.
This avoids getting some nodes panic at startup beacuse of empty era markers retrieval.
This will avoid panic on Mithril nodes in case of rollback, which results in empty markers on chain.
We currently sign the penultimate immutable chunk, which means that transactions could either be in the ultimate immutable chunk or in the volatile db, thus not signed by Mithril. This has lead to having only a subset of the submitted transactions verified by the client.
This will avoid to have an unnecessary BFT node running in the devnet when started from the e2e test.
- 'mithril-aggregator' from '0.4.39' to '0.4.40' - 'mithril-common' from '0.3.7' to 0.3.8' - 'mithril-relay' from '0.1.11' to '0.1.12' - 'mithril-enn-to-end' from '0.3.3' to '0.4.0' - 'mithril-devnet' from '0.3.0' to '0.3.1'
jpraynaud
force-pushed
the
jpraynaud/1147-fix-e2e-flakiness
branch
from
45629d1
to
fa5e910
Compare
jpraynaud
temporarily deployed
to
testing-preview
GitHub Actions
Inactive
jpraynaud
temporarily deployed
to
testing-sanchonet
GitHub Actions
Inactive
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Content
This PR includes fixes to remove observed flakiness in the CI lately:
Error: 'mithril-aggregator genesis bootstrap' exited with code: 1Timeout exhausted for target epoch to be reachedMinimum expected mithril stake distribution epoch not reached : XX < YYRun devnet exited with status code: 1Write era marker on chain exited with status code: 1Timeout exhausted assert_node_producing_mithril_stake_distribution, no response from http://localhost:8080/aggregator/artifact/mithril-stake-distributionsPre-submit checklist
Issue(s)
Closes #1147