ci: Prevent E2E timeouts on release changes (#26846) [cherry-pick] #26856
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## **Description** Our CI was setup to compare each branch with `develop`, and run any tests that have changed since `develop` extra times to catch new flaky test regressions. Unfortunately this was happening even for PRs that do not target develop, resulting in massive lists of "changed" tests that ended up causing persistent test timeouts. The quality gate should only impact PRs that target `develop`. PRs that target other branches no longer repeat "changed" tests. This should fix release PR e2e test timeouts caused by excessive e2e test runs. You can see an example of this problem occurring here: https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/98357/workflows/b61a16b5-acfd-411f-b82e-7a31706ca658/jobs/3660843 Notice that `/home/circleci/project/test/e2e/tests/request-queuing/ui.spec.js` appears 6 times in the full test list, as to every other "changed" test. [](https://codespaces.new/MetaMask/metamask-extension/pull/26846?quickstart=1) ## **Related issues** This quality gate was first introduced in #24556 ## **Manual testing steps** The script can be run locally if you setup "fake" CircleCI environment variables to allow it to run properly. For example: ``` CIRCLE_PULL_REQUEST=#26822 yarn tsx ./.circleci/scripts/git-diff-develop.ts ``` The CI run for this PR can be viewed as well: https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/98369/workflows/e9de2170-a19e-433e-a83c-846eeb239842/jobs/3661034 ## **Screenshots/Recordings** N/A ## **Pre-merge author checklist** - [x] I've followed [MetaMask Contributor Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Extension Coding Standards](https://github.com/MetaMask/metamask-extension/blob/develop/.github/guidelines/CODING_GUIDELINES.md). - [x] I've completed the PR template to the best of my ability - [x] I’ve included tests if applicable - [x] I’ve documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [x] I’ve applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-extension/blob/develop/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. ## **Pre-merge reviewer checklist** - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.
danjm
approved these changes
Sep 3, 2024
](https://codespaces.new/MetaMask/metamask-extension/pull/26846?quickstart=1){kind=link}
metamaskbot
added
the
release-12.2.0
|
No release label on PR. Adding release label release-12.2.0 on PR, as PR was cherry-picked in branch 12.2.0. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a cherry-pick of #26846 for v12.2.0. Original description:
Description
Our CI was setup to compare each branch with
develop, and run any tests that have changed sincedevelopextra times to catch new flaky test regressions. Unfortunately this was happening even for PRs that do not target develop, resulting in massive lists of "changed" tests that ended up causing persistent test timeouts.The quality gate should only impact PRs that target
develop. PRs that target other branches no longer repeat "changed" tests. This should fix release PR e2e test timeouts caused by excessive e2e test runs.You can see an example of this problem occurring here: https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/98357/workflows/b61a16b5-acfd-411f-b82e-7a31706ca658/jobs/3660843 Notice that
/home/circleci/project/test/e2e/tests/request-queuing/ui.spec.jsappears 6 times in the full test list, as to every other "changed" test.Related issues
This quality gate was first introduced in #24556
Manual testing steps
The script can be run locally if you setup "fake" CircleCI environment variables to allow it to run properly. For example:
The CI run for this PR can be viewed as well:
https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/98369/workflows/e9de2170-a19e-433e-a83c-846eeb239842/jobs/3661034
Screenshots/Recordings
N/A
Pre-merge author checklist
Standards.
Pre-merge reviewer checklist