chore: adds quality gate for rerunning e2e spec files that are new or have been modified #24556

seaona · 2024-05-16T11:15:53Z

Description

This PR adds a quality gate for new or modified e2e spec files. Whenever there is a PR which modifies or changes a test, this will be run more times, in order to prevent introducing a flakiness accidentally. It is done as follows:

Identifies any new or modified e2e file from inside the test/ folder by making a github request and using these 2 filters:
- file.filename.startsWith('test/e2e/') &&
- file.filename.endsWith('.spec.js') || file.filename.endsWith('.spec.ts')
Re-runs those files x5 times (meaning 1+5) -> this number is arbitrary, we could modify it to any value we want.
Since we already had a flag which could support the re-running successful tests, --retry-until-failure I just leveraged this into the for loop for each test, and if that testcase was identified as new/modified, the flag is added together with the new retries number (in this case 5)

Note: I am not adding any new job specific for this task. The problem of doing so is that, either:

we would have to have 1 job for each built against where the new/changed test is run --> this will require ~5 new jobs as we run the same specs agains multiple builts (chrome, firefox, mv3, multichain, redesign...)
or if we group everything into 1 job, we would miss the information against which built the test is failing

That's why the approach of using the same existings jobs was taken.
This has been discussed together with @chloeYue .

If there is a failure, the retry stops and returns error

Related issues

Fixes: #24009

Manual testing steps

Check ci runs (notice previous runs had failing and changed tests on purpose, in order to try the different scenarios described below)

Screenshots/Recordings

🟢 Case 1: A test has changed -> it's rerun 1+5 times and it's successful

personal-sign-runx5-times.mp4

https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/81343/workflows/55ac5a22-b195-404d-951f-529af37100f8

🟢 Case 2: A test has changed, but it has a mistake in the code (intentionally to simulate a flaky test) -> it fails immediately and there are no more retries

https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/81343/workflows/55ac5a22-b195-404d-951f-529af37100f8

🟢 Case 3: A PR has no test spec files changed -> nothing different happens. See Spec files that will be re-run is empty

Pre-merge author checklist

I’ve followed MetaMask Coding Standards.
I've completed the PR template to the best of my ability
I’ve included tests if applicable
I’ve documented my code using JSDoc format if applicable
I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

github-actions · 2024-05-16T11:16:06Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

test/e2e/run-all.js

seaona · 2024-05-16T14:33:16Z

test/e2e/run-all.js

@@ -212,12 +213,26 @@ async function main() {

  console.log('My test list:', myTestList);

+  const changedOrNewTests = await fetchChangedE2eFiles();
+  console.log('Spec files that will be re-run:', changedOrNewTests);


this is left for debugging purposes

You're going to sacrifice some parallel execution speed by doing it this way. It also only has to be done if it's running on CircleCI, and not locally.

It would be a better approach if at the top of function runningOnCircleCI(), you looked at testPaths and then duplicated 5 times each thing that was also in changedOrNewTests.

This would allow it to distribute the workload evenly across the VMs.

thank you for your suggestion @HowardBraham That's definetly a good point! I was thinking it would be negligible given the small amount of retries, but it's true that we can benefit from parallelization witha small tweak. I'll add the changes 🙇‍♀️

test/e2e/run-all.js

…logic is working properly in failing tests

test/e2e/run-all.js

seaona · 2024-05-21T07:56:42Z

development/lib/retry.js

+  if (retryUntilFailure) {
+    return null;
+  }
+


before it assumed that if we reach the max retries, it was an error and throw the error with rejection message 'Retry limit reached', but in the case of using retryUntilFailure, reaching the max retry limit it's actually a good thing, as it means the test has not failed. That's why in this case I'm returning just null

Can we rename this parameter stopAfterOneFailure for better clarity?

codecov · 2024-05-21T09:50:02Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.40%. Comparing base (cb7cc64) to head (71bbf50).
Report is 17 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop   #24556   +/-   ##
========================================
  Coverage    67.40%   67.40%           
========================================
  Files         1289     1289           
  Lines        50248    50248           
  Branches     13011    13011           
========================================
  Hits         33865    33865           
  Misses       16383    16383

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

metamaskbot · 2024-05-21T13:05:13Z

Builds ready [71bbf50]

builds: chrome, firefox
builds (beta): chrome
builds (flask): chrome, firefox
builds (MMI): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6
- common: 0, 1, 2, 3, 4, 5, 6, 7, 8
- content-script: 0
- ui: 0, 1, 10, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (623 ± 464 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	60	133	88	21	10
		domContentLoaded	9	54	17	12	6
		load	48	2385	623	966	464
		domInteractive	9	54	17	12	6

Bundle size diffs

background: 0 Bytes (0.00%)
ui: 0 Bytes (0.00%)
common: 0 Bytes (0.00%)

danjm · 2024-05-22T15:33:54Z

test/e2e/fetch-changed-files.js

@@ -0,0 +1,33 @@
+const axios = require('axios');


I'm not sure, but we might be able to avoid a fetch to github here, and instead just use git. Not sure if we have the repo and git history in the right time and place on CI to do it, but it would be good if we could: more efficient than a network request, and not dependent on network conditions or the github api possibly being down.

thank you Dan! That is a good point. I have an alternative branch where I explored a bit the git ci option, however it was not straight-forward and I believe we might need to tweak some things on the config.yml file in order to accomplish it. I decided to go for this option to not over-engineer it, but for the reason you mention it might be worth to try the ci option a bit further.

https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/80853/workflows/9ac00125-66fc-400d-82ac-3f22b4c928c1/jobs/2859112

I didn't try but something like this might work?

gitDiff = await git.diffSummary(['--name-only', `origin/develop...${process.env.CIRCLE_SHA1}`, 'test/**/*.spec.*s']);

https://github.com/MetaMask/metamask-extension/blob/develop/development/generate-rc-commits.js#L2

https://github.com/steveukx/git-js?tab=readme-ov-file#git-diff

Does the local git command like that work if you're doing a shallow checkout? Also, the way this is written, it will be hitting the GitHub API hundreds of times per workflow (because there are hundreds of parallel machines running the workflow). Perhaps do it in prep-deps, and then persist_to_workspace?

Good point on the shallow checkout. Could prob be addressed by a clever enough git fetch invocation (possibly in prep-deps)..? 🤔

thank you for the suggestions 🙏 ❤️ In the shallow clone we do this:

git clone --depth 1 --no-checkout "$CIRCLE_REPOSITORY_URL" . git fetch --depth 1 origin "$CIRCLE_SHA1"

I was thinking we would need to do git fetch develop explicitly to get available the develop branch commits in the ci environment? 🤔 I'm also not sure if we would need to increase the depth too here 🤔 I can investigate this further

HowardBraham · 2024-05-24T03:00:44Z

test/e2e/run-all.js

+      const retryIndex = args.indexOf('--retries');
+      if (retryIndex !== -1) {
+        args.splice(retryIndex, 2);
+      }
+
+      const extraArgs = isTestChangedOrNew
+        ? ['--retry-until-failure', `--retries=${retriesForChangedOrNewTests}`]
+        : [];


There's no need for this duct-taped workaround complexity. Look at line 191 in this file. Just put retries into extraArgs down here instead of args up there.

(if you go with my other suggestion about runningOnCircleCI though, you can probably remove most of this anyway)

thank you!! if we follow the approach of copying the spec files x5 times on runningOnCircleCI then we can indeed remove all of this logic. Just something I'm wondering, would it be okay then to treat each of those new specs the same as rest (same flags)? or we still want them to fail immediately with no-retries?

Given that @legobeat has a PR for reducing test retries, maybe it's fine to treat all the tests equally; then there's no need to add any extra flag, and just copy the spec file x times in the test list from runningOnCiircleCi, and this would simplify a lot the logic

What do you think?

## **Description**  [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/MetaMask/metamask-extension/pull/24787?quickstart=1) ## **Related issues** Fixes: ## **Manual testing steps** 1. Go to this page... 2. 3. ## **Screenshots/Recordings**  ### **Before**  ### **After**  ## **Pre-merge author checklist** - [ ] I’ve followed [MetaMask Coding Standards](https://github.com/MetaMask/metamask-extension/blob/develop/.github/guidelines/CODING_GUIDELINES.md). - [ ] I've completed the PR template to the best of my ability - [ ] I’ve included tests if applicable - [ ] I’ve documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [ ] I’ve applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-extension/blob/develop/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. ## **Pre-merge reviewer checklist** - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

danjm · 2024-05-29T12:36:49Z

.circleci/scripts/get-changed-files.sh

+echo "$DIFF_RESULT"
+
+# Store the output of git diff
+git diff --name-only develop..."$CIRCLE_SHA1" >> changed-files/changed-files.txt


maybe this should be a diff with origin/develop? because above there is the code to git fetch origin develop, but those commits are not pulled locally

quality gate mock alt

e66f2fa

seaona commented May 16, 2024

View reviewed changes

test/e2e/run-all.js Outdated Show resolved Hide resolved

seaona added 3 commits May 16, 2024 13:33

fix export

d2298f6

fix PR number match

d89be42

overwrite retries number for retry-until-failure cases

57269c9

seaona changed the title ~~chore: quality gate mock alt~~ chore: adds quality gate for rerunning e2e spec files that are new or have been modified May 16, 2024

seaona added 2 commits May 16, 2024 16:17

change file to ts and add log for debugging

3a8b0fa

revert js

90690cd

seaona commented May 16, 2024

View reviewed changes

test/e2e/run-all.js Outdated Show resolved Hide resolved

add ts spec files to the filter and make another test fail to verify …

65b3467

…logic is working properly in failing tests

DDDDDanica reviewed May 20, 2024

View reviewed changes

test/e2e/run-all.js Outdated Show resolved Hide resolved

seaona added 2 commits May 21, 2024 09:19

address dev review: move retries to variable remove try/catch

df8d21e

leave the specs as they were before (changed for ci testing purposes)

5029d49

seaona commented May 21, 2024

View reviewed changes

Merge branch 'develop' into quality-gate-gh

71bbf50

seaona marked this pull request as ready for review May 21, 2024 15:08

seaona requested review from kumavis and a team as code owners May 21, 2024 15:08

seaona added the team-extension-platform label May 21, 2024

seaona self-assigned this May 21, 2024

DDDDDanica previously approved these changes May 21, 2024

View reviewed changes

vthomas13 previously approved these changes May 21, 2024

View reviewed changes

seaona added the DO-NOT-MERGE Pull requests that should not be merged label May 22, 2024

danjm reviewed May 22, 2024

View reviewed changes

seaona requested review from vthomas13 and DDDDDanica May 23, 2024 11:17

HowardBraham reviewed May 24, 2024

View reviewed changes

seaona dismissed stale reviews from vthomas13 and DDDDDanica via 1c728ac May 27, 2024 14:13

seaona requested a review from a team as a code owner May 27, 2024 14:13

danjm reviewed May 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: adds quality gate for rerunning e2e spec files that are new or have been modified #24556

chore: adds quality gate for rerunning e2e spec files that are new or have been modified #24556

seaona commented May 16, 2024 •

edited

github-actions bot commented May 16, 2024

seaona May 16, 2024

HowardBraham May 24, 2024

seaona May 24, 2024

seaona May 21, 2024

HowardBraham May 24, 2024

codecov bot commented May 21, 2024 •

edited

metamaskbot commented May 21, 2024

danjm May 22, 2024

seaona May 23, 2024

legobeat May 23, 2024 •

edited

HowardBraham May 24, 2024

legobeat May 24, 2024 •

edited

seaona May 24, 2024

HowardBraham May 24, 2024

HowardBraham May 24, 2024

seaona May 24, 2024

danjm May 29, 2024

chore: adds quality gate for rerunning e2e spec files that are new or have been modified #24556

Are you sure you want to change the base?

chore: adds quality gate for rerunning e2e spec files that are new or have been modified #24556

Conversation

seaona commented May 16, 2024 • edited

Description

Related issues

Manual testing steps

Screenshots/Recordings

Pre-merge author checklist

Pre-merge reviewer checklist

github-actions bot commented May 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 21, 2024 • edited

Codecov Report

metamaskbot commented May 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

legobeat May 23, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

legobeat May 24, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seaona commented May 16, 2024 •

edited

codecov bot commented May 21, 2024 •

edited

legobeat May 23, 2024 •

edited

legobeat May 24, 2024 •

edited