Skip to content

Conversation

@Mrtenz
Copy link
Member

@Mrtenz Mrtenz commented Mar 14, 2025

This migrates all browser tests which were previously run with WebdriverIO to Vitest. WebdriverIO had several issues like flakiness, lack of certain APIs since it uses Mocha, and performance in CI. Vitest does not seem to have these issues.

Long term we can start migrating all tests to Vitest instead of Jest. The APIs are largely the same between the two, so the migration should be relatively straightforward, and it simplifies some of our custom logic like merging coverage.

@socket-security
Copy link

socket-security bot commented Mar 14, 2025

New, updated, and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package New capabilities Transitives Size Publisher
npm/@vitest/browser@3.1.1 environment Transitive: filesystem, network +31 5.81 MB vitestbot
npm/@vitest/coverage-istanbul@3.0.8 Transitive: environment, eval, filesystem +11 2.07 MB vitestbot
npm/@vitest/coverage-v8@3.1.1 Transitive: environment, filesystem +10 1.3 MB vitestbot
npm/istanbul-lib-coverage@3.2.03.2.2 None 0 34.4 kB oss-bot
npm/istanbul-lib-report@3.0.03.0.1 None 0 37.6 kB oss-bot
npm/istanbul-reports@3.1.53.1.7 None 0 294 kB oss-bot
npm/make-dir@3.1.04.0.0 None 0 9.91 kB sindresorhus
npm/playwright-core@1.49.1 environment, eval, filesystem, network, shell, unsafe 0 7.97 MB yurys
npm/playwright@1.49.1 None +1 3.31 MB yurys
npm/vite-plugin-node-polyfills@0.23.0 None +10 1.26 MB davidmyersdev
npm/vitest@3.1.1 Transitive: environment, filesystem, shell, unsafe +30 1.98 MB

🚮 Removed packages: npm/@jridgewell/gen-mapping@0.3.3, npm/@jridgewell/set-array@1.1.2, npm/@testim/chrome-version@1.1.4, npm/@tootallnate/quickjs-emscripten@0.23.0, npm/@types/yauzl@2.10.0, npm/@wdio/logger@8.16.17, npm/ast-types@0.13.4, npm/axios@1.8.3, npm/basic-ftp@5.0.3, npm/big-integer@1.6.51, npm/binary@0.3.0, npm/bluebird@3.4.7, npm/buffer-crc32@0.2.13, npm/buffer-indexof-polyfill@1.0.2, npm/buffers@0.1.1, npm/chainsaw@0.1.0, npm/chromedriver@135.0.0, npm/compare-versions@6.1.0, npm/data-uri-to-buffer@5.0.1, npm/decamelize@6.0.0, npm/degenerator@5.0.1, npm/extract-zip@2.0.1, npm/fd-slicer@1.1.0, npm/fetch-blob@3.2.0, npm/formdata-polyfill@4.0.10, npm/fs-extra@8.1.0, npm/fstream@1.0.12, npm/geckodriver@4.2.0, npm/get-uri@6.0.1, npm/ip-regex@4.3.0, npm/is-url@1.2.4, npm/is2@2.0.9, npm/jackspeak@2.3.6, npm/jsonfile@4.0.0, npm/listenercount@1.0.1, npm/loglevel-plugin-prefix@0.8.4, npm/loglevel@1.8.1, npm/lru-cache@10.1.0, npm/netmask@2.0.2, npm/node-domexception@1.0.0, npm/node-fetch@3.3.2, npm/pac-proxy-agent@7.0.1, npm/pac-resolver@7.0.1, npm/path-scurry@1.10.1, npm/pend@1.2.0, npm/proxy-agent@6.4.0, npm/proxy-from-env@1.1.0, npm/tcp-port-used@1.0.2, npm/traverse@0.3.9, npm/universalify@0.1.2, npm/unzipper@0.10.14, npm/web-streams-polyfill@3.2.1, npm/webdriverio@8.43.0, npm/yauzl@2.10.0

View full report↗︎

@socket-security
Copy link

socket-security bot commented Mar 14, 2025

👍 Dependency issues cleared. Learn more about Socket for GitHub ↗︎

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

Ignoring: npm/tinypool@1.0.2, npm/playwright-core@1.49.1, npm/tinyexec@0.3.2, npm/istanbul-lib-report@3.0.1

View full report↗︎

Next steps

Take a deeper look at the dependency

Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support [AT] socket [DOT] dev.

Remove the package

If you happen to install a dependency that Socket reports as Known Malware you should immediately remove it and select a different dependency. For other alert types, you may may wish to investigate alternative packages or consider if there are other ways to mitigate the specific risk posed by the dependency.

Mark a package as acceptable risk

To ignore an alert, reply with a comment starting with @SocketSecurity ignore followed by a space separated list of ecosystem/package-name@version specifiers. e.g. @SocketSecurity ignore npm/foo@1.0.0 or ignore all packages with @SocketSecurity ignore-all

@Mrtenz Mrtenz force-pushed the mrtenz/vitest-browsers branch 2 times, most recently from e03fd92 to 5ce17c2 Compare March 27, 2025 12:02
@Mrtenz Mrtenz changed the title Use Vitest for snaps-execution-environments browser tests Use Vitest for browser tests Mar 27, 2025
@Mrtenz Mrtenz force-pushed the mrtenz/vitest-browsers branch from 8f5175c to d061a4b Compare March 28, 2025 09:07
@Mrtenz
Copy link
Member Author

Mrtenz commented Mar 28, 2025

@SocketSecurity ignore npm/istanbul-lib-report@3.0.1

New author is ok.

};
}, {}]
}, {}, [1]);
//# sourceMappingURL=data:application/json;charset=utf-8;base64,eyJ2ZXJzaW9uIjozLCJuYW1lcyI6WyJyIiwiZSIsIm4iLCJ0IiwibyIsImkiLCJmIiwiYyIsInJlcXVpcmUiLCJ1IiwiYSIsIkVycm9yIiwiY29kZSIsInAiLCJleHBvcnRzIiwiY2FsbCIsImxlbmd0aCIsIm1vZHVsZSIsIm9uUnBjUmVxdWVzdCIsInJlcXVlc3QiLCJjb25zb2xlIiwibG9nIiwibWV0aG9kIiwiaWQiXSwic291cmNlcyI6WyIuLi8uLi9ub2RlX21vZHVsZXMvYnJvd3Nlci1wYWNrL19wcmVsdWRlLmpzIiwiX3N0cmVhbV8wLmpzIl0sInNvdXJjZXNDb250ZW50IjpbIihmdW5jdGlvbigpe2Z1bmN0aW9uIHIoZSxuLHQpe2Z1bmN0aW9uIG8oaSxmKXtpZighbltpXSl7aWYoIWVbaV0pe3ZhciBjPVwiZnVuY3Rpb25cIj09dHlwZW9mIHJlcXVpcmUmJnJlcXVpcmU7aWYoIWYmJmMpcmV0dXJuIGMoaSwhMCk7aWYodSlyZXR1cm4gdShpLCEwKTt2YXIgYT1uZXcgRXJyb3IoXCJDYW5ub3QgZmluZCBtb2R1bGUgJ1wiK2krXCInXCIpO3Rocm93IGEuY29kZT1cIk1PRFVMRV9OT1RfRk9VTkRcIixhfXZhciBwPW5baV09e2V4cG9ydHM6e319O2VbaV1bMF0uY2FsbChwLmV4cG9ydHMsZnVuY3Rpb24ocil7dmFyIG49ZVtpXVsxXVtyXTtyZXR1cm4gbyhufHxyKX0scCxwLmV4cG9ydHMscixlLG4sdCl9cmV0dXJuIG5baV0uZXhwb3J0c31mb3IodmFyIHU9XCJmdW5jdGlvblwiPT10eXBlb2YgcmVxdWlyZSYmcmVxdWlyZSxpPTA7aTx0Lmxlbmd0aDtpKyspbyh0W2ldKTtyZXR1cm4gb31yZXR1cm4gcn0pKCkiLCJcbiAgbW9kdWxlLmV4cG9ydHMub25ScGNSZXF1ZXN0ID0gKHsgcmVxdWVzdCB9KSA9PiB7XG4gICAgY29uc29sZS5sb2coXCJIZWxsbywgd29ybGQhXCIpO1xuXG4gICAgY29uc3QgeyBtZXRob2QsIGlkIH0gPSByZXF1ZXN0O1xuICAgIHJldHVybiBtZXRob2QgKyBpZDtcbiAgfTtcbiJdLCJtYXBwaW5ncyI6IkFBQUE7RUFBQSxTQUFBQSxFQUFBQyxDQUFBLEVBQUFDLENBQUEsRUFBQUMsQ0FBQTtJQUFBLFNBQUFDLEVBQUFDLENBQUEsRUFBQUMsQ0FBQTtNQUFBLEtBQUFKLENBQUEsQ0FBQUcsQ0FBQTtRQUFBLEtBQUFKLENBQUEsQ0FBQUksQ0FBQTtVQUFBLElBQUFFLENBQUEsd0JBQUFDLE9BQUEsSUFBQUEsT0FBQTtVQUFBLEtBQUFGLENBQUEsSUFBQUMsQ0FBQSxTQUFBQSxDQUFBLENBQUFGLENBQUE7VUFBQSxJQUFBSSxDQUFBLFNBQUFBLENBQUEsQ0FBQUosQ0FBQTtVQUFBLElBQUFLLENBQUEsT0FBQUMsS0FBQSwwQkFBQU4sQ0FBQTtVQUFBLE1BQUFLLENBQUEsQ0FBQUUsSUFBQSx1QkFBQUYsQ0FBQTtRQUFBO1FBQUEsSUFBQUcsQ0FBQSxHQUFBWCxDQUFBLENBQUFHLENBQUE7VUFBQVMsT0FBQTtRQUFBO1FBQUFiLENBQUEsQ0FBQUksQ0FBQSxLQUFBVSxJQUFBLENBQUFGLENBQUEsQ0FBQUMsT0FBQSxZQUFBZCxDQUFBO1VBQUEsSUFBQUUsQ0FBQSxHQUFBRCxDQUFBLENBQUFJLENBQUEsS0FBQUwsQ0FBQTtVQUFBLE9BQUFJLENBQUEsQ0FBQUYsQ0FBQSxJQUFBRixDQUFBO1FBQUEsR0FBQWEsQ0FBQSxFQUFBQSxDQUFBLENBQUFDLE9BQUEsRUFBQWQsQ0FBQSxFQUFBQyxDQUFBLEVBQUFDLENBQUEsRUFBQUMsQ0FBQTtNQUFBO01BQUEsT0FBQUQsQ0FBQSxDQUFBRyxDQUFBLEVBQUFTLE9BQUE7SUFBQTtJQUFBLFNBQUFMLENBQUEsd0JBQUFELE9BQUEsSUFBQUEsT0FBQSxFQUFBSCxDQUFBLE1BQUFBLENBQUEsR0FBQUYsQ0FBQSxDQUFBYSxNQUFBLEVBQUFYLENBQUEsSUFBQUQsQ0FBQSxDQUFBRCxDQUFBLENBQUFFLENBQUE7SUFBQSxPQUFBRCxDQUFBO0VBQUE7RUFBQSxPQUFBSixDQUFBO0FBQUE7RUFBQSxjQUFBUSxPQUFBLEVBQUFTLE1BQUEsRUFBQUgsT0FBQTtJQ0NBRyxNQUFBLENBQUFILE9BQUEsQ0FBQUksWUFBQTtNQUFBQztJQUFBO01BQ0FDLE9BQUEsQ0FBQUMsR0FBQTtNQUVBO1FBQUFDLE1BQUE7UUFBQUM7TUFBQSxJQUFBSixPQUFBO01BQ0EsT0FBQUcsTUFBQSxHQUFBQyxFQUFBO0lBQ0EifQ=="
Copy link
Member Author

@Mrtenz Mrtenz Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changed because of a bump in dependencies responsible for source maps.

Comment on lines -91 to +94
expect.assertions(4);
expect.assertions(5);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This number was wrong initially, it seems WDIO just ignores it.

Comment on lines -14 to -27
const snap = top.querySelector('iframe[data-testid="snaps-iframe"]');
// Parent is the iframe created by Vitest containing the Snap iframe
// and this page as a sibling iframe. We try to access the Snap iframe
// from the parent iframe.
const snap = parent.document.querySelector(
'iframe[data-testid="snaps-iframe"]',
);
if (!snap) {
window.parent.postMessage('Failed to find snap iframe.', '*');
window.parent.postMessage('Failed to find Snap iframe.', '*');
return;
}

// We try to access the snap iframe document. If this works, it means
// that the iframe is not sandboxed. Otherwise it will throw an error.
snap.document;

window.parent.postMessage(
'Same origin frames access worked which might suggest sandboxing failed.',
'*',
);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test was wrong in the first place. The iframe was served from a different origin, and trying to access top.querySelector resulted in:

SecurityError: Failed to read a named property 'querySelector' from 'Window': Blocked a frame with origin "http://localhost:4567/" from accessing a cross-origin frame.

So the snap.document was never reached, and top.querySelector isn't even a function even if it was accessible. 😅

I confirmed that the new test properly tests the sandboxing behaviour, as it's now served from the same origin. Removing the sandbox property from the iframe now results in a test failure.

@@ -0,0 +1 @@
export const MOCK_BLOCK_NUMBER = '0xa70e75';
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved this to a separate file because Vitest was trying to import Node.js dependencies through ./test-utils/index.ts.

@codecov
Copy link

codecov bot commented Mar 28, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.72%. Comparing base (8e418f6) to head (5f54a7f).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3236      +/-   ##
==========================================
- Coverage   94.88%   94.72%   -0.16%     
==========================================
  Files         518      518              
  Lines       11357    11932     +575     
  Branches     1752     1831      +79     
==========================================
+ Hits        10776    11303     +527     
- Misses        581      629      +48     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment on lines +14 to +34
// @ts-expect-error: Incompatibility between Vite versions.
NodeModulesPolyfillPlugin(),

// @ts-expect-error: Incompatibility between Vite versions.
NodeGlobalsPolyfillPlugin({
buffer: true,
}),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These plugins seem to be unmaintained, but swapping them out for vite-plugin-node-polyfills like I did in snaps-utils and snaps-controllers doesn't seem to work with SES (because the process is incompatible?). I haven't looked into it much further, but for now we can keep using these here.

@Mrtenz Mrtenz force-pushed the mrtenz/vitest-browsers branch from ea808c9 to ff90098 Compare March 28, 2025 12:49
"@metamask/*": ["../*/src"]
},
"resolveJsonModule": true,
"skipLibCheck": true,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vitest seems to import non-existing types from Vite. This seems like the easiest way to avoid it, and actually speeds up building too.

@Mrtenz
Copy link
Member Author

Mrtenz commented Mar 28, 2025

@SocketSecurity ignore npm/mute-stream@2.0.0
@SocketSecurity ignore npm/@bundled-es-modules/cookie@2.0.1

New author is ok.

@SocketSecurity ignore npm/tinypool@1.0.2
@SocketSecurity ignore npm/tinyexec@0.3.2

Shell access fine.

@Mrtenz Mrtenz marked this pull request as ready for review March 28, 2025 12:55
@Mrtenz Mrtenz requested a review from a team as a code owner March 28, 2025 12:56
*
* @returns The summary of the merged coverage.
*/
// TODO: Check if Vite's coverage is actually inaccurate.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's worth looking into in this PR, especially if we can avoid unnecessary computation.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like either Jest or Vite is inaccurate. I checked the iframe execution service for example: Vite shows it as completely covered, but after merging with Jest, one line is not covered.

The actual computation done here isn't really significant anyway though, and long term we can likely remove this once we migrate all tests to Vitest.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it turns out that we need this logic, then we should probably make this logic re-usable since we also use it in snaps-controllers

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use this in snaps-controllers, snaps-execution-environments, and snaps-utils. I agree it could be reusable, but I would prefer to do that in a separate PR.

import 'ses';

describe('BaseSnapExecutor', () => {
before(() => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why isn't this needed anymore?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why it was necessary in the first place anymore. It works without it now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can reuse this config as well?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Long term I want to start migrating more stuff to Vitest, at which point we can use workspace configs. For now I would prefer to keep it like this.

@hmalik88
Copy link
Contributor

Proof of concept for using Vitest for browser tests.

No longer a POC, i think? 😄

@Mrtenz Mrtenz requested a review from hmalik88 March 31, 2025 06:50
hmalik88
hmalik88 previously approved these changes Mar 31, 2025
@Mrtenz Mrtenz force-pushed the mrtenz/vitest-browsers branch from 79723a1 to 45b124c Compare April 3, 2025 10:31
@Mrtenz
Copy link
Member Author

Mrtenz commented Apr 3, 2025

@SocketSecurity ignore npm/playwright-core@1.49.1

Network access and shell access are fine.

uses: MetaMask/action-retry-command@v1
with:
command: yarn install-chrome
command: yarn playwright install --with-deps --only-shell chromium firefox
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How will this work locally? Do you need to run any commands besides yarn and yarn test to run browser tests locally?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, Vitest will tell you to run yarn playwright install on the first time running yarn test.

'src/**/test-utils/**',
'src/**/__mocks__/**',
'src/**/node-js/**',
'src/services/node.ts',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reasoning for this and the index.ts exclude?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It caused the coverage merge script to break because Vitest had coverage for these files, but Jest didn't. We call jestMap.fileCoverageFor(file), which throws if the file doesn't exist.

@Mrtenz Mrtenz enabled auto-merge April 3, 2025 11:22
@Mrtenz Mrtenz added this pull request to the merge queue Apr 3, 2025
Merged via the queue into main with commit 7e6ad03 Apr 3, 2025
172 of 173 checks passed
@Mrtenz Mrtenz deleted the mrtenz/vitest-browsers branch April 3, 2025 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants