Cypress uses Chrome Devtools Protocol #53

skitterm · 2024-01-05T21:16:15Z

What Changed

Cypress now uses Chrome Devtools Protocol (CDP) to archive the requested resources! This enables us to share most of the network-watching code (95%) between Playwright and Cypress.

Previously we were using the client-side cy.intercept(), which plays well with Cypress but would have required us to duplicate our network-watching code (and we'd have to remember to keep bugfixes and features up-to-date in both places).

The approach

Like Playwright, we use the Watcher class to listen to network requests/responses via CDP, and store those on the watcher.archive object. However, we use one watcher instance for the entire test run, instead of one instance per test (we still write the archives to disk at the end of each test). This is because Cypress doesn't give us a way to share the watcher between the "beforeEach" and "afterEach" on the server.

We still take the DOM snapshot on the client, and pass that to the server (where we add the watcher archive and send things off to be written to disk).

The API

Users will now need to add a second command to their cypress.config.js|ts file, as well as the on('before:browser:launch') lifecycle method. In practice, that will look like:

import { defineConfig } from 'cypress';
import { setupNetworkListener, onBeforeBrowserLaunch, saveArchives } from '../src/cypress-api';

export default defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      // implement node event listeners here
      on('task', {
        setupNetworkListener,
        saveArchives,
      });
      on('before:browser:launch', async (browser, launchOptions) => {
        await onBeforeBrowserLaunch(browser, launchOptions);

        return launchOptions;
      });
    },
  },
});

This is in addition to the user importing our client-side before/after hooks in their support.ts file (which they had to do before as well).

Tradeoffs/limitations

When using Electron (the default browser Cypress uses), we don't know what port CDP will be on. So we'll require users to supply that port number as an environment variable if they're using Electron. That would mean doing ELECTRON_EXTRA_LAUNCH_ARGS=--remote-debugging-port=<port-number> cypress run instead of just cypress run. I'll make sure this gets added to the docs. This limitation doesn't exist if we use the client-side Cy.intercept() to archive. Not ideal, but other network-watching Cypress plugins have the same limitation.
Snapshots aren't taken when Cypress is run in Firefox. Luckily the tests still execute and finish, so it wouldn't impede workflows if people currently test in, say, Chrome and Firefox with Cypress.
For now, Cypress doesn't listen for network idleness to wrap up the archive. It just waits for the full test timeout amount (10s). I'll handle the network-idleness in a future PR so tests don't take forever.

How to test

See that the UI tests for Playwright still look the same
Run yarn test:playwright locally and confirm that the tests aren't waiting for the global 10s timeout to finish (making sure they're still using the network-idle-checking)
Observe the UI Tests for Cypress, and make sure they're the same as they were in previous builds when Cypress was using the client-side cy.intercept() for network-watching.

Author QA
Reviewer QA

Change Type

📦 Published PR as canary version: 0.0.54--canary.53.8d7bcec.0

✨ Test out this PR locally via:

npm install @chromaui/test-archiver@0.0.54--canary.53.8d7bcec.0
# or 
yarn add @chromaui/test-archiver@0.0.54--canary.53.8d7bcec.0

…teven/try-cdp

…Session type

skitterm · 2024-01-05T21:18:13Z

package.json

@@ -38,7 +38,7 @@
    "test:unit": "jest",
    "test:playwright": "playwright test",
    "test:cypress": "start-server-and-test test:server 3000 test:do-cypress",
-    "test:do-cypress": "cypress run --project tests",
+    "test:do-cypress": "ELECTRON_EXTRA_LAUNCH_ARGS=--remote-debugging-port=8192 cypress run --project tests",


We need to pass the ELECTRON_EXTRA_LAUNCH_ARGS stuff so that Cypress will know what port the devtools protocol is on. Electron for some reason doesn't pass this info along (and Electron is the default browser for Cypress runs). I'll make sure this gets added to our docs as well for users.

skitterm · 2024-01-05T21:25:11Z

src/resource-archive/index.ts

@@ -64,7 +70,7 @@ export class Watcher {
    await this.client.send('Fetch.enable');
  }

-  async idle(page: Page) {
+  async idle(page?: Page) {


For now, the network-idle listening doesn't happen for Cypress. This will be resolved when I add that in a future PR

skitterm · 2024-01-05T21:25:40Z

tests/cypress.config.ts

      on('task', {
-        archiveCypress,
+        setupNetworkListener,
+        saveArchives,
+      });
+      on('before:browser:launch', async (browser, launchOptions) => {
+        await onBeforeBrowserLaunch(browser, launchOptions);
+
+        return launchOptions;


This is the stuff the user will have to add to their cypress.config file.

skitterm · 2024-01-05T21:46:17Z

package.json

@@ -102,6 +103,7 @@
  },
  "dependencies": {
    "@segment/analytics-node": "^1.1.0",
+    "chrome-remote-interface": "^0.33.0",


This enables us to listen to CDP. Big thanks to cypress-har-generator plugin, which uses this lib and was a good pattern to follow.

tmeasday

Seems good! I guess we are still conversing about ways we might improve the API.

tmeasday · 2024-01-08T04:37:20Z

src/cypress-api/index.ts

-  await doArchive(params);
+// using a single Watcher instance across all tests (for the test run)
+// each time a test completes, we'll save to disk whatever archives are there at that point.
+// This should be safe since the same resource from the same URL should be the same during the entire test run.


Is this safe with the resource remapping stuff @jwir3 did?

// This should be safe since the same resource from the same URL should be the same during the entire test run.

As long as this invariant holds, then it should be fine with the resource mapping stuff. If the content of the url changes, though (the response), then it could be problematic.

@jwir3 can you elaborate on how the resource mapping stuff would make this a problem?

I'd be surprised if the resource (at the same URL) changed during the test run, but wanted to make sure I'm not missing something here.

tmeasday · 2024-01-08T04:38:09Z

src/cypress-api/index.ts

+// We use this lifecycle hook because we need to know what host and port Chrome Devtools Protocol is listening at.
+export const onBeforeBrowserLaunch = (


Did you consider making this part unnecessary if you are running in electron and passing in the env var?

I guess that'd complicate the DX and not save people much effort.

@tmeasday good point. We could remove the need for this lifecycle event handler if we expected everyone to always pass their port as an environment variable (I suppose we could require hostname as well, but I'd assume it's always 127.0.0.1).

Right now people only need to pass it if they're using Electron. But that is the default, and having a bunch of "in-this-case-you-need-to-add-this" messaging would not be good DX. WDYT?

I'm not sure, perhaps a question for DX / @byebrianwong

jwir3 · 2024-01-08T15:49:26Z

src/cypress-api/index.ts

-  await doArchive(params);
+// using a single Watcher instance across all tests (for the test run)
+// each time a test completes, we'll save to disk whatever archives are there at that point.
+// This should be safe since the same resource from the same URL should be the same during the entire test run.


// This should be safe since the same resource from the same URL should be the same during the entire test run.

As long as this invariant holds, then it should be fine with the resource mapping stuff. If the content of the url changes, though (the response), then it could be problematic.

tevanoff

Question about this comment in the PR message:

For now, Cypress doesn't listen for network idleness to wrap up the archive. It just waits for the full test timeout amount (10s)

Does that mean each test will take the full 10 seconds to run?

skitterm · 2024-01-08T18:29:02Z

Does that mean each test will take the full 10 seconds to run?

@tevanoff yes, it does. I'm working on a follow-on PR to use network idleness.

tevanoff

Looks good! Are you planning on holding off on merging this until the network idle PR is ready to go as well?

skitterm · 2024-01-08T21:07:46Z

Looks good!

@tevanoff thanks!

Are you planning on holding off on merging this until the network idle PR is ready to go as well?

I'm not planning on waiting to merge for the idle stuff -- since we haven't onboarded people onto Cypress yet, I figure we're free to make changes at will still.

skitterm · 2024-01-09T01:26:19Z

@tmeasday re-requesting your review as I've updated the on('before:browser:launch)' syntax and wanted to get your thoughts on always using the env variable or not. Seems we could wait on the latter change in a follow-up PR if needed.

tmeasday

LGTM. Let's look at the env var stuff in a followup.

thafryer · 2024-01-09T17:11:49Z

🚀 PR was released in v0.0.54 🚀

skitterm · 2024-01-09T17:59:01Z

@tevanoff PR #54 (just submitted) gets Cypress tests faster again (instead of waiting the global 10 seconds after each test).

skitterm added 13 commits December 29, 2023 15:18

Merge remote-tracking branch 'origin/steven/frameworkless-cdp' into s…

4f73788

…teven/try-cdp

Chrome-remote-interface package added

4280375

Cypress uses CDP instead of cy.intercept for network watching

2c1debc

Rely on watcher to know when we're done

0167859

Merge remote-tracking branch 'origin/main' into steven/try-cdp

b9530ca

Cypress tests write archives to disk from CDP

fb3d0fc

Better interface names, port can be found with electron or chrome

fbf6805

More-descriptive Cypress task names

e3d44d0

Cypress doesn't use the network idle listener for now

3cc105b

Most TS errors fixed

b3659e5

Using common, custom CDP client interface instead of Playwright's CDP…

7fe6e73

…Session type

Final comments

f648e60

Small promise change

3a69f4e

skitterm commented Jan 5, 2024

View reviewed changes

Clarifying comments

d18aaf3

skitterm commented Jan 5, 2024

View reviewed changes

Assumption stated

5fd1411

skitterm changed the title ~~Steven/try cdp~~ Cypress using Chrome Devtools Protocol Jan 5, 2024

skitterm changed the title ~~Cypress using Chrome Devtools Protocol~~ Cypress uses Chrome Devtools Protocol Jan 5, 2024

Clarification on what environment different parts of code are run in

a2baf20

skitterm commented Jan 5, 2024

View reviewed changes

skitterm requested review from tmeasday and tevanoff January 5, 2024 21:50

skitterm marked this pull request as ready for review January 5, 2024 22:04

tmeasday approved these changes Jan 8, 2024

View reviewed changes

jwir3 approved these changes Jan 8, 2024

View reviewed changes

tevanoff reviewed Jan 8, 2024

View reviewed changes

tevanoff approved these changes Jan 8, 2024

View reviewed changes

Use function reference for brevity

8d7bcec

skitterm requested a review from tmeasday January 9, 2024 01:24

tmeasday approved these changes Jan 9, 2024

View reviewed changes

skitterm merged commit 5962626 into main Jan 9, 2024
8 checks passed

thafryer added the released label Jan 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cypress uses Chrome Devtools Protocol #53

Cypress uses Chrome Devtools Protocol #53

skitterm commented Jan 5, 2024 •

edited by thafryer

skitterm Jan 5, 2024

skitterm Jan 5, 2024

skitterm Jan 5, 2024

skitterm Jan 5, 2024

tmeasday left a comment

tmeasday Jan 8, 2024

jwir3 Jan 8, 2024

skitterm Jan 8, 2024

tmeasday Jan 8, 2024

skitterm Jan 8, 2024 •

edited

tmeasday Jan 9, 2024

jwir3 Jan 8, 2024

tevanoff left a comment

skitterm commented Jan 8, 2024

tevanoff left a comment •

edited

skitterm commented Jan 8, 2024

skitterm commented Jan 9, 2024 •

edited

tmeasday left a comment

thafryer commented Jan 9, 2024

skitterm commented Jan 9, 2024

		// We use this lifecycle hook because we need to know what host and port Chrome Devtools Protocol is listening at.
		export const onBeforeBrowserLaunch = (

Cypress uses Chrome Devtools Protocol #53

Cypress uses Chrome Devtools Protocol #53

Conversation

skitterm commented Jan 5, 2024 • edited by thafryer

What Changed

The approach

The API

Tradeoffs/limitations

How to test

Change Type

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tmeasday left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skitterm Jan 8, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tevanoff left a comment

Choose a reason for hiding this comment

skitterm commented Jan 8, 2024

tevanoff left a comment • edited

Choose a reason for hiding this comment

skitterm commented Jan 8, 2024

skitterm commented Jan 9, 2024 • edited

tmeasday left a comment

Choose a reason for hiding this comment

thafryer commented Jan 9, 2024

skitterm commented Jan 9, 2024

skitterm commented Jan 5, 2024 •

edited by thafryer

skitterm Jan 8, 2024 •

edited

tevanoff left a comment •

edited

skitterm commented Jan 9, 2024 •

edited