Playwright vs Cypress

This is a PoC for Playwright vs Cypress a simple React TS app.

Table of contents

• Setup
• Playwright
• Cypress
• CI Test results
• Comparison

Setup

After cloning this repo install the project dependencies at the project directory:

npm i

Start the app

npm start

Work around the Playwright MacOS permission issue

This workaround will prevent the dialog "Do you want the application “Chromium.app” to accept incoming network connection?"

Alternatively you can turn off the firewall.

Playwright

The tests are under ./playwright.

In a new tab inside your terminal, run the tests with the following command:

npm run playwright

There are two ways to view the UI while executing Playwright tests:

1. To execute tests with debugger, use Playwright Inspector.

   PWDEBUG=1 npm run playwright

2. Add { headless: false } to chromium launch function in the spec file.

Cypress

The tests are under ./cypress/integration.

To start the test runner:

npm run cypress:open

To run headlessly:

npm run cypress:run

CI Test results

Cypress Dashboard - uses 2 parallel machines.

Github Actions

yml file

Comparison

	Cypress Studio	Playwright
multi-domain, multi-tab scenarios	on the road map	supported
WebKit support	on the road map	supported
API testing	Great	n/a
component testing	supported Storybook plugin	n/a
network handling capabilities	Great Network can be fully stubbed, effectively enabling ui-component-integration testing	supported
point & click test recording (for beginners)	Cypress Studio	some support
local failure diagnosis	Debugging workflow that Cypress provides is still unmatched. The time-traveling and inspecting alongside the command log is an unignorable productivity boost	supported
access & debug at app source-code level	can use Cypress as dev environment instead of the browser Application action	n/a
community	Vast	at infancy
training & learning resources	Some of the best documentation online Recipies / patterns with code samples for most known challenges Vast amount of training	at infancy (some Puppeteer StackOverflow info still applies)
target audience	Open source and made for profit, monetized by Cypress Dashboard	made by Microsoft for Microsoft
reporting	Cypress Dashboard	built-in, Jest-like, and some 3rd party
parallelization	CI Load Balancing & Parallelization	local to the machine using workers
CI failure diagnosis	artifacts (screenshot, video), CLI code snippets of failure past execution data (failures, flake, commit correlation)	artifacts (screenshot, video) and CLI Trace viewer
CI test flake	Flakey test management Test Muting (ignore test flake you will not address) Test Burn-in (run new tests x times to ensure flake-freeness)	n/a
CI efficiency	Smart orchestration (run failed spec first, cancel run on failure)	n/a
analytics	run status & duration test suite size slowest tests top failures	n/a
SSO	Jira Okta Github Enterprise	n/a
tech support	included with Cypress Dashboard sub	n/a
automatic waiting	both
video capture, mobile emulation	both
supports Chrome dev protocol	both