WebKit 18 fails driver-integration-tests-webkit (cookies.cy.js)

[MikeMcC399]

Current behavior

If an attempt is made to update playwright-webkit to the latest version in Cypress source code, it fails the Cypress test spec

• packages/driver/cypress/e2e/commands/cookies.cy.js

which is part of driver-integration-tests-webkit.

This is reproducible on Ubuntu with versions of playwright-webkit starting with 1.37.0 and extending to the current version (with WebKit 17.4 18.0).

Cypress is currently configured to use playwright-webkit@1.24.2 released in July 2022:

cypress/package.json

Line 189 in d8884e0

"playwright-webkit": "1.24.2",

Desired behavior

WebKit should pass all tests from driver-integration-tests-webkit when using the latest version of playwright-webkit.

• Qualify if this is due to bug(s) in Playwright/WebKit or Cypress
• Depending on outcome, report bugs to Playwright and / or fix bugs in Cypress

Test code to reproduce

npm install yarn@latest -g
git clone https://github.com/cypress-io/cypress
cd cypress
yarn add playwright-webkit@latest -D -W
yarn playwright install-deps webkit # Linux only

then

yarn workspace @packages/driver cypress:run --browser webkit --spec cypress/e2e/commands/cookies.cy.js

or

yarn workspace @packages/driver cypress:open --e2e --browser webkit

and select cypress/e2e/commands/cookies.cy.js to run in UI.

Playwright Version

First reported on playwright-webkit@1.45.3
Continues to be reproducible on playwright-webkit@1.48.2

Cypress Version

First reported on 13.13.2
Continues to be reproducible on 13.15.1

Node version

v18.17.1

Operating System

Ubuntu 22.04.4 LTS
Ubuntu 22.04.5 LTS and
Ubuntu 24.04.1 LTS

Debug Logs

See failures in CircleCI at
https://app.circleci.com/pipelines/github/cypress-io/cypress/63152/workflows/e49a5390-7dc4-432e-abce-391b205c080b/jobs/2615104

  1) src/cy/commands/cookies - no stub
       #getCookies
         returns cookies for the specified domain:

      AssertionError: expected 'key2' to equal 'key1'
      + expected - actual

      -'key2'
      +'key1'

      at  ([native code])
      at  (http://www.barbaz.com:3500/__cypress/tests:1717:41)
  getRet@
  tryCatcher@
  @
  thenFn@
  @
  tryCatcher@
  @
  @
  @
  _drainQueueStep@
  _drainQueue@
  @
  @

  2) src/cy/commands/cookies - no stub
       #getCookie
         returns the cookie from the specified domain:

      AssertionError: expected 'foobar.com value' to equal 'www.foobar.com value'
      + expected - actual

      -'foobar.com value'
      +'www.foobar.com value'

      at  ([native code])
      at  (http://www.barbaz.com:3500/__cypress/tests:1870:38)
  getRet@
  tryCatcher@
  @
  thenFn@
  @
  tryCatcher@
  @
  @
  @
  _drainQueueStep@
  _drainQueue@
  @
  @

  3) src/cy/commands/cookies
       #setCookie
         can set cookies with sameSite:

      AssertionError: expected { Object (name, value, ...) } to have property 'sameSite' of 'no_restriction', but got 'lax'
      + expected - actual

      -'lax'
      +'no_restriction'

      at  ([native code])
  applyChainer@
  arrayReduce@
  reduce@
  applyChainers@
  tryCatcher@
  @
  shouldFn@
  @
  tryCatcher@
  @
  @
  @
  _drainQueueStep@
  _drainQueue@
  @
  @

Other

• See WebKit (Experimental)
• https://playwright.dev/docs/browsers#webkit
• See chore(webkit): update playwright-webkit to 1.45.3 #29955 for failed attempt to update.

[MikeMcC399]

Further test results which may assist troubleshooting:

Results of yarn workspace @packages/driver cypress:run --browser webkit --spec cypress/e2e/commands/cookies.cy.js dependent on version of playwright-webkit

Playwright	pass	fail	WebKit	Notes
1.24.2	129	0	16.0	(1)
1.34.3	129	0	16.4	(2)
1.35.0	129	0	16.4	(3)
1.36.2	129	0	17.0	(3)
1.37.0	128	1	17.0	(4)
1.37.1	127	2	17.0	(5)
1.38.1	127	2	17.0	(5)
1.39.0	127	2	17.4	(5)
1.40.1	128	1	17.4	(4)
1.41.2	127	2	17.4	(5)
1.42.1	127	2	17.4	(5)
1.43.1	127	2	17.4	(5)
1.44.1	127	2	17.4	(5)
1.45.0	127	2	17.4	(6)
1.45.1	126	3	17.4	(7)
1.45.3	126	3	17.4	(7)
1.46.0	127	3	18.0	(7)

(1) Currently configured version
(2) Previously recommended version
(3) Test does not exit and instead wrongly opens Cypress runner at end in global mode
(4) Failure is
AssertionError: expected 'foobar.com value' to equal 'www.foobar.com value'
(5) Failures as (4) above, additionally
AssertionError: expected 'key2' to equal 'key1'
(6) Failure as (4) above, additionally
AssertionError: expected { Object (name, value, ...) } to have property 'sameSite' of 'no_restriction', but got 'lax'
(7) Failures as (4), (5) & (6) above

Change version with for instance:

yarn add playwright-webkit@1.45.3 -D -W
yarn playwright install-deps webkit

Test with:

yarn workspace @packages/driver cypress:run --browser webkit --spec cypress/e2e/commands/cookies.cy.js

View WebKit version with

yarn workspace @packages/driver cypress:open --e2e --browser webkit

Settings > Project settings > Resolved configuration (Fails to launch Cypress Runner with some old versions)

For debug logs use:

export DEBUG=cypress:server:browsers:webkit-automation