Switch Puppeteer tests from CDP to WebDriver BiDi #17172

whimboo · 2023-10-25T08:40:46Z

Over the last months we have made great progress in implementing WebDriver BiDi in both Firefox and Chrome. Also Puppeteer uses already a lot of the new APIs when the webDriverBiDi protocol is enabled.

It would be good to see if pdf.js Puppeteer tests would already be in a position so that the underlying protocol can be changed to an officially spec'ed one. CC @marco-c.

All the testing commands by the bot can be found at #11851 (comment).

I'm going to use this PR to check which APIs might still be missing or failing.

Internally we are going to use this meta bug to track the remaining work on the WebDriver BiDi side:
https://bugzilla.mozilla.org/show_bug.cgi?id=1860992

calixteman · 2023-10-25T08:43:51Z

/botio test

moz-tools-bot · 2023-10-25T08:43:53Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/c321a5329bf3279/output.txt

moz-tools-bot · 2023-10-25T08:43:53Z

From: Bot.io (Windows)

Received

Command cmd_test from @calixteman received. Current queue size: 1

Live output at: http://54.193.163.58:8877/cbf9641d3f42e20/output.txt

moz-tools-bot · 2023-10-25T09:29:02Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/c321a5329bf3279/output.txt

Total script time: 45.14 mins

Font tests: Passed
Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.241.84.105:8877/c321a5329bf3279/reftest-analyzer.html#web=eq.log

moz-tools-bot · 2023-10-25T09:53:45Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/cbf9641d3f42e20/output.txt

Total script time: 60.00 mins

whimboo · 2023-10-25T10:17:20Z

/botio test

whimboo · 2023-10-25T10:23:08Z

/botio integrationtest

moz-tools-bot · 2023-10-25T10:23:09Z

From: Bot.io (Linux m4)

Received

Command cmd_integrationtest from @whimboo received. Current queue size: 0

Live output at: http://54.241.84.105:8877/4fba339dba81846/output.txt

moz-tools-bot · 2023-10-25T10:23:10Z

From: Bot.io (Windows)

Received

Command cmd_integrationtest from @whimboo received. Current queue size: 0

Live output at: http://54.193.163.58:8877/2fa84486562b274/output.txt

moz-tools-bot · 2023-10-25T11:01:04Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/2fa84486562b274/output.txt

Total script time: 37.89 mins

Integration Tests: FAILED

timvandermeij · 2023-10-28T12:39:46Z

Thank you for doing this, and good to see that WebdriverBidi is making good progress!

whimboo · 2023-10-30T13:04:17Z

I want to give a quick update. All the failures that I noticed should be fixed now and available via a recent Firefox release, and an up-to-date checkout of this repository. Only for Puppeteer we have to await a new release for the outstanding issue. Once that's done I'll try again if switching to BiDi will work across all platforms. I'm hoping for that this will happen during this week.

We should still check if the tests will work with Chrome as well over BiDi.

timvandermeij · 2023-11-12T18:24:07Z

@whimboo FYI, in case you want to give this another try, the Puppeteer fixes for puppeteer/puppeteer#11259 and puppeteer/puppeteer#11298 should be available in version 21.5.1, and yesterday we updated to that version here in 8e2c9a3 so it can be used if this patch is rebased.

calixteman · 2023-11-12T19:01:25Z

@whimboo I just tested on mac and few tests are failing with Firefox:

FTEST-UNEXPECTED-FAIL | must check that we've all the contents on copy/paste
FTEST-UNEXPECTED-FAIL | must check that text change can be undone/redone
FTEST-UNEXPECTED-FAIL | must update an existing annotation
FTEST-UNEXPECTED-FAIL | must move an annotation
FTEST-UNEXPECTED-FAIL | must not remove an empty annotation
FTEST-UNEXPECTED-FAIL | must open an existing annotation and check that the position are good
FTEST-UNEXPECTED-FAIL | must open an existing rotated annotation and check that the position are good
FTEST-UNEXPECTED-FAIL | must check the text can be selected with the mouse
FTEST-UNEXPECTED-FAIL | must check the keyboard event is limited to the input

whimboo · 2023-11-14T13:40:59Z

To give a brief summary we have two more issues for dblclick and tripleclick in our BiDi implementation. We are going to work on both soon to get it properly supported.

whimboo · 2023-12-01T08:30:57Z

The double and triple click issues are now fixed in Firefox Nightly. But there is now one remaining issue with a test of the page number input which fails to wait for the keyup event. We are working on it via https://bugzilla.mozilla.org/show_bug.cgi?id=1867523.

whimboo · 2023-12-07T09:15:10Z

The double and triple click issues are now fixed in Firefox Nightly. But there is now one remaining issue with a test of the page number input which fails to wait for the keyup event. We are working on it via https://bugzilla.mozilla.org/show_bug.cgi?id=1867523.

Actually this is an issue with Puppeteer, which is fixed but hasn't shipped yet, and as such for now would require a workaround in the pdf.js tests. We will have to wait landing this change until #17387 has been merged.

Nevertheless I'll update this PR now so that it is ready for getting merged.

whimboo · 2023-12-07T09:22:10Z

/botio integrationtest

moz-tools-bot · 2023-12-07T09:22:12Z

From: Bot.io (Linux m4)

Received

Command cmd_integrationtest from @whimboo received. Current queue size: 0

Live output at: http://54.241.84.105:8877/faef049b9ac2539/output.txt

moz-tools-bot · 2023-12-07T09:22:12Z

From: Bot.io (Windows)

Received

Command cmd_integrationtest from @whimboo received. Current queue size: 0

Live output at: http://54.193.163.58:8877/f4e0d8e9d9153c9/output.txt

moz-tools-bot · 2023-12-07T10:01:18Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/faef049b9ac2539/output.txt

Total script time: 39.09 mins

Integration Tests: FAILED

moz-tools-bot · 2023-12-07T10:17:23Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/f4e0d8e9d9153c9/output.txt

Total script time: 55.17 mins

Integration Tests: FAILED

whimboo · 2023-12-07T10:51:56Z

There are lot of issues with timeouts calling browsingContext.close. I'm not able to reproduce that locally, so I pushed an update which enables trace logs for the remote agent. I hope it will help to investigate the problem.

whimboo · 2023-12-07T10:52:06Z

/botio integrationtest

moz-tools-bot · 2023-12-07T10:52:08Z

From: Bot.io (Linux m4)

Received

Command cmd_integrationtest from @whimboo received. Current queue size: 0

Live output at: http://54.241.84.105:8877/33e8bee5181a907/output.txt

moz-tools-bot · 2023-12-07T21:59:42Z

From: Bot.io (Windows)

Received

Command cmd_integrationtest from @whimboo received. Current queue size: 1

Live output at: http://54.193.163.58:8877/f31a87d7136f189/output.txt

moz-tools-bot · 2023-12-07T22:04:45Z

From: Bot.io (Linux m4)

Success

Full output at http://54.241.84.105:8877/520c42f4365ee9d/output.txt

Total script time: 5.57 mins

Integration Tests: Passed

moz-tools-bot · 2023-12-07T22:10:30Z

From: Bot.io (Linux m4)

Success

Full output at http://54.241.84.105:8877/00e2160b3b4e762/output.txt

Total script time: 5.75 mins

Integration Tests: Passed

moz-tools-bot · 2023-12-07T22:14:59Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/fbec175849968b4/output.txt

Total script time: 15.80 mins

Integration Tests: FAILED

moz-tools-bot · 2023-12-07T22:29:38Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/f31a87d7136f189/output.txt

Total script time: 14.64 mins

Integration Tests: FAILED

whimboo · 2023-12-08T07:35:23Z

@Snuffleupagus could you maybe help us with the upgrade of Puppeteer? Due to broken depencencies I'm not able to do that from a M1 MBP.

whimboo · 2023-12-08T19:33:11Z

/botio integrationtest

moz-tools-bot · 2023-12-08T19:33:14Z

From: Bot.io (Linux m4)

Received

Command cmd_integrationtest from @whimboo received. Current queue size: 0

Live output at: http://54.241.84.105:8877/d4734d31a00af40/output.txt

moz-tools-bot · 2023-12-08T19:33:14Z

From: Bot.io (Windows)

Received

Command cmd_integrationtest from @whimboo received. Current queue size: 0

Live output at: http://54.193.163.58:8877/58d578eeb8d35c2/output.txt

moz-tools-bot · 2023-12-08T19:38:56Z

From: Bot.io (Linux m4)

Success

Full output at http://54.241.84.105:8877/d4734d31a00af40/output.txt

Total script time: 5.69 mins

Integration Tests: Passed

moz-tools-bot · 2023-12-08T19:50:13Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/58d578eeb8d35c2/output.txt

Total script time: 16.96 mins

Integration Tests: FAILED

whimboo · 2023-12-08T21:00:08Z

/botio integrationtest

moz-tools-bot · 2023-12-08T21:00:11Z

From: Bot.io (Linux m4)

Received

Command cmd_integrationtest from @whimboo received. Current queue size: 0

Live output at: http://54.241.84.105:8877/921433b493e40e1/output.txt

moz-tools-bot · 2023-12-08T21:00:11Z

From: Bot.io (Windows)

Received

Command cmd_integrationtest from @whimboo received. Current queue size: 0

Live output at: http://54.193.163.58:8877/0bba05088306154/output.txt

moz-tools-bot · 2023-12-08T21:05:57Z

From: Bot.io (Linux m4)

Success

Full output at http://54.241.84.105:8877/921433b493e40e1/output.txt

Total script time: 5.75 mins

Integration Tests: Passed

moz-tools-bot · 2023-12-08T21:14:05Z

From: Bot.io (Windows)

Success

Full output at http://54.193.163.58:8877/0bba05088306154/output.txt

Total script time: 13.89 mins

Integration Tests: Passed

calixteman · 2023-12-08T21:16:30Z

/botio test

moz-tools-bot · 2023-12-08T21:16:32Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/36c27c4b63b6553/output.txt

moz-tools-bot · 2023-12-08T21:16:32Z

From: Bot.io (Windows)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.193.163.58:8877/89075c4b5130fef/output.txt

whimboo · 2023-12-08T21:26:05Z

test/test.mjs

@@ -1056,9 +1056,6 @@ async function closeSession(browser) {
      continue;
    }
    if (session.browser !== undefined) {
-      for (const page of await session.browser.pages()) {
-        await page.close();
-      }
      await session.browser.close();


@OrKoN this actually fixes an issue with browsingContext.close() in puppeteer with Firefox when the context as passed into the BiDi call is null. Could this be a bug in Puppeteer? Or will some internal state change for other tabs as well when trying to close them all via such a loop?

moz-tools-bot · 2023-12-08T21:40:46Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/36c27c4b63b6553/output.txt

Total script time: 24.23 mins

Unit tests: Passed
Integration Tests: Passed
Regression tests: FAILED

  different ref/snapshot: 16
  different first/second rendering: 2

Image differences available at: http://54.241.84.105:8877/36c27c4b63b6553/reftest-analyzer.html#web=eq.log

moz-tools-bot · 2023-12-08T21:55:51Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/89075c4b5130fef/output.txt

Total script time: 39.30 mins

Unit tests: Passed
Integration Tests: Passed
Regression tests: FAILED

  errors: 1
  different ref/snapshot: 25

Image differences available at: http://54.193.163.58:8877/89075c4b5130fef/reftest-analyzer.html#web=eq.log

calixteman

LGTM. Thank you.

whimboo marked this pull request as draft October 25, 2023 08:40

timvandermeij added the test label Oct 28, 2023

whimboo force-pushed the webdriver-bidi branch from 1e2b530 to 177d874 Compare December 7, 2023 09:20

whimboo marked this pull request as ready for review December 7, 2023 09:50

whimboo force-pushed the webdriver-bidi branch from 177d874 to 7192ea8 Compare December 7, 2023 10:50

Run integration tests with WebDriver BiDi instead of CDP for Firefox.

4db0174

whimboo force-pushed the webdriver-bidi branch from a24c5b3 to 4db0174 Compare December 8, 2023 19:32

Remove unnecessary loop for closing open pages

d04d28a

whimboo commented Dec 8, 2023

View reviewed changes

calixteman approved these changes Dec 8, 2023

View reviewed changes

calixteman merged commit 07aa706 into mozilla:master Dec 8, 2023
8 of 9 checks passed

timvandermeij mentioned this pull request Dec 10, 2023

Failing GitHub Actions font-test was marked as successful #17396

Closed

whimboo deleted the webdriver-bidi branch December 11, 2023 08:40

timvandermeij mentioned this pull request Feb 17, 2024

[Regression]: page.goto API throws a TypeError if Firefox uses the WebDriverBiDi protocol puppeteer/puppeteer#11938

Closed

2 tasks

Switch Puppeteer tests from CDP to WebDriver BiDi #17172

Switch Puppeteer tests from CDP to WebDriver BiDi #17172

Conversation

whimboo commented Oct 25, 2023 • edited

calixteman commented Oct 25, 2023

moz-tools-bot commented Oct 25, 2023

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Oct 25, 2023

From: Bot.io (Windows)

Received

moz-tools-bot commented Oct 25, 2023

From: Bot.io (Linux m4)

Failed

moz-tools-bot commented Oct 25, 2023

From: Bot.io (Windows)

Failed

whimboo commented Oct 25, 2023

whimboo commented Oct 25, 2023

moz-tools-bot commented Oct 25, 2023

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Oct 25, 2023

From: Bot.io (Windows)

Received

moz-tools-bot commented Oct 25, 2023

From: Bot.io (Windows)

Failed

timvandermeij commented Oct 28, 2023

whimboo commented Oct 30, 2023 • edited

timvandermeij commented Nov 12, 2023 • edited

calixteman commented Nov 12, 2023

whimboo commented Nov 14, 2023

whimboo commented Dec 1, 2023

whimboo commented Dec 7, 2023

whimboo commented Dec 7, 2023

moz-tools-bot commented Dec 7, 2023

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Dec 7, 2023

From: Bot.io (Windows)

Received

moz-tools-bot commented Dec 7, 2023

From: Bot.io (Linux m4)

Failed

moz-tools-bot commented Dec 7, 2023

From: Bot.io (Windows)

Failed

whimboo commented Dec 7, 2023

whimboo commented Dec 7, 2023

moz-tools-bot commented Dec 7, 2023

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Dec 7, 2023

From: Bot.io (Windows)

Received

moz-tools-bot commented Dec 7, 2023

From: Bot.io (Linux m4)

Success

moz-tools-bot commented Dec 7, 2023

From: Bot.io (Linux m4)

Success

moz-tools-bot commented Dec 7, 2023

From: Bot.io (Windows)

Failed

moz-tools-bot commented Dec 7, 2023

From: Bot.io (Windows)

Failed

whimboo commented Dec 8, 2023

whimboo commented Dec 8, 2023

moz-tools-bot commented Dec 8, 2023

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Dec 8, 2023

From: Bot.io (Windows)

Received

moz-tools-bot commented Dec 8, 2023

From: Bot.io (Linux m4)

Success

moz-tools-bot commented Dec 8, 2023

whimboo commented Oct 25, 2023 •

edited

whimboo commented Oct 30, 2023 •

edited

timvandermeij commented Nov 12, 2023 •

edited