request: APIs and infrastructure to simplify cross-browser automated testing of extensions

[twschiller]

Context

Automated testing tools, e.g., Selenium, Cypress, Puppeteer, and Playwright are invaluable for cross browser web application testing. There's limited support for them for testing browser extensions

There are mocks (e.g,. jest-chrome) for unit testing, but these don't capture the complexity of extension communication and browser quirks

Automated cross-browser testing will be valuable for checking the progress of the browsers vs. this w3c group's proposals.

Request

I don't have a clear recommendation for what's needed here. My goal is to get the conversation kicked off (with people more knowledgable than me!) On the PixieBrix project, we're just now trying to set up our cross-browser browser automated testing infrastructure

I suspect there's needs to be some mix of:

• Exposing new APIs: e.g., for error detection, for adjusting permissions without user interaction, etc.
• The extension community contributing to the automated testing tool projects (code and documentation)
• Providing examples/documentation of testing different features across browsers

Related context:

• https://github.com/pixiebrix/pixiebrix-extension/issues/534
• https://groups.google.com/a/chromium.org/g/chromium-extensions/c/2GU7e-OKwtQ/m/CeLBiMe_BwAJ
• Firefox extensions with v2.1.0 puppeteer puppeteer/puppeteer#5532 (comment)
• https://groups.google.com/a/chromium.org/g/chromium-extensions/c/a2xNMNu3DCc/m/gEqUyOIXCAAJ

[Techdojo]

I’m especially interested in this as we’re currently working on ways to implement automated testing on browser extensions, feel free to reach out to me and we can discuss our respective strategies. Jon…

…

Sent from my iPhone

On 23 Jun 2021, at 18:37, Todd Schiller ***@***.***> wrote:  Context Automated testing tools, e.g., Selenium, Cypress, Puppeteer, and Playwright are invaluable for cross browser web application testing. There's limited support for them for testing browser extensions There are mocks (e.g,. jest-chrome) for unit testing, but these don't capture the complexity of extension communication and browser quirks Automated cross-browser testing will be important for checking the progress of the browsers vs. this w3c groups proposals. Request I don't have a clear recommendation for what's needed here. My goal is to get the conversation kicked off (with people more knowledgable than me!) On the PixieBrix project, we're just now trying to set up our cross-browser browser automated testing infrastructure I suspect there's needs to be some mix of: Exposing new APIs: e.g., for error detection, for adjusting permissions without user interaction, etc. The extension community contributing to the automated testing tool projects (code and documentation) Providing examples of testing different features across browsers Related context: pixiebrix/pixiebrix-extension#534 https://groups.google.com/a/chromium.org/g/chromium-extensions/c/2GU7e-OKwtQ/m/CeLBiMe_BwAJ puppeteer/puppeteer#5532 (comment) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[ghostwords]

At Privacy Badger, we use Selenium to continuously test functionality in Chrome and Firefox. The benefit of Selenium is that we exercise the actual extension in an actual browser.

Off the top of my head, we wish we had the ability to interact with browser chrome, in particular programmatically open the browser popup. As is, we have to resort to various hacks (examples: opening the popup, opening new browser windows).

[dotproto]

I’m interested in digging in here to better understand how automated end to end and unit testing of browser extensions can be improved. At the moment the biggest gap I know of is the highlighted by @ghostwords - there’s no way in WebDriver or Chrome DevTools Protocol to trigger a browser action. What other gaps are there?

[fregante]

@dotproto it seems that no tool currently allows running code in specific contexts, for example in the background page/worker, a devtool_page, or a content script. From what I gather, the only way to do so currently is to:

1. Load a page under chrome-extension:// in a new tab
2. Communicate to the other contexts from there
3. Have each context actually run the code locally

This means that in order to run test code in one of these contexts, the code needs to exist in the extension itself — at least during the testing phase.

Privacy Badger above seems to use getBackgroundPage() to access its window, but the code is still not executed in the background page (which may be a requirement in some cases).

Most guides online focus on loading the extension in the browser, but there's almost nothing about actually running extension code.

[tregagnon]

@dotproto Yes the main issue is about the Popup (or any browser chrome UI that can display an extension, like the Sidebar in Firefox).

Without a command to trigger the browser action, and open the Popup, usually the workaround for extension developers is to load the Popup page in a new tab using an extension URL (like chrome-extension://).

The problem with this method, is that it makes irrelevant any tests about features that access information about the current tab. Because extension URLs (chrome-extension://) are usually treated differently. And for proper testing, it would be better to test different situations.

[ghostwords]

Privacy Badger above seems to use getBackgroundPage() to access its window, but the code is still not executed in the background page (which may be a requirement in some cases).

What cases are those? Are you mixing up unit testing with functional testing?

[fregante]

Privacy Badger above seems to use getBackgroundPage() to access its window, but the code is still not executed in the background page (which may be a requirement in some cases).

What cases are those? Are you mixing up unit testing with functional testing?

I have several npm modules for WebExtensions so it'd be great that they are tested against real browsers and not just against mocks, which don't reflect the continuous changes, bugfixes and regressions that browsers get.

One explicit example:

• https://github.com/fregante/webext-detect-page has a test to detect the code is run in a specific context. This is generally used to share code/modules across contexts, but ensuring that they're not called the wrong context

But also:

• Ensuring that the content script is receiving messages from the background script and and it's filtering messages from any other page

[dotproto]

Last week I opened issue 3848 on the ChromeDriver project. My understanding at the time was that WebDriver was a better fit for this capability than CDP, but, as noted in that issue, after some discussion with the relevant team it seems like Chrome may experiment with this request in the Chrome DevTools Protocol.

[xeenon]

I would prefer we tackle this in WebDriver, since that is more at the browser level than the dev tool protocol level.

[ariasuni]

For reference, the «8. Testing Extensions with WebDriver» section of the draft of the previous CG:
https://browserext.github.io/browserext/#wd

[dotproto]

@xeenon, for clarity, are you referring to information gathering or where these features would be implemented? For information gathering, I think the WECG may be a good place to discuss use cases that are not well served today, to discuss potential solutions, and potentially even draft suggestions for consideration by other WGs. On the other hand if you meant that it should be implemented in WebDriver, I don't disagree.

The idea of using Chrome DevTools Protocol (CDP) was for experimentation, not as a long term solution. As I understand it, the Chrome DevTools team maintains CDP and may also be taking over ChromeDriver. Since CDP is a browser-specific feature, Puppeteer is built on CDP & is used in automated testing workflows, and that team has relevant expertise with both, it seemed like a good opportunity to explore concrete ways of addressing extension testing gaps, then bringing those findings to the WebDriver WG for discussion.

[ariasuni]

I just want to mention that there’s webextensions-api-mock who’s using the JSON schema used internally in Chrome and Firefox to automatically mock the whole APIs, and unit testing is a thing that should be taken into consideration also.

[cuylerstuwe]

The nature of browser extensions is that most of them intend to cause side-effects within the environment in which they're intended to run.

Therefore, for most extensions, end-to-end testing is overwhelmingly preferred, as they're the often the only test cases that prove anything meaningful.