Add support for testing the WebExtensions API in WPT

[kiaraarose]

WPT PR can be found here.

[jgraham]

Can we get an update here to detail the current proposal? At the moment what's in the RFC doesn't seem to match what's actually in the patch.

My understanding is that the proposal is for a new test wrapper indicated by .extension.js in the filename, a new testdriver API, and a function exposed by the wrapper that implements a protocol for transferring test results back from the extension? In particular the latter protocol needs to be specified somewhere (I assume it's what browser.test does, but unless that has a written specification somewhere we need enough detail in the RFC that one can understand what the requirements are without reading Chromium-specific documents).

[gsnedders]

I think the RFC should probably also address the question of "do we want to add a new testdriver feature", given that's important to Chromium where running these tests will inherently depend on BiDi, even though that's not true in other implementations.

Just using the "bidi" flag is potentially problematic if we want to closely tie that to BiDi usage generally, given other implementations will run these tests without BiDi.

[gsnedders]

I think putting this section into the Web Extensions spec would probably be sensible? Like sure, it's not super well-defined, but it's a starting point for a spec to then be iterated on from.

[kiaraarose]

That is the goal. There are still APIs within this namespace that would need documentation. I only added documentation for the APIs used in the PR.

[gsnedders]

I think from a Core Team point of view this is our only real concern with the RFC as it stands; splitting out this section into some CG document would be an improvement.

If that can be done quickly, that would be ideal to do before merging.

[zombie]

I have confirmed this design would work for Gecko's implementation of WebExtensions, I have a stack of patches that are ready to be submitted for review once this rfc is approved.

[OrKoN]

I think Chromium will be able to support these tests using the WebDriver BiDi implementation with the current proposal. Optional feedback and some notes:

• feature=extensions allows Chromium to switch over to WebDriver BiDi implementation but the PR might require some changes (I encountered some errors trying to run the PR so I could not test it). The feature flag could also serve as indication if the test globals need to be exposed to not potentially affect other tests.
• When discussing internally, we were not sure if a new wrapper type (.extension.js) is really beneficial given that the test code runs in a page similar to the .window.js tests and not in an extension.
• I still do not think that there is a need for an alternative WebDriver Classic specification that is not meant to become the standard. Even if we end up linking to it, I think it'd be good to put methods under the extensions namespace in testdriver.js test_driver.web_extension.install/test_driver.web_extension.uninstall to add future WebDriver BiDi methods there too if needed (e.g., w3c/webdriver-bidi#898). I do not have a strong opinion here but it was brought up in discussions.

@oliverdunk I have not reviewed browser.test APIs but I assume Chromium will work with what is proposed in this doc.

[rdcronin]

Thank you, Kiara!

[Rob--W]

Looks good with the following feedback addressed. Where possible I provided suggested edits for clarification.

[rdcronin]

Thanks, Kiara!

[rdcronin]

Thanks, Kiara! There are a couple comments here that we've talked about before and have been marked as resolved, but I don't think have been fully integrated into the proposal.

[rdcronin]

Thanks, Kiara! I'd still like to iterate on how we pass the test status to the outer / WPT runner, but I think we're generally aligned on this shape, and I don't want to hold this up. Thanks for all your hard work here!

[gsnedders]

Note #219 (comment), given GitHub is terrible at threading:

I think from a Core Team point of view this is our only real concern with the RFC as it stands; splitting out [the browser.test API] section into some CG document would be an improvement.

If that can be done quickly, that would be ideal to do before merging.

[jgraham]

This is broadly fine, and although I have some feedback, it's not blocking.

I also think that it would be ideal to move the test API to an actual spec in the WebExt WG (or at least a note in the CG), rather than trying to document it here. But in the absence of that hopefully this is enough to allow people to build more or less interoperable implementations.

[jgraham]

I think for web-platform-tests expectedError must not be used unless the error messages are more standardized than normal JS/DOM exceptions, since otherwise it's embedding browser-specific expectations into the test.

[jgraham]

Just as a style note, these are usually written in the present or future tense rather than the past, as a description of what the proposal is and what changes it will make.

[jgraham]

This is obviously your existing API, but for web-platform-tests you really want to structure things so that tests are as independent as possible i.e. later tests running doesn't depend on earlier tests passing. We generally have the semantics that asserts fail the test and stop further execution of that one test, but as far as possible don't prevent future tests running. Otherwise you can get into a situation where a browser that would pass all but one of the subtests in a specific file actually stops after the first (failing) subtest.