Skip to content

Tentative testdriver methods #226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

jcsteh
Copy link

@jcsteh jcsteh commented Jun 3, 2025

To answer these questions, it would be helpful if browser vendors could collaborate on a tentative but working implementation.

## Details
1. There must be publicly documented (e.g. as part of an Interop Investigation Area) consensus between at least two browser vendors and intent to work towards a specification.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that the tests for these tentative methods should be marked with tentative. But I think we should treat these tentative methods similarly to how we treat tentative tests for the sake of consistency. Given tentative tests do not require consensus currently, I would remove this requirement from tentative methods.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd argue that tentative testdriver API pose a greater risk, because removing tests that rely on undocumented API is painful and hard — see #172.

Copy link
Member

@gsnedders gsnedders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of my comments below are really "let's make this more explicit", rather than any real objection.

And mostly a question for the Core Team, but I do wonder what standard we thing we're holding things to based on #127: is an explainer enough for a testdriver API, or do we want a spec? Because if an explainer is enough, I'd question how often we'd actually add tentative API, because we probably do want something defining what the behaviour of the endpoint is, even if only at a high-level.

# RFC 226: Tentative testdriver methods

## Summary
Allow tentative methods to be added to testdriver.js that are not yet included in the WebDriver or WebDriver BiDi specifications.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's be more explicit; we already have plenty of non-tentative things in testdriver which are not in either spec — #127 (final RFC) merely requires a WebDriver endpoint, not that it is in one of those specs.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. I had assumed that the existence of a WebDriver endpoint implied it needed to be in the WebDriver spec. So how is WebDriver endpoint defined, given that Gecko uses Marionette for WPT and no other vendors implement this WebDriver endpoint yet?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, https://webbluetoothcg.github.io/web-bluetooth/#automated-testing defines a whole load of endpoints.

The policy is that it requires a spec for the endpoint, not that that spec be in any specific standard.

As part of the [Interop 2025 Accessibility Investigation Area](https://github.com/web-platform-tests/interop-accessibility/issues/148), the intention is to extend browsers and WPT to allow additional accessibility properties to be tested by web platform tests.
However, there are open questions regarding the shape of the API for exposing these properties, what properties should be exposed, how the tests should be written, etc.
While the end goal is to extend the WebDriver specification with the required new endpoints, these open questions need to be answered before this is feasible.
To answer these questions, it would be helpful if browser vendors could collaborate on a tentative but working implementation.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For things like the Accessibility Investigation, is there any intention for there to be any, even high-level, explainer of what the proposed endpoints are? I'm mostly concerned about the risk of ending up somewhere where it is hard for anyone else to start passing the tests written using the tentative endpoint.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is WICG/aom#203. However, that's an evolving issue, not a document. I could create an explainer document if that's what we need, though.

To answer these questions, it would be helpful if browser vendors could collaborate on a tentative but working implementation.

## Details
1. There must be publicly documented (e.g. as part of an Interop Investigation Area) consensus between at least two browser vendors and intent to work towards a specification.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd argue that tentative testdriver API pose a greater risk, because removing tests that rely on undocumented API is painful and hard — see #172.

Comment on lines +15 to +16
2. Any tentative methods added to testdriver.js should be prefixed with `tentative_`.
3. Any utility methods which call tentative testdriver methods should also be prefixed with `tentative_`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we want there to be a tentative object, or whether we just want to prefix everything like this?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if having a tentative object would make it trickier to "graduate" things from tentative once the spec is final (and my thinking is that we want to incentivise finalising tentative things as soon as is feasible). With prefixing, you can basically just search/replace once the spec is final. With an object, things will need to be moved around. That's obviously not a show stopper if an object is preferred, just something I thought worth considering.

Copy link
Contributor

@Ms2ger Ms2ger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no fundamental concerns with this, assuming we'd use this for experimentation, not things that stick around indefinitely.

1. There must be publicly documented (e.g. as part of an Interop Investigation Area) consensus between at least two browser vendors and intent to work towards a specification.
2. Any tentative methods added to testdriver.js should be prefixed with `tentative_`.
3. Any utility methods which call tentative testdriver methods should also be prefixed with `tentative_`.
4. Any tests directly or indirectly calling tentative testdriver methods must be marked tentative.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
4. Any tests directly or indirectly calling tentative testdriver methods must be marked tentative.
4. Any tests directly or indirectly calling tentative testdriver methods must be marked tentative by including `.tentative.` in the file name.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather we just cross-reference https://web-platform-tests.org/writing-tests/file-names.html#:~:text=.tentative,-:%20 rather than state that specifically — we aren't trying to require that the tests have .tentative. in their name specifically, merely that they match our normal tentative semantics.

## Alternatives considered
1. Avoid any changes to the WPT repository until the specification is finalised.
This makes it very difficult for browser vendors to collaborate and to learn about problems that are much easier to discover and understand "in practice".
In contrast, if the specification were developed without a working implementation, there is a much higher risk of fundamental design problems in the specification which are much harder to fix later.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps the initial cross-engine prototype of the property bag implementation should include only the properties already exposed by other WebDriver methods, like computedrole and computedlabel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants