Skip to content

IDLHarness subtests "member names are unique" are missleading (pass always) and don't actually test anything on the browser #52659

Open
@clopez

Description

@clopez

One of the idlharness subtests that run on mixin interfaces is named member names are unique

And, as far as I can tell, what this subtest seem to be doing is to compare the fragment IDLs of a mixin interface to check that the members defined inside that interface are unique and not defined on other IDL interfaces where this can be included.

But for doing that check it is simply parsing the IDL files in the WPT directory interfaces and not actually testing anything on the browser.

So, this is really a test for the IDL files and not a test for the browser.

For example, given this IDL fragment:

interface mixin NavigatorBadge {
  Promise<undefined> setAppBadge(
    optional [EnforceRange] unsigned long long contents
  );
  Promise<undefined> clearAppBadge();
};

Navigator includes NavigatorBadge;

The code would happily create a test named Navigator includes NavigatorBadge: member names are unique which will simply pass if setAppBadge and clearAppBadge are not defined on any of the other IDL files that mixin with Navigator.

This means that this subtests passing or not doesn't depend at all on the browser running the test. It doesn't even check if setAppBadge and clearAppBadge are defined in navigator

Actually this sub-tests are always going to pass (assuming the IDL files are right).

And this causes miss-leading results like this one:

17/22 pass-rate on the badging/idlharness.https.any.html test on browsers that don't implement at all this interface (wpewebkit, ladybird, etc)

So I think we should either:

  • Assert that the browser actually defines the checked members on the interface.
    (or)
  • Move this tests to be python tests or other kind of unit tests related to wpt itself and not to the browser

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions