Run only a supported set of WPT test types by default.

[jdm]

wptrunner introduced a new test type for print reftests, and by default any unsupported type causes the test runner to report an unexpected error, even if none of those tests are run. These changes avoid similar breakage by limiting the default test types to ones that are supported by our test runners.

[highfive]

Heads up! This PR modifies the following files:

• @wafflespeanut: python/servo/testing_commands.py

[jdm]

r? @Manishearth

[SimonSapin]

The name and description of this flag seem contradictory

[jdm]

Default in this case refers to the test of test types that are selected by mach's test-wpt command when this flag is not present. Is there a better way to describe it?

[jdm]

It's true - the default in the command name refers to the user experience and the flags selected by test-wpt. The default in the description refers to the flags selected by wptrunner without any other input.

[SimonSapin]

Default in this case refers to […] when this flag is not present.

In that case, the flag’s description "run the default set" is wrong since setting the flag makes a different set be used than the one called default.

Unless the "by wptrunner" (as opposed to mach) means that it intentionally refers to a different set. In this case, it’s not great to use the same word "default" to refer to two different sets in the flag’s name and its description.

[jdm]

@SimonSapin I tried a different description.

[SimonSapin]

Sorry to nit pick :) This description is better but --no-default-test-types sounds like it runs fewer test types, but it runs more. How about --all-test-types?

[Manishearth]

r=me, I also prefer all-test-types

[jdm]

So, I like the current argument name because it also allows --no-default-test-types --test-types reftest print-reftest. I find that easier to reason about than --all-test-types --test-types reftest print-reftest.

[Manishearth]

oh! i see. can you mention --test-types in the help, then?

[jdm]

Done.

[SimonSapin]

Oh I hadn’t realized. Does --no-default-test-types by itself run nothing? If not, could/should we make it so?

[jdm]

No, --no-default-test-types causes the set of all known test types from wptrunner to be run. It can be thought of as enabling the behaviour that currently exists on master.

[jdm]

Personally, as the person who interacts most with wptrunner internals, I am interested in the following use cases:

• enabling the default behaviour of wptrunner on-demand, but disabled by default because it can break running tests (--no-default-test-types)
• running an explicit subset of test types on a per-run basis, to test specific harness behaviours (--no-default-test-types --test-types foo bar)
• running with a known-good configuration on CI and locally so that nobody else has to think about this (``)

[Manishearth]

r=me

[jdm]

@bors-servo r=manishearth

[bors-servo]

📌 Commit 21b670f has been approved by manishearth

[bors-servo]

⌛ Testing commit 21b670f with merge 17ac381...

[bors-servo]

☀️ Test successful - status-taskcluster
Approved by: manishearth
Pushing 17ac381 to master...