Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[meta] Servo support #106

Open
2 tasks
sagudev opened this issue May 30, 2024 · 2 comments
Open
2 tasks

[meta] Servo support #106

sagudev opened this issue May 30, 2024 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@sagudev
Copy link
Collaborator

sagudev commented May 30, 2024

Basic support for servo has landed in #92, but there are still some things we need before servo can move off #80 :

  • Allow test outcome to be PASS/FAIL (see Errors on parsing valid metadata #79)
  • Use only first line of wpt report (servo runs wpt twice, second time only on unexpected results to detect flakes but they are logged in second wpt report in same json file; ideally we would reconcile it)

Also, when developing new webgpu features on servo I found useful to have (773f118) to only set new good expectations, or 039200b to set only those that are reported (if we do partial run of WPT tests)

@sagudev sagudev self-assigned this May 30, 2024
@ErichDonGubler
Copy link
Owner

I have yet to really go through each of the points in the OP, but: Partial runs are actually the motivation behind #25.

@ErichDonGubler ErichDonGubler added the enhancement New feature or request label May 30, 2024
@ErichDonGubler
Copy link
Owner

Thinking things through more, @sagudev, you might be interested in the workflow that we're undertaking at Mozilla. We recently added the implementation-status: backlog property to all test cases (see upstream docs on metadata), and have incrementally removed it as we've determined tests are worth blocking on in CI. We now use both tier 2 and tier 3 of Firefox CI.

I'm currently calling the migration of a test from a less stable tier to a more stable tier by removing implementation-status: backlog a "promotion". We do experiments in promoting tests according to heuristics like "promote permanently PASSing tests" (already implemented), "promote tests that aren't observed to FAIL or CRASH" (to be implemented in #109), with more to come. The workflow then becomes:

  1. Run tests, and gather wptreport.json files.
  2. Run update-expected as desired for backlogged tests, and create a commit.
  3. Run update-backlog to tentatively promote tests. Commit and submit to CI as an experiment.
  4. Check tentatively promoted tests are successful in the above experiment. Where they are not, demote them, preferably with bugs in Bugzilla.

We consider using implementation-status: backlog to be valuable because:

  • wptrunner has CLI support for filtering tests by implementation status, so the lift to adjust our CI was light.
  • It's a clear signal that a test is not yet considered valuable to run as a blocker in CI yet.
  • It's orthogonal to expected; sometimes, we want to model that a test should pass, but not block CI on it yet. This came up recently with bug 1897131.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants