Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: new typescript WPT runner #9269

Merged
merged 25 commits into from
Jan 27, 2021

Conversation

lucacasonato
Copy link
Member

@lucacasonato lucacasonato commented Jan 25, 2021

This is a new WPT runner, written in TypeScript. It is a lot more advanced than the integration test for WPT we had previously.

  1. It can auto setup /etc/hosts.
  2. It can run only a subset of tests.
  3. It can automatically update test expectations based on reality.
  4. It can output JSON for consumption by deno.land (not yet integrated).
  5. A lot easier to extend in the future.
  6. Uses HTML WPT tests (from the WPT test util) and parses out scripts using a WASM based DOM parser, instead of trying to do all of the resolving of meta comments ourselves.

TODO:

  • automatically acquire manifest.json
  • run filtering: wpt.ts run -- encoding
  • wpt.ts setup to auto configure /etc/hosts file and check correct python version
  • wpt.ts update to update the expectation.json based on currently passing tests
  • filtering for wpt.ts update
  • move tools/wpt/readme.md to manual, and expand

@lucacasonato
Copy link
Member Author

@kitsonk @bartlomieju @ry @kt3k @piscisaureus @bnoordhuis Please try this out locally:

./tools/wpt.ts setup
./tools/wpt.ts run --quiet

I want to make sure it actually works for everyone before proceeding.

docs/contributing/web_platform_tests.md Outdated Show resolved Hide resolved
docs/contributing/web_platform_tests.md Outdated Show resolved Hide resolved
docs/contributing/web_platform_tests.md Outdated Show resolved Hide resolved
docs/contributing/web_platform_tests.md Outdated Show resolved Hide resolved
tools/wpt.ts Outdated Show resolved Hide resolved
tools/wpt.ts Outdated Show resolved Hide resolved
tools/wpt/runner.ts Outdated Show resolved Hide resolved
tools/wpt/runner.ts Show resolved Hide resolved
tools/wpt/utils.ts Outdated Show resolved Hide resolved
tools/wpt/utils.ts Show resolved Hide resolved
@caspervonb

This comment has been minimized.


### FAQ

#### Upgrading the wpt submodule:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this section necessary? test_util/wpt in .gitmodules seems still pointing to the upstream.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not, but I think it doesn't hurt to be clear about it.

@kt3k
Copy link
Member

kt3k commented Jan 27, 2021

sudo ./tools/wpt.ts setup worked great without prompt issues, and ./tools/wpt.ts run worked and passed without any problem on my machine! great job!

The instruction and the error messages look great. I hope this addresses the issues concerning new contributors.

@ry
Copy link
Member

ry commented Jan 27, 2021

Nit: if I haven't compiled Deno yet, I get an error which is difficult to understand:

~/src/deno> ./tools/wpt.ts run --quiet
Going to run 186 test files.
----------------------------------------
/WebCryptoAPI/getRandomValues.any.html

error: Uncaught (in promise) NotFound: No such file or directory (os error 2)
    throw new ErrorClass(res.err.message);
          ^
    at processResponse (deno:core/core.js:212:11)
    at Object.jsonOpSync (deno:core/core.js:235:12)
    at opRun (deno:runtime/js/40_process.js:19:17)
    at Object.run (deno:runtime/js/40_process.js:103:17)
    at runSingleTest (runner.ts:70:21)
    at async wpt.ts:154:22
    at async runWithTestUtil (runner.ts:35:12)
    at async run (wpt.ts:149:19)
    at async wpt.ts:48:5

Copy link
Member

@ry ry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me.

@lucacasonato
Copy link
Member Author

@ry Addressed this by automatically invoking cargo build before starting the tests now.

@lucacasonato lucacasonato merged commit 2638aa0 into denoland:master Jan 27, 2021
@lucacasonato lucacasonato deleted the ts-wpt-runner branch January 27, 2021 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants