Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Origin policy tests not running on wpt.fyi #22113

Closed
domenic opened this issue Mar 6, 2020 · 5 comments
Closed

Origin policy tests not running on wpt.fyi #22113

domenic opened this issue Mar 6, 2020 · 5 comments

Comments

@domenic
Copy link
Member

@domenic domenic commented Mar 6, 2020

If you visit https://wpt.fyi/results/origin-policy/content-security/valid-with-semicolon.https.html?label=master&label=experimental&aligned, you see two tests: "Prerequisite check: running on HTTPS" and "Test setup of the iframe".

If you visit https://wpt.live/origin-policy/content-security/valid-with-semicolon.https.html, you see four tests; the two additional ones are "eval must be disallowed" and "img loading must be disallowed". Those are the actual things being tested.

This leads to the unfortunate result of the dashboard being useless for origin policy: https://wpt.fyi/results/origin-policy?label=master&label=experimental&aligned shows everyone passing everything, basically.

@Hexcles
Copy link
Member

@Hexcles Hexcles commented Mar 9, 2020

I can reproduce locally

./wpt run --binary `which google-chrome-unstable` --log-mach - chrome origin-policy/content-security/valid-with-semicolon.https.html

only runs two tests and I don't see any console errors in Chrome.

@stephenmcgruer
Copy link
Contributor

@stephenmcgruer stephenmcgruer commented Mar 16, 2020

But it does work under ./wpt serve - loading even the non-https version in a browser after serving still shows all 4 subtests.

No further understandings yet; I thought the problem was that 'getmessages' was being sent to the iframe before it registered its listener for it, but that also happens for the working ./wpt serve version.

@domenic
Copy link
Member Author

@domenic domenic commented Mar 16, 2020

Could this be related to 644a206#diff-6f05ece846271f4b15119cfe565da64dR72-R96 (omitting the origin policy subdomains from the web driver environment variables)? Although if that's it, then there's probably a separate bug (in testharness.js?) where these tests should be marked as failing instead of passing, if the calls to fetch_tests_from_window don't work.

@stephenmcgruer
Copy link
Contributor

@stephenmcgruer stephenmcgruer commented Mar 16, 2020

I don't think its related to that; that should really only affect the webdriver tests.

I've gone very far down the rabbithole here, and I'm not making progress. For some reason, when using wpt run the main frame isn't getting any message_functions messages (e.g. test start, state change, etc) from the iframe. It looks like the tests are running, I just can't sort out the ordering to figure out what is different between the two cases that is causing messages not to happen :(.

One big difference is that wptrunner injects its own setup and config changes, but I don't know how to tell if that is all working as expected or not.

cc @jgraham, in case he has magical knowledge about the workings of RemoteContext/RemoteTest and might know why its working here under wpt serve but not wpt run.

@stephenmcgruer stephenmcgruer self-assigned this Mar 16, 2020
@stephenmcgruer
Copy link
Contributor

@stephenmcgruer stephenmcgruer commented Mar 16, 2020

Argh, I had figured it hours ago but had failed to realize that!

The problem is with the testharnessreport.js injected by wptrunner. It re-calls setup on the test environment, passing a filtered list of message_events:

  var props = {output: %(output)d,
               timeout_multiplier: %(timeout_multiplier)s,
               explicit_timeout: %(explicit_timeout)s,
    message_events: ["completion"]};

This causes us to remove our previously-added callbacks for start, test_state, and results, which then means that the iframe doesn't dispatch any messages about them to the main parent window, so the main parent doesn't know about them when reporting results.

The reason I had overlooked it is that if you're using wpt run but just refreshing the test in the browser, you don't get a new version of testharnessreport.js when you refresh (I think). You have to re-run wpt run entirely.

As to a fix, I'm not immediately sure. Adding the other events back into message_events appears to work, but may have unintended consequences.

stephenmcgruer added a commit that referenced this issue Mar 24, 2020
…22284)

Including the testharnessreport.js here prevents the subframe tests from being reported to the main frame, which means wpt run misses them. Removing it shows a bunch of previously unreported tests.

Fixes #22113
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants