Start running webdriver tests #13261

jdm · 2016-09-13T23:20:49Z

This branch includes the patches necessary to start running webdriver tests consistently.

./mach build -d does not report any errors
./mach test-tidy does not report any errors
There are tests for these changes

This change is


        Rename mach command to upgrade wptrunner and make it work more than o…

…nce.


        Avoid #12321 by creating environment before switching directories.


        Upgrade wptrunner to e7e40be.


        Do not treat a prefix of / as a real URL prefix.


        Add alert.dismiss command to webdriver session.


        Implement webdriver stubs for window switching and alert dismissing.


        Run webdriver tests.

highfive · 2016-09-13T23:20:52Z

Heads up! This PR modifies the following files:

@asajeffrey: components/webdriver_server/lib.rs
@wafflespeanut: python/servo/devenv_commands.py
@jgraham: tests/wpt/harness/wptrunner/wpttest.py, tests/wpt/harness/wptrunner/wptmanifest/tests/test_serializer.py, tests/wpt/include.ini, tests/wpt/harness/wptrunner/browsers/servo.py, tests/wpt/harness/wptrunner/webdriver_server.py, tests/wpt/harness/wptrunner/wptcommandline.py, tests/wpt/harness/wptrunner/executors/executormarionette.py, tests/wpt/harness/wptrunner/tests/test_chunker.py, tests/wpt/harness/wptrunner/executors/pytestrunner/fixtures.py, tests/wpt/harness/wptrunner/executors/executorservo.py, tests/wpt/harness/wptrunner/wptrunner.py, tests/wpt/harness/wptrunner/browsers/firefox.py, tests/wpt/harness/wptrunner/browsers/init.py, tests/wpt/harness/wptrunner/tests/test_update.py, tests/wpt/harness/wptrunner/executors/executorservodriver.py, tests/wpt/harness/wptrunner/executors/webdriver.py, tests/wpt/harness/wptrunner/executors/executorselenium.py, components/webdriver_server/lib.rs, tests/wpt/harness/wptrunner/executors/testharness_webdriver.js, tests/wpt/harness/wptrunner/testrunner.py, tests/wpt/harness/wptrunner/tests/test_hosts.py

jdm · 2016-09-13T23:20:55Z

@bors-servo: try

bors-servo · 2016-09-13T23:20:58Z

⌛ Trying commit 739f0d2 with merge 7e55e86...


        Auto merge of #13261 - jdm:wptrunnerup2, r=<try>

Start running webdriver tests This branch includes the patches necessary to start running webdriver tests consistently. --- - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] There are tests for these changes

jdm · 2016-09-13T23:21:32Z

3c04711, c73ecf7, cdfe796, and 2449bc7 will need to be upstreamed.

bors-servo · 2016-09-14T00:29:39Z

💔 Test failed - linux-rel


        Start webdriver session after creating it.

jdm · 2016-09-14T01:10:02Z

@bors-servo: try

bors-servo · 2016-09-14T01:10:05Z

⌛ Trying commit cdfe796 with merge c06d7f2...


        Auto merge of #13261 - jdm:wptrunnerup2, r=<try>

Start running webdriver tests This branch includes the patches necessary to start running webdriver tests consistently. --- - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] There are tests for these changes  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/13261)

bors-servo · 2016-09-14T02:03:58Z

💔 Test failed - linux-rel


        Replace uses of Session.window_handle[s] with Session.handle[s].

jdm · 2016-09-14T03:42:02Z

@bors-servo: try

bors-servo · 2016-09-14T03:42:05Z

⌛ Trying commit 2449bc7 with merge f653ef8...


        Auto merge of #13261 - jdm:wptrunnerup2, r=<try>

Start running webdriver tests This branch includes the patches necessary to start running webdriver tests consistently. --- - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] There are tests for these changes  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/13261)

bors-servo · 2016-09-14T04:31:06Z

💔 Test failed - linux-rel

jdm · 2016-09-14T14:20:00Z

I fear that the webdriver tests don't deal well with concurrent processes. I'm not sure how to reduce the number of processes used only for that test type, however. Any ideas, @jgraham?

jgraham · 2016-09-14T15:42:36Z

Run them using a separate command on buildbot which limits the test type using --test-type wdspec and process count using --processes=1? The existing invocation will also need to select the test types it wants to run rather than doing everything.

jdm · 2016-09-15T19:38:17Z

Bah, that's ~~impossible~~difficult to test until we finish converting our buildbot instances to use in-tree test configurations :(

jdm · 2016-09-15T19:39:13Z

Specifically, blocked on servo/saltfs#316.

bors-servo · 2016-10-18T14:05:02Z

☔ The latest upstream changes (presumably #13614) made this pull request unmergeable. Please resolve the merge conflicts.

aneeshusa · 2016-10-19T21:41:45Z

The final PR for servo/saltfs#316 has been merged and deployed, so this is no longer blocked.

cbrewster · 2016-10-29T23:27:48Z

r? @jgraham or someone else

jgraham · 2016-11-02T10:52:15Z

Remember to upstream changes to wptrunner.

Reviewed 1 of 1 files at r1, 1 of 1 files at r2, 16 of 18 files at r3, 1 of 1 files at r5, 1 of 1 files at r6, 1 of 1 files at r7, 1 of 1 files at r8, 1 of 1 files at r9.
Review status: all files reviewed at latest revision, 3 unresolved discussions, some commit checks failed.

components/webdriver_server/lib.rs, line 509 at r9 (raw file):

    fn handle_dismiss_alert(&mut self) -> WebDriverResult<WebDriverResponse> {
        //TODO figure out how to interact with blocking native UI
        Ok(WebDriverResponse::Void)

Can't we return unsupported operation for now?

tests/wpt/harness/wptrunner/executors/executorservo.py, line 295 at r9 (raw file):

    def setup(self, runner):
        try:
            self.server = ServoDriverServer(self.logger, binary=self.browser.binary, binary_args=self.browser.binary_args, render_backend=self.browser.render_backend)

Some more linebreaks here would be nice.

tests/wpt/harness/wptrunner/executors/executorservo.py, line 334 at r9 (raw file):

class ServoWdspecExecutor(WdspecExecutor):

Not necessarily a blocker, but this seems to share all of the code with MarionetteWdSpecExecutor except the protocol. Maybe we can construct a base class that is just parameterized by the protocol class.

Comments from Reviewable

jdm · 2017-01-27T19:35:43Z

Tracking this work in #15274.

jdm added 7 commits Jul 4, 2016

Rename mach command to upgrade wptrunner and make it work more than o…

b7f6d17

…nce.

Avoid #12321 by creating environment before switching directories.

ee454f8

Upgrade wptrunner to e7e40be.

3245293

Do not treat a prefix of / as a real URL prefix.

3c04711

Add alert.dismiss command to webdriver session.

c73ecf7

Implement webdriver stubs for window switching and alert dismissing.

966b43b

Run webdriver tests.

Loading status checks…

739f0d2

highfive assigned cbrewster Sep 13, 2016

highfive added the S-awaiting-review label Sep 13, 2016

highfive added the S-tests-failed label Sep 14, 2016

Start webdriver session after creating it.

Loading status checks…

cdfe796

highfive removed the S-tests-failed label Sep 14, 2016

highfive added the S-tests-failed label Sep 14, 2016

Replace uses of Session.window_handle[s] with Session.handle[s].

Loading status checks…

2449bc7

highfive removed the S-tests-failed label Sep 14, 2016

highfive added the S-tests-failed label Sep 14, 2016

jdm added S-blocked-on-external and removed S-awaiting-review labels Sep 15, 2016

jdm mentioned this pull request Sep 26, 2016

Add OSMesa headless mode, run WPT against Webrender. #13382

Merged

highfive added the S-needs-rebase label Oct 18, 2016

aneeshusa removed the S-blocked-on-external label Oct 19, 2016

highfive assigned jgraham and unassigned cbrewster Oct 29, 2016

jdm mentioned this pull request Jan 27, 2017

Run automated webdriver tests #15274

Open

jdm closed this Jan 27, 2017

servo / servo

Start running webdriver tests #13261

Start running webdriver tests #13261

jdm commented Sep 13, 2016 •

edited by larsbergstrom

highfive commented Sep 13, 2016

jdm commented Sep 13, 2016

bors-servo commented Sep 13, 2016

jdm commented Sep 13, 2016 •

edited

bors-servo commented Sep 14, 2016

jdm commented Sep 14, 2016

bors-servo commented Sep 14, 2016

bors-servo commented Sep 14, 2016

jdm commented Sep 14, 2016

bors-servo commented Sep 14, 2016

bors-servo commented Sep 14, 2016

jdm commented Sep 14, 2016

jgraham commented Sep 14, 2016

jdm commented Sep 15, 2016 •

edited

jdm commented Sep 15, 2016

bors-servo commented Oct 18, 2016

aneeshusa commented Oct 19, 2016

cbrewster commented Oct 29, 2016

jgraham commented Nov 2, 2016

jdm commented Jan 27, 2017

servo / servo

Join GitHub today

Start running webdriver tests #13261

Start running webdriver tests #13261

Conversation

jdm commented Sep 13, 2016 • edited by larsbergstrom

highfive commented Sep 13, 2016

jdm commented Sep 13, 2016

bors-servo commented Sep 13, 2016

jdm commented Sep 13, 2016 • edited

bors-servo commented Sep 14, 2016

jdm commented Sep 14, 2016

bors-servo commented Sep 14, 2016

bors-servo commented Sep 14, 2016

jdm commented Sep 14, 2016

bors-servo commented Sep 14, 2016

bors-servo commented Sep 14, 2016

jdm commented Sep 14, 2016

jgraham commented Sep 14, 2016

jdm commented Sep 15, 2016 • edited

jdm commented Sep 15, 2016

bors-servo commented Oct 18, 2016

aneeshusa commented Oct 19, 2016

cbrewster commented Oct 29, 2016

jgraham commented Nov 2, 2016

jdm commented Jan 27, 2017

jdm commented Sep 13, 2016 •

edited by larsbergstrom

jdm commented Sep 13, 2016 •

edited

jdm commented Sep 15, 2016 •

edited