Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unskip testdriver Actions infra tests #19497

Merged

Conversation

@LukeZielinski
Copy link
Contributor

LukeZielinski commented Oct 3, 2019

The tests in infrastructure/testdriver/actions were being skipped due to a Chromedriver bug (https://bugs.chromium.org/p/chromedriver/issues/detail?id=3114). This seems unreproducible with the latest version so I'm re-enabling these tests.

@LukeZielinski

This comment has been minimized.

Copy link
Contributor Author

LukeZielinski commented Oct 4, 2019

Ok so this is interesting. When running Chrome on MacOS normally, the test actionsWithKeyPressed Fails and the next test (elementPosition) passes. If I add a failure expectation for actionsWithKeyPressed then it fails (and is treated as OK) but then elementPosition times out.

Logs for state 1: No expectation, actionsWithKeyPressed fails unexpected and elementPosition passes.

TEST-START | /infrastructure/testdriver/actions/actionsWithKeyPressed.html
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
TEST-UNEXPECTED-FAIL | /infrastructure/testdriver/actions/actionsWithKeyPressed.html | TestDriver actions: actions with key pressed - assert_array_equals: lengths differ, expected 3 got 1
    at Test. (http://web-platform.test:8000/infrastructure/testdriver/actions/actionsWithKeyPressed.html:64:34)
    at Test.step (http://web-platform.test:8000/resources/testharness.js:1908:25)
    at http://web-platform.test:8000/resources/testharness.js:1948:32
TEST-OK | /infrastructure/testdriver/actions/actionsWithKeyPressed.html | took 2621ms
Closing logging queue
queue closed
PID 1567 | Starting ChromeDriver 79.0.3921.0 (15bace96a14b174529914de372e04fd14c78eac7-refs/heads/master@{#698801}) on port 49748
PID 1567 | Only local connections are allowed.
PID 1567 | Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Starting runner
PID 1567 | [1570131757.641][WARNING]: You are using an unsupported command-line switch: --disable-build-check. Please don't report bugs that cannot be reproduced with this switch removed.
TEST-START | /infrastructure/testdriver/actions/elementPosition.html
TEST-PASS | /infrastructure/testdriver/actions/elementPosition.html | TestDriver actions: element position 
TEST-OK | /infrastructure/testdriver/actions/elementPosition.html | took 1011ms

Logs for state 2: Failure expectation for actionsWithKeyPressed and it fails as expected, but elementPosition times out.

TEST-START | /infrastructure/testdriver/actions/actionsWithKeyPressed.html
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
STDERR: No handlers could be found for logger "mod_pywebsocket.standalone.WebSocketServer"
TEST-FAIL | /infrastructure/testdriver/actions/actionsWithKeyPressed.html | TestDriver actions: actions with key pressed - assert_array_equals: lengths differ, expected 3 got 1
    at Test. (http://web-platform.test:8000/infrastructure/testdriver/actions/actionsWithKeyPressed.html:64:34)
    at Test.step (http://web-platform.test:8000/resources/testharness.js:1908:25)
    at http://web-platform.test:8000/resources/testharness.js:1948:32
TEST-OK | /infrastructure/testdriver/actions/actionsWithKeyPressed.html | took 2791ms
TEST-START | /infrastructure/testdriver/actions/elementPosition.html
TEST-UNEXPECTED-TIMEOUT | /infrastructure/testdriver/actions/elementPosition.html | Waiting on browser:
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 778, in __bootstrap
    raise
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 855, in __bootstrap_inner
    pass
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 758, in run
    del self.__target, self.__args, self.__kwargs
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 312, in _run
    self.result_flag.set()
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 355, in do_testharness
    parent_window = protocol.testharness.close_old_windows()
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 97, in close_old_windows
    pass
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/webdriver/webdriver/client.py", line 20, in inner
    return func(self, *args, **kwargs)
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/webdriver/webdriver/client.py", line 576, in close
    handles = self.send_session_command("DELETE", "window")
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/webdriver/webdriver/client.py", line 516, in send_session_command
    return self.send_command(method, url, body)
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/webdriver/webdriver/client.py", line 480, in send_command
    raise err
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/webdriver/webdriver/transport.py", line 176, in send
    return Response.from_http(response, decoder=decoder, **codec_kwargs)
  File "/Users/vsts/agent/2.158.0/work/1/s/tools/webdriver/webdriver/transport.py", line 191, in _request
    return self.connection.getresponse()
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1132, in getresponse
    return response
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 515, in begin
    self.will_close = 1
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 429, in _read_status
    return version, status, reason
  File "/usr/local/Cellar/python@2/2.7.16_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
TEST-INFO took 20040ms
@jgraham

This comment has been minimized.

Copy link
Contributor

jgraham commented Oct 4, 2019

Bet you end up with a context menu open or something and that interfers with the next test.

@LukeZielinski

This comment has been minimized.

Copy link
Contributor Author

LukeZielinski commented Oct 4, 2019

@jgraham Sure seems that way - the next test times out waiting for close_old_windows. Sounds like you may have run into this before, any advice?

@jgraham

This comment has been minimized.

Copy link
Contributor

jgraham commented Oct 4, 2019

Nothing specific, I just remembered it was doing something weird on mac. I can't find a bug to suggest that we had to fix it specially, so I don't know I have anything to offer other than "either leave it disabled on mac or fix chromedriver to close context menus that would interfere with subsequent commands". The latter of which is possibly lots of work; I don't know.

@LukeZielinski

This comment has been minimized.

Copy link
Contributor Author

LukeZielinski commented Oct 4, 2019

One more bit of context - the reason this behaviour is manifesting when the failure expectation is added seems to be related to whether we restart chromedriver in between these tests.

In State 1 (No expectation, actionsWithKeyPressed fails unexpected and elementPosition passes) Chromedriver is restarted between the two tests because the first failure is unexpected.

In State 2 (Failure expectation for actionsWithKeyPressed and it fails as expected, but elementPosition times out) the first failure is expected so Chromedriver is not restarted, thus causing the timeout in the 2nd test.

@jugglinmike

This comment has been minimized.

Copy link
Contributor

jugglinmike commented Oct 4, 2019

The WPT CLI's --no-restart-on-unexepected option might help with debugging

@LukeZielinski

This comment has been minimized.

Copy link
Contributor Author

LukeZielinski commented Oct 7, 2019

Suspicion confirmed - one of the tests sends a Ctrl+Click action which opens a context menu on Mac. Test will be updated.

@LukeZielinski LukeZielinski force-pushed the LukeZielinski:unskip-actions-infra-test branch from 8661ef1 to cf29fd0 Oct 7, 2019
@LukeZielinski LukeZielinski force-pushed the LukeZielinski:unskip-actions-infra-test branch from cf29fd0 to c0b9e50 Oct 9, 2019
@LukeZielinski LukeZielinski assigned foolip and unassigned gsnedders Oct 9, 2019
@LukeZielinski

This comment has been minimized.

Copy link
Contributor Author

LukeZielinski commented Oct 9, 2019

The offending test has been fixed (crrev.com/c/1845994), this PR starts running at again. @foolip: PTAL?

@foolip
foolip approved these changes Oct 9, 2019
@foolip

This comment has been minimized.

Copy link
Contributor

foolip commented Oct 9, 2019

Sweet, thanks @LukeZielinski!

@foolip foolip merged commit 8f21bdd into web-platform-tests:master Oct 9, 2019
11 checks passed
11 checks passed
Azure Pipelines Build #20191009.103 succeeded
Details
Azure Pipelines (./wpt test-jobs) ./wpt test-jobs succeeded
Details
Azure Pipelines (affected tests without changes: Safari Technology Preview) affected tests without changes: Safari Technology Preview succeeded
Details
Azure Pipelines (affected tests: Safari Technology Preview) affected tests: Safari Technology Preview succeeded
Details
Azure Pipelines (infrastructure/ tests: macOS) infrastructure/ tests: macOS succeeded
Details
Azure Pipelines (wpt.fyi hook: safari-preview-affected-tests) wpt.fyi hook: safari-preview-affected-tests succeeded
Details
Azure Pipelines (wpt.fyi hook: safari-preview-affected-tests-without-changes) wpt.fyi hook: safari-preview-affected-tests-without-changes succeeded
Details
Taskcluster (pull_request) TaskGroup: success
Details
wpt.fyi - chrome[experimental] Chrome results
Details
wpt.fyi - firefox[experimental] Firefox results
Details
wpt.fyi - safari[experimental] Safari results
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.