Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reland "[webdriver] Close old windows at the end of each test as well as beginning" #25567

Merged
merged 5 commits into from Sep 18, 2020

Conversation

@stephenmcgruer
Copy link
Contributor

@stephenmcgruer stephenmcgruer commented Sep 16, 2020

This is a reland of #24879 (which was reverted in #25544). It updates the code
to only close the windows if the user hasn't asked us to pause after a test has run.

From the original PR:


Previously, we closed old windows at the start of each test. This was nice in terms
of defensiveness (don't assume the last run left the world in a good state), but made
it hard to find problematic tests that left dialogs open (since they wouldn't throw until
the next test).

Instead, this patch does both - close both at the start and end of a test. This should
improve the blaming situation, whilst still being defensive.

There are potential performance implications to this patch, however test runs are
inconclusive. Full runs of Chrome and Safari show +- 2%, which is possibly within
margin of error. Running locally, some directories showed a ~2% slowdown, whilst
others had little or no difference.

@stephenmcgruer
Copy link
Contributor Author

@stephenmcgruer stephenmcgruer commented Sep 16, 2020

@jgraham @gsnedders - PTAL. I'm not confident in re-using close_after_done here - it is only used today in marionette (despite being set all over the place...) and I don't quite understand what it is meant to be for. (It sounds related...)

I'm also not confident in my methodology for deciding when we can and can't close windows, there's quite likely something I've missed :D

@stephenmcgruer
Copy link
Contributor Author

@stephenmcgruer stephenmcgruer commented Sep 16, 2020

Capturing IRC comments so I don't lose them.

smcgruer_[EST]: So close_after_done originally was supposed to be about running each test in a clean browsing context (i.e. new tab) vs navigating a single tab
10:04 AM It wasn't really supposed to be about cleaning up other windows
10:04 AM <smcgruer_[EST]> Stephen McGruer Ah, ok. I'm happy to use a new flag then
10:05 AM "may_pause_after_test"
10:05 AM Or if we want to be commanding: "close_windows_after_test"

jgraham also suggested just getting rid of close_after_done since it was never really utilized, but I'd like to do that cleanup separately from this change.

@stephenmcgruer
Copy link
Contributor Author

@stephenmcgruer stephenmcgruer commented Sep 17, 2020

@jgraham @gsnedders - I've switched to may_pause_after_test, PTAL. (Still interested in thoughts on where, if anywhere, to move the added logic in tools/wptrunner/wptrunner/wptrunner.py to)

@wpt-pr-bot wpt-pr-bot temporarily deployed to wpt-preview-25567 Sep 17, 2020 Inactive
tools/wptrunner/wptrunner/wptrunner.py Outdated Show resolved Hide resolved
@stephenmcgruer stephenmcgruer requested a review from jgraham Sep 17, 2020
@wpt-pr-bot wpt-pr-bot temporarily deployed to wpt-preview-25567 Sep 17, 2020 Inactive
@stephenmcgruer stephenmcgruer merged commit ed3c2cd into master Sep 18, 2020
50 checks passed
50 checks passed
build-and-publish
Details
build-and-tag
Details
detect-deployment
Details
Azure Pipelines Build #20200918.82 succeeded
Details
Azure Pipelines (./wpt test-jobs) ./wpt test-jobs succeeded
Details
Azure Pipelines (affected tests without changes: Safari Technology Preview) affected tests without changes: Safari Technology Preview succeeded
Details
Azure Pipelines (affected tests: Safari Technology Preview) affected tests: Safari Technology Preview succeeded
Details
Azure Pipelines (infrastructure/ tests: macOS) infrastructure/ tests: macOS succeeded
Details
Azure Pipelines (tools/ unittests: Windows + Python 3.6) tools/ unittests: Windows + Python 3.6 succeeded
Details
Azure Pipelines (tools/ unittests: Windows + Python 3.8) tools/ unittests: Windows + Python 3.8 succeeded
Details
Azure Pipelines (tools/ unittests: Windows) tools/ unittests: Windows succeeded
Details
Azure Pipelines (tools/ unittests: macOS + Python 3.6) tools/ unittests: macOS + Python 3.6 succeeded
Details
Azure Pipelines (tools/ unittests: macOS + Python 3.8) tools/ unittests: macOS + Python 3.8 succeeded
Details
Azure Pipelines (tools/ unittests: macOS) tools/ unittests: macOS succeeded
Details
Azure Pipelines (tools/wpt/ tests: Windows + Python 3.6) tools/wpt/ tests: Windows + Python 3.6 succeeded
Details
Azure Pipelines (tools/wpt/ tests: Windows + Python 3.8) tools/wpt/ tests: Windows + Python 3.8 succeeded
Details
Azure Pipelines (tools/wpt/ tests: Windows) tools/wpt/ tests: Windows succeeded
Details
Azure Pipelines (tools/wpt/ tests: macOS + Python 3.6) tools/wpt/ tests: macOS + Python 3.6 succeeded
Details
Azure Pipelines (tools/wpt/ tests: macOS + Python 3.8) tools/wpt/ tests: macOS + Python 3.8 succeeded
Details
Azure Pipelines (tools/wpt/ tests: macOS) tools/wpt/ tests: macOS succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: Windows + Python 3.6) tools/wptrunner/ unittests: Windows + Python 3.6 succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: Windows + Python 3.8) tools/wptrunner/ unittests: Windows + Python 3.8 succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: Windows) tools/wptrunner/ unittests: Windows succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: macOS + Python 3.6) tools/wptrunner/ unittests: macOS + Python 3.6 succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: macOS + Python 3.8) tools/wptrunner/ unittests: macOS + Python 3.8 succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: macOS) tools/wptrunner/ unittests: macOS succeeded
Details
Azure Pipelines (wpt.fyi hook: safari-preview-affected-tests) wpt.fyi hook: safari-preview-affected-tests succeeded
Details
Azure Pipelines (wpt.fyi hook: safari-preview-affected-tests-without-changes) wpt.fyi hook: safari-preview-affected-tests-without-changes succeeded
Details
download-firefox-nightly Community-TC (pull_request)
Details
infrastructure/ tests Community-TC (pull_request)
Details
infrastructure/ tests (Python 3) Community-TC (pull_request)
Details
lint Community-TC (pull_request)
Details
resources/ tests Community-TC (pull_request)
Details
sink-task Community-TC (pull_request)
Details
tools/ integration tests (Python 2) Community-TC (pull_request)
Details
tools/ integration tests (Python 3.6) Community-TC (pull_request)
Details
tools/ integration tests (Python 3.8) Community-TC (pull_request)
Details
tools/ unittests (Python 2) Community-TC (pull_request)
Details
tools/ unittests (Python 3.6) Community-TC (pull_request)
Details
tools/ unittests (Python 3.8) Community-TC (pull_request)
Details
wpt-chrome-dev-results Community-TC (pull_request)
Details
wpt-chrome-dev-results-without-changes Community-TC (pull_request)
Details
wpt-chrome-dev-stability Community-TC (pull_request)
Details
wpt-decision-task Community-TC (pull_request)
Details
wpt-firefox-nightly-results Community-TC (pull_request)
Details
wpt-firefox-nightly-results-without-changes Community-TC (pull_request)
Details
wpt-firefox-nightly-stability Community-TC (pull_request)
Details
wpt.fyi - chrome[experimental] Chrome results
Details
wpt.fyi - firefox[experimental] Firefox results
Details
wpt.fyi - safari[experimental] Safari results
Details
@stephenmcgruer stephenmcgruer deleted the smcgruer/reland-window-closing branch Sep 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.