Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Safari Technology Preview to 103 #22475

Merged

Conversation

@autofoolip
Copy link
Contributor

autofoolip commented Mar 26, 2020

Version as reported by safaridriver --version:
Included with Safari Technology Preview (Release 103, 15610.1.7.3.2)

Source: https://developer.apple.com/safari/download/
Build: https://dev.azure.com/foolip/safari-technology-preview-updater/_build/results?buildId=1595&view=logs

Version as reported by safaridriver --version:
Included with Safari Technology Preview (Release 103, 15610.1.7.3.2)

Source: https://developer.apple.com/safari/download/
Build: https://dev.azure.com/foolip/safari-technology-preview-updater/_build/results?buildId=1595&view=logs
@foolip
Copy link
Member

foolip commented Mar 26, 2020

@foolip
Copy link
Member

foolip commented Mar 26, 2020

That can be compared with https://wpt.fyi/results/?run_id=448800002, with just two flexbox changes being merged to master in the meantime, #22465 and #22408.

@stephenmcgruer
Copy link
Contributor

stephenmcgruer commented Mar 26, 2020

Diff should be https://wpt.fyi/results/?diff&filter=ADC&run_id=448800002&run_id=462410002

Looks mostly good at a glance; more positives than negatives and no obvious issues being introduced.

@foolip
Copy link
Member

foolip commented Mar 27, 2020

Filtering a bit it looks to me like this release is actually an overall regression. Comparing just STP 102 to 103 there are 61 fixed subtests but 108 regressed subtests.

Looking at Safari-specific failures, there are 32 fixes but 54 new failures.

I don't know what the influence of flakiness here, having the ability to easily filter out historically flaky tests would really help. nudge nudge wink wink

Numerically, most of the regressions are in html and webdriver.

@foolip
Copy link
Member

foolip commented Mar 27, 2020

I've filed https://bugs.webkit.org/show_bug.cgi?id=209646 for the webdriver regressions.

@stephenmcgruer can you have a look at what happened in HTML? I'll go ahead and merge this since there's nothing to suggest the results are incorrect. Having more complete runs should also confirm the regressions and rule out flakiness as the cause.

@foolip
Copy link
Member

foolip commented Mar 27, 2020

@svillar the tools/ unittests are failing on macOS and Windows with Python 3.8 in this PR. Can you check if they're already failing on master? I'm going to merge anyway since I strongly suspect they just aren't in a good state, and this PR is not the cause of it anyway.

@foolip
foolip approved these changes Mar 27, 2020
@foolip foolip merged commit e0ed4e0 into web-platform-tests:master Mar 27, 2020
33 of 35 checks passed
33 of 35 checks passed
build-and-publish
Details
build-and-tag
Details
Azure Pipelines (tools/ unittests: Windows (Python 3.8)) tools/ unittests: Windows (Python 3.8) failed
Details
Azure Pipelines (tools/ unittests: macOS (Python 3.8)) tools/ unittests: macOS (Python 3.8) failed
Details
Azure Pipelines Build #20200326.70 succeeded
Details
Azure Pipelines (./wpt test-jobs) ./wpt test-jobs succeeded
Details
Azure Pipelines (affected tests without changes: Safari Technology Preview) affected tests without changes: Safari Technology Preview succeeded
Details
Azure Pipelines (affected tests: Safari Technology Preview) affected tests: Safari Technology Preview succeeded
Details
Azure Pipelines (all tests: Safari Technology Preview 1) all tests: Safari Technology Preview 1 succeeded
Details
Azure Pipelines (all tests: Safari Technology Preview 2) all tests: Safari Technology Preview 2 succeeded
Details
Azure Pipelines (all tests: Safari Technology Preview 3) all tests: Safari Technology Preview 3 succeeded
Details
Azure Pipelines (all tests: Safari Technology Preview 4) all tests: Safari Technology Preview 4 succeeded
Details
Azure Pipelines (all tests: Safari Technology Preview 5) all tests: Safari Technology Preview 5 succeeded
Details
Azure Pipelines (infrastructure/ tests: macOS) infrastructure/ tests: macOS succeeded
Details
Azure Pipelines (tools/ unittests: Windows (Python 3.6)) tools/ unittests: Windows (Python 3.6) succeeded
Details
Azure Pipelines (tools/ unittests: Windows) tools/ unittests: Windows succeeded
Details
Azure Pipelines (tools/ unittests: macOS (Python 3.6)) tools/ unittests: macOS (Python 3.6) succeeded
Details
Azure Pipelines (tools/ unittests: macOS) tools/ unittests: macOS succeeded
Details
Azure Pipelines (tools/wpt/ tests: Windows (Python 3.6)) tools/wpt/ tests: Windows (Python 3.6) succeeded
Details
Azure Pipelines (tools/wpt/ tests: Windows (Python 3.8)) tools/wpt/ tests: Windows (Python 3.8) succeeded
Details
Azure Pipelines (tools/wpt/ tests: Windows) tools/wpt/ tests: Windows succeeded
Details
Azure Pipelines (tools/wpt/ tests: macOS (Python 3.6)) tools/wpt/ tests: macOS (Python 3.6) succeeded
Details
Azure Pipelines (tools/wpt/ tests: macOS (Python 3.8)) tools/wpt/ tests: macOS (Python 3.8) succeeded
Details
Azure Pipelines (tools/wpt/ tests: macOS) tools/wpt/ tests: macOS succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: Windows (Python 3.6)) tools/wptrunner/ unittests: Windows (Python 3.6) succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: Windows (Python 3.8)) tools/wptrunner/ unittests: Windows (Python 3.8) succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: Windows) tools/wptrunner/ unittests: Windows succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: macOS (Python 3.6)) tools/wptrunner/ unittests: macOS (Python 3.6) succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: macOS (Python 3.8)) tools/wptrunner/ unittests: macOS (Python 3.8) succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: macOS) tools/wptrunner/ unittests: macOS succeeded
Details
Azure Pipelines (wpt.fyi hook: safari-preview-affected-tests) wpt.fyi hook: safari-preview-affected-tests succeeded
Details
Azure Pipelines (wpt.fyi hook: safari-preview-affected-tests-without-changes) wpt.fyi hook: safari-preview-affected-tests-without-changes succeeded
Details
Azure Pipelines (wpt.fyi hook: safari-preview-results) wpt.fyi hook: safari-preview-results succeeded
Details
Community-TC (pull_request) TaskGroup: success
Details
wpt.fyi - safari[experimental] Safari results
Details
@foolip foolip deleted the autofoolip:safari-technology-preview-103 branch Mar 27, 2020
@foolip
Copy link
Member

foolip commented Mar 27, 2020

This landed right on top of the current epochs/three_hourly and I triggered another full run. When both are done the results from commits a11782a and e0ed4e0 can be compared to verify the regressions.

@stephenmcgruer
Copy link
Contributor

stephenmcgruer commented Mar 27, 2020

https://wpt.fyi/results/?diff&filter=ADC&run_id=439350003&run_id=435310001 is the diff from those two SHAs.

Diffing that against the previous comparison shows:

  • The webdriver/ failures seem flaky; the only overlap from the first run and the second run is get_active_element/user_prompts.py.

  • The regressions in html/semantics/embedded-content are consistent. html/semantics/embedded-content/media-elements/track/track-element/track-cue-rendering-after-controls-removed.html, is existing flake. html/semantics/embedded-content/the-img-element/sizes/parse-a-sizes-attribute-display-none.html appears to be a new regression (no previously reported flakes).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.