Support finding EWS runs from GitHub PRs in update-test-expectations #18037

gsnedders · 2023-09-21T20:20:58Z

`1a61da0`

Support finding EWS runs from GitHub PRs in update-test-expectations
https://bugs.webkit.org/show_bug.cgi?id=257158

Reviewed by Jonathan Bedard.

This adds support for fetching layout test results based on GitHub PR
statuses. To do this, we refactor test_expectation_updater to be able to
support multiple sources of layout tests results, and add various
subcommands to it. With the subcommands added, we rename the script to
update-test-expectations, and add a legacy
update-test-expectations-from-bugzilla which calls the bugzilla
subcommand.

* Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/remote/git_hub.py:
(GitHub._commit_response):
* Tools/Scripts/update-test-expectations: Copied from Tools/Scripts/update-test-expectations-from-bugzilla.
* Tools/Scripts/update-test-expectations-from-bugzilla:
* Tools/Scripts/webkitpy/common/net/bugzilla/results_fetcher.py:
(lookup_ews_results_from_pr):
(lookup_ews_results_from_repo_via_pr):
* Tools/Scripts/webkitpy/common/net/bugzilla/results_fetcher_unittest.py:
(ResultsFetcherTest.test_is_relevant_results):
(ResultsFetcherTest.test_lookup_ews_results_from_bugzilla):
(ResultsFetcherGitHubTest):
(ResultsFetcherGitHubTest.webserver):
* Tools/Scripts/webkitpy/common/net/bugzilla/test_expectation_updater.py:
(configure_logging.LogHandler):
(argument_parser):
(TestExpectationUpdater.__init__):
(TestExpectationUpdater.fetch_from_bugzilla):
(TestExpectationUpdater):
(TestExpectationUpdater.fetch_from_github_pr):
(TestExpectationUpdater.do_update):
(main):
* Tools/Scripts/webkitpy/common/net/bugzilla/test_expectation_updater_unittest.py:
(TestExpectationUpdaterTest.test_update_test_expectations):

Canonical link: https://commits.webkit.org/272152@main

454825b

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
~~🧪 style~~	~~🛠 ios~~	~~🛠 mac~~	~~🛠 wpe~~	~~🛠 wincairo~~
~~🧪 bindings~~	~~🛠 ios-sim~~	~~🛠 mac-AS-debug~~	~~🧪 wpe-wk2~~
~~🧪 webkitperl~~	~~🧪 ios-wk2~~	~~🧪 api-mac~~	~~🧪 api-wpe~~
~~🧪 webkitpy~~	~~🧪 ios-wk2-wpt~~	~~🧪 mac-wk1~~	~~🛠 gtk~~
	~~🧪 api-ios~~	~~🧪 mac-wk2~~	~~🧪 gtk-wk2~~
	~~🛠 tv~~	~~🧪 mac-AS-debug-wk2~~	~~🧪 api-gtk~~
~~🧪 services~~	~~🛠 tv-sim~~	~~🧪 mac-wk2-stress~~
	~~🛠 watch~~
✅ 🛠 🧪 unsafe-merge	~~🛠 watch-sim~~

webkit-early-warning-system · 2023-09-21T20:25:00Z

EWS run on previous version of this PR (hash 9d5e0f5)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
~~🧪 style~~	~~🛠 ios~~	~~🛠 mac~~	~~🛠 wpe~~	~~🛠 wincairo~~
~~🧪 bindings~~	~~🛠 ios-sim~~	~~🛠 mac-AS-debug~~	~~🧪 wpe-wk2~~
~~🧪 webkitperl~~	~~🧪 ios-wk2~~	~~🧪 api-mac~~	~~🛠 gtk~~
~~🧪 webkitpy~~	~~🧪 ios-wk2-wpt~~	~~🧪 mac-wk1~~	~~🧪 gtk-wk2~~
	~~🧪 api-ios~~	~~🧪 mac-wk2~~	~~🧪 api-gtk~~
	~~🛠 tv~~	~~🧪 mac-AS-debug-wk2~~
~~🧪 services~~	~~🛠 tv-sim~~	~~🧪 mac-wk2-stress~~
	~~🛠 watch~~
	~~🛠 watch-sim~~

Tools/Scripts/libraries/webkitscmpy/webkitscmpy/pull_request.py

Tools/Scripts/webkitpy/common/net/bugzilla/results_fetcher.py

gsnedders · 2023-09-28T18:58:14Z

If we care about being able to update results from post-commit, we probably want to be getting status checks from a commit, rather than from a PR. And given that's how both GitHub and Bitbucket Server have these in their APIs, that's probably where it should live for us too.

webkit-early-warning-system · 2023-10-06T23:12:07Z

EWS run on previous version of this PR (hash b268d30)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
~~🧪 style~~	~~🛠 ios~~	~~🛠 mac~~	~~🛠 wpe~~	~~🛠 wincairo~~
~~🧪 bindings~~	~~🛠 ios-sim~~	~~🛠 mac-AS-debug~~	~~🧪 wpe-wk2~~
~~🧪 webkitperl~~	~~🧪 ios-wk2~~	~~🧪 api-mac~~	~~🛠 gtk~~
~~🧪 webkitpy~~	~~🧪 ios-wk2-wpt~~	~~🧪 mac-wk1~~	~~🧪 gtk-wk2~~
	~~🧪 api-ios~~	~~🧪 mac-wk2~~	~~🧪 api-gtk~~
	~~🛠 tv~~	~~🧪 mac-AS-debug-wk2~~
~~🧪 services~~	~~🛠 tv-sim~~	~~🧪 mac-wk2-stress~~
	~~🛠 watch~~
	~~🛠 watch-sim~~

webkit-early-warning-system · 2023-11-14T01:11:24Z

EWS run on previous version of this PR (hash 05116fd)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
~~🧪 style~~	~~🛠 ios~~	~~🛠 mac~~	~~🛠 wpe~~	~~🛠 wincairo~~
~~🧪 bindings~~	~~🛠 ios-sim~~	~~🛠 mac-AS-debug~~	~~🧪 wpe-wk2~~
~~🧪 webkitperl~~	~~🧪 ios-wk2~~	~~🧪 api-mac~~	~~🛠 gtk~~
~~🧪 webkitpy~~	~~🧪 ios-wk2-wpt~~	~~🧪 mac-wk1~~	~~🧪 gtk-wk2~~
	~~🧪 api-ios~~	~~🧪 mac-wk2~~	~~🧪 api-gtk~~
	~~🛠 tv~~	~~🧪 mac-AS-debug-wk2~~
~~🧪 services~~	~~🛠 tv-sim~~	~~🧪 mac-wk2-stress~~
	~~🛠 watch~~
	~~🛠 watch-sim~~

gsnedders · 2023-11-14T17:14:17Z

Okay, trying to apply this to #20412:

% ./Tools/Scripts/update-test-expectations-from-bugzilla -d github-pr 
Updating results from bot mac-wk2 (generic)
mac-wk2 archive: https://ews-build.s3-us-west-2.amazonaws.com/macOS-AppleSilicon-Sonoma-Debug-WK2-Tests-EWS/da9a3797-1341.zip
Traceback (most recent call last):
  File "/Volumes/gsnedders/projects/Safari/OS2/./Tools/Scripts/update-test-expectations-from-bugzilla", line 35, in <module>
    sys.exit(test_expectation_updater.main(sys.argv[1:], sys.stdout, sys.stderr))
  File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/net/bugzilla/test_expectation_updater.py", line 211, in main
    updater.do_update()
  File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/net/bugzilla/test_expectation_updater.py", line 171, in do_update
    self._update_for_generic_bot(platform_name)
  File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/net/bugzilla/test_expectation_updater.py", line 119, in _update_for_generic_bot
    for test_name, expected_content in self._tests_to_update(platform_name).items():
  File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/net/bugzilla/test_expectation_updater.py", line 113, in _tests_to_update
    return {result: zip_file.read(TestResultWriter.actual_filename(result, self.filesystem)) for result in results_to_update}
  File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/net/bugzilla/test_expectation_updater.py", line 113, in <dictcomp>
    return {result: zip_file.read(TestResultWriter.actual_filename(result, self.filesystem)) for result in results_to_update}
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/zipfile.py", line 1463, in read
    with self.open(name, "r", pwd) as fp:
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/zipfile.py", line 1502, in open
    zinfo = self.getinfo(name)
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/zipfile.py", line 1429, in getinfo
    raise KeyError(
KeyError: "There is no item named 'imported/w3c/web-platform-tests/editing/other/delete-in-child-of-head.tentative-actual.txt' in the archive"

webkit-early-warning-system · 2023-11-15T17:03:44Z

EWS run on previous version of this PR (hash 25f8538)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 wincairo
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
✅ 🧪 webkitpy	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 gtk
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🧪 gtk-wk2
	✅ 🛠 tv	✅ 🧪 mac-AS-debug-wk2	✅ 🧪 api-gtk
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2023-11-17T13:19:38Z

EWS run on previous version of this PR (hash 14bc721)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 wincairo
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
✅ 🧪 webkitpy	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 gtk
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🧪 gtk-wk2
	✅ 🛠 tv	✅ 🧪 mac-AS-debug-wk2	✅ 🧪 api-gtk
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

JonWBedard · 2023-12-16T00:28:04Z

Tools/Scripts/webkitpy/common/net/bugzilla/results_fetcher.py

+    source_remote = repository.default_remote
+    remote_repo = repository.remote(name=source_remote)
+    if not remote_repo.pull_requests:
+        msg = "Remote repo doesn't support pull requests"


Why not just raise ValueError directly?

see https://docs.astral.sh/ruff/rules/raw-string-in-exception/; essentially it avoids the string being duplicated in the traceback, which makes it somewhat harder to read

JonWBedard · 2023-12-16T00:28:37Z

Tools/Scripts/webkitpy/common/net/bugzilla/results_fetcher.py

+
+    if existing_pr is None:
+        msg = "No PR found for current branch"
+        raise ValueError(msg)


Ditto on raising ValueError(<str>) directly.

https://bugs.webkit.org/show_bug.cgi?id=257158 Reviewed by Jonathan Bedard. This adds support for fetching layout test results based on GitHub PR statuses. To do this, we refactor test_expectation_updater to be able to support multiple sources of layout tests results, and add various subcommands to it. With the subcommands added, we rename the script to update-test-expectations, and add a legacy update-test-expectations-from-bugzilla which calls the bugzilla subcommand. * Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/remote/git_hub.py: (GitHub._commit_response): * Tools/Scripts/update-test-expectations: Copied from Tools/Scripts/update-test-expectations-from-bugzilla. * Tools/Scripts/update-test-expectations-from-bugzilla: * Tools/Scripts/webkitpy/common/net/bugzilla/results_fetcher.py: (lookup_ews_results_from_pr): (lookup_ews_results_from_repo_via_pr): * Tools/Scripts/webkitpy/common/net/bugzilla/results_fetcher_unittest.py: (ResultsFetcherTest.test_is_relevant_results): (ResultsFetcherTest.test_lookup_ews_results_from_bugzilla): (ResultsFetcherGitHubTest): (ResultsFetcherGitHubTest.webserver): * Tools/Scripts/webkitpy/common/net/bugzilla/test_expectation_updater.py: (configure_logging.LogHandler): (argument_parser): (TestExpectationUpdater.__init__): (TestExpectationUpdater.fetch_from_bugzilla): (TestExpectationUpdater): (TestExpectationUpdater.fetch_from_github_pr): (TestExpectationUpdater.do_update): (main): * Tools/Scripts/webkitpy/common/net/bugzilla/test_expectation_updater_unittest.py: (TestExpectationUpdaterTest.test_update_test_expectations): Canonical link: https://commits.webkit.org/272152@main

webkit-commit-queue · 2023-12-16T01:08:13Z

Committed 272152@main (1a61da0): https://commits.webkit.org/272152@main

Reviewed commits have been landed. Closing PR #18037 and removing active labels.

webkit-early-warning-system · 2023-12-16T01:14:02Z

EWS run on current version of this PR (hash 454825b)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
~~🧪 style~~	~~🛠 ios~~	~~🛠 mac~~	~~🛠 wpe~~	~~🛠 wincairo~~
~~🧪 bindings~~	~~🛠 ios-sim~~	~~🛠 mac-AS-debug~~	~~🧪 wpe-wk2~~
~~🧪 webkitperl~~	~~🧪 ios-wk2~~	~~🧪 api-mac~~	~~🧪 api-wpe~~
~~🧪 webkitpy~~	~~🧪 ios-wk2-wpt~~	~~🧪 mac-wk1~~	~~🛠 gtk~~
	~~🧪 api-ios~~	~~🧪 mac-wk2~~	~~🧪 gtk-wk2~~
	~~🛠 tv~~	~~🧪 mac-AS-debug-wk2~~	~~🧪 api-gtk~~
~~🧪 services~~	~~🛠 tv-sim~~	~~🧪 mac-wk2-stress~~
	~~🛠 watch~~
✅ 🛠 🧪 unsafe-merge	~~🛠 watch-sim~~

gsnedders self-assigned this Sep 21, 2023

gsnedders added Tools / Tests Tools in the Tools directory, build issues, test infrastructure, and bugs in test cases skip-ews Applied to prevent a change from being run on EWS labels Sep 21, 2023

gsnedders commented Sep 21, 2023

View reviewed changes

Tools/Scripts/libraries/webkitscmpy/webkitscmpy/pull_request.py Outdated Show resolved Hide resolved

Tools/Scripts/webkitpy/common/net/bugzilla/results_fetcher.py Show resolved Hide resolved

Tools/Scripts/webkitpy/common/net/bugzilla/results_fetcher.py Outdated Show resolved Hide resolved

gsnedders force-pushed the results_fetcher_github_status branch from 9d5e0f5 to b268d30 Compare October 6, 2023 23:11

JonWBedard force-pushed the results_fetcher_github_status branch from b268d30 to 05116fd Compare November 14, 2023 01:11

JonWBedard mentioned this pull request Nov 14, 2023

[webkitscmpy] Add commit status to pull-request #20494

Merged

gsnedders removed the skip-ews Applied to prevent a change from being run on EWS label Nov 15, 2023

gsnedders force-pushed the results_fetcher_github_status branch from 05116fd to 25f8538 Compare November 15, 2023 17:03

gsnedders force-pushed the results_fetcher_github_status branch from 25f8538 to 14bc721 Compare November 17, 2023 13:19

gsnedders mentioned this pull request Nov 17, 2023

fixup! Re-import web-platform-tests/dom https://bugs.webkit.org/show_bug.cgi?id=264734 #20412

Closed

gsnedders changed the title ~~Support finding EWS runs from GitHub in update-test-expectations-from-bugzilla~~ Support finding EWS runs from GitHub PRs in update-test-expectations Dec 16, 2023

gsnedders force-pushed the results_fetcher_github_status branch from 14bc721 to 454825b Compare December 16, 2023 00:18

JonWBedard reviewed Dec 16, 2023

View reviewed changes

JonWBedard approved these changes Dec 16, 2023

View reviewed changes

gsnedders marked this pull request as ready for review December 16, 2023 00:37

gsnedders requested a review from facetothefate as a code owner December 16, 2023 00:37

gsnedders added the unsafe-merge-queue Applied to send a pull request to merge-queue, but skip building and testing label Dec 16, 2023

webkit-commit-queue force-pushed the results_fetcher_github_status branch from 454825b to 1a61da0 Compare December 16, 2023 01:07

webkit-commit-queue merged commit 1a61da0 into WebKit:main Dec 16, 2023

webkit-commit-queue removed the unsafe-merge-queue Applied to send a pull request to merge-queue, but skip building and testing label Dec 16, 2023

gsnedders deleted the results_fetcher_github_status branch December 16, 2023 01:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support finding EWS runs from GitHub PRs in update-test-expectations #18037

Support finding EWS runs from GitHub PRs in update-test-expectations #18037

gsnedders commented Sep 21, 2023 •

edited by webkit-early-warning-system

Loading

webkit-early-warning-system commented Sep 21, 2023 •

edited

Loading

gsnedders commented Sep 28, 2023

webkit-early-warning-system commented Oct 6, 2023 •

edited

Loading

webkit-early-warning-system commented Nov 14, 2023 •

edited

Loading

gsnedders commented Nov 14, 2023

webkit-early-warning-system commented Nov 15, 2023 •

edited

Loading

webkit-early-warning-system commented Nov 17, 2023 •

edited

Loading

JonWBedard Dec 16, 2023

gsnedders Dec 16, 2023

JonWBedard Dec 16, 2023

webkit-commit-queue commented Dec 16, 2023

webkit-early-warning-system commented Dec 16, 2023 •

edited

Loading

Support finding EWS runs from GitHub PRs in update-test-expectations #18037

Support finding EWS runs from GitHub PRs in update-test-expectations #18037

Conversation

gsnedders commented Sep 21, 2023 • edited by webkit-early-warning-system Loading

1a61da0

webkit-early-warning-system commented Sep 21, 2023 • edited Loading

gsnedders commented Sep 28, 2023

webkit-early-warning-system commented Oct 6, 2023 • edited Loading

webkit-early-warning-system commented Nov 14, 2023 • edited Loading

gsnedders commented Nov 14, 2023

webkit-early-warning-system commented Nov 15, 2023 • edited Loading

webkit-early-warning-system commented Nov 17, 2023 • edited Loading

JonWBedard Dec 16, 2023

Choose a reason for hiding this comment

gsnedders Dec 16, 2023

Choose a reason for hiding this comment

JonWBedard Dec 16, 2023

Choose a reason for hiding this comment

webkit-commit-queue commented Dec 16, 2023

webkit-early-warning-system commented Dec 16, 2023 • edited Loading

gsnedders commented Sep 21, 2023 •

edited by webkit-early-warning-system

Loading

`1a61da0`

webkit-early-warning-system commented Sep 21, 2023 •

edited

Loading

webkit-early-warning-system commented Oct 6, 2023 •

edited

Loading

webkit-early-warning-system commented Nov 14, 2023 •

edited

Loading

webkit-early-warning-system commented Nov 15, 2023 •

edited

Loading

webkit-early-warning-system commented Nov 17, 2023 •

edited

Loading

webkit-early-warning-system commented Dec 16, 2023 •

edited

Loading