Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Remote Inspection] Element targeting should additionally find nearby out-of-flow elements #26435

Merged
merged 1 commit into from
Mar 26, 2024

Conversation

whsieh
Copy link
Member

@whsieh whsieh commented Mar 25, 2024

6458185

[Remote Inspection] Element targeting should additionally find nearby out-of-flow elements
https://bugs.webkit.org/show_bug.cgi?id=271616

Reviewed by Abrar Protyasha.

When targeting elements for remote inspection, additionally surface nearby elements which aren't
precisely under the hit test location. Our strategy for this consists of the following:

1.  While collecting targets, aggregate a `Region` containing the rects of all targeted out-of-flow
    elements (and hit-tested elements underneath targeted elements).

2.  After building the list of targets, scan the entire DOM for out-of-flow renderers that are also
    contained in the "nearby targets" region, which also satisfy the same criteria for element
    targeting.

3.  Add these as "nearby targets" to the final list of target infos, to the end of the array.

See below for more details.

Test: ElementTargeting.NearbyOutOfFlowElements

* Source/WebCore/page/ElementTargeting.cpp:
(WebCore::targetedElementInfo):

Refactor this code to pull common logic into lambdas, and implement the steps detailed above.

(WebCore::findTargetedElements):
* Source/WebCore/page/ElementTargetingTypes.h:
* Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in:
* Source/WebKit/UIProcess/API/APITargetedElementInfo.h:
* Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm:
(-[WKWebView _requestTargetedElementInfo:completionHandler:]):
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementInfo.h:

Add a new `isUnderPoint` property, which is `YES` for elements that are directly hit-tested, and
`NO` for nearby targets.

* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementInfo.mm:
(-[_WKTargetedElementInfo isUnderPoint]):
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementRequest.h:

Add a new `canIncludeNearbyElements` (default: `YES`) which determines whether or not element
targeting should include elements that have not been hit-tested, but are visually contained within
another element that has been hit-tested.

* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementRequest.mm:
(-[_WKTargetedElementRequest init]):
* Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* Tools/TestWebKitAPI/Tests/WebKitCocoa/ElementTargetingTests.mm:
(TestWebKitAPI::TEST):
* Tools/TestWebKitAPI/Tests/WebKitCocoa/element-targeting-2.html: Added.

Add a new API test to exercise the change.

Canonical link: https://commits.webkit.org/276670@main

e305a09

Misc iOS, tvOS & watchOS macOS Linux Windows
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe ✅ 🛠 wincairo
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug 🧪 wpe-wk2
✅ 🧪 webkitperl 🧪 ios-wk2 🧪 api-mac ❌ 🧪 api-wpe
🧪 ios-wk2-wpt ✅ 🧪 mac-wk1 ✅ 🛠 wpe-skia
🧪 api-ios ✅ 🧪 mac-wk2 ✅ 🛠 gtk
🛠 tv ✅ 🧪 mac-AS-debug-wk2 🧪 gtk-wk2
✅ 🛠 tv-sim ✅ 🧪 api-gtk
✅ 🛠 🧪 merge 🛠 watch
✅ 🛠 watch-sim

@whsieh whsieh self-assigned this Mar 25, 2024
@whsieh whsieh added the Platform Portability improvements and other general platform improvements not driven directly by site bugs. label Mar 25, 2024
@whsieh whsieh requested a review from rr-codes March 25, 2024 22:06
@whsieh whsieh marked this pull request as ready for review March 25, 2024 22:06
@whsieh whsieh requested a review from cdumez as a code owner March 25, 2024 22:06
Source/WebCore/page/ElementTargeting.cpp Outdated Show resolved Hide resolved
@whsieh whsieh added the merge-queue Applied to send a pull request to merge-queue label Mar 26, 2024
… out-of-flow elements

https://bugs.webkit.org/show_bug.cgi?id=271616

Reviewed by Abrar Protyasha.

When targeting elements for remote inspection, additionally surface nearby elements which aren't
precisely under the hit test location. Our strategy for this consists of the following:

1.  While collecting targets, aggregate a `Region` containing the rects of all targeted out-of-flow
    elements (and hit-tested elements underneath targeted elements).

2.  After building the list of targets, scan the entire DOM for out-of-flow renderers that are also
    contained in the "nearby targets" region, which also satisfy the same criteria for element
    targeting.

3.  Add these as "nearby targets" to the final list of target infos, to the end of the array.

See below for more details.

Test: ElementTargeting.NearbyOutOfFlowElements

* Source/WebCore/page/ElementTargeting.cpp:
(WebCore::targetedElementInfo):

Refactor this code to pull common logic into lambdas, and implement the steps detailed above.

(WebCore::findTargetedElements):
* Source/WebCore/page/ElementTargetingTypes.h:
* Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in:
* Source/WebKit/UIProcess/API/APITargetedElementInfo.h:
* Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm:
(-[WKWebView _requestTargetedElementInfo:completionHandler:]):
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementInfo.h:

Add a new `isUnderPoint` property, which is `YES` for elements that are directly hit-tested, and
`NO` for nearby targets.

* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementInfo.mm:
(-[_WKTargetedElementInfo isUnderPoint]):
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementRequest.h:

Add a new `canIncludeNearbyElements` (default: `YES`) which determines whether or not element
targeting should include elements that have not been hit-tested, but are visually contained within
another element that has been hit-tested.

* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementRequest.mm:
(-[_WKTargetedElementRequest init]):
* Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* Tools/TestWebKitAPI/Tests/WebKitCocoa/ElementTargetingTests.mm:
(TestWebKitAPI::TEST):
* Tools/TestWebKitAPI/Tests/WebKitCocoa/element-targeting-2.html: Added.

Add a new API test to exercise the change.

Canonical link: https://commits.webkit.org/276670@main
@webkit-commit-queue
Copy link
Collaborator

Committed 276670@main (6458185): https://commits.webkit.org/276670@main

Reviewed commits have been landed. Closing PR #26435 and removing active labels.

@webkit-commit-queue webkit-commit-queue merged commit 6458185 into WebKit:main Mar 26, 2024
@webkit-commit-queue webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Mar 26, 2024
@whsieh whsieh deleted the eng/271616 branch March 26, 2024 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Platform Portability improvements and other general platform improvements not driven directly by site bugs.
Projects
None yet
5 participants