Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add WebKit API scaffolding to request targeted elements for remote inspection #26207

Merged
merged 1 commit into from
Mar 21, 2024

Conversation

whsieh
Copy link
Member

@whsieh whsieh commented Mar 20, 2024

3aef7bc

Add WebKit API scaffolding to request targeted elements for remote inspection
https://bugs.webkit.org/show_bug.cgi?id=271333

Reviewed by Aditya Keerthi.

Add WebKit API objects and SPI stubs to hit-test DOM elements for improved element targeting when
the web view is in element selection mode. See below for more details.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/page/ElementTargeting.cpp: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.
(WebCore::findTargetedElements):
* Source/WebCore/page/ElementTargeting.h: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.
* Source/WebCore/page/ElementTargetingTypes.h: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.

Add a header declaring the new core types in WebCore: `TargetedElementInfo` (which corresponds to
the API object of the same name in WebKit) and `TargetedElementRequest` (which corresponds to
`_WKTargetedElementRequest`).

* Source/WebKit/Scripts/webkit/messages.py:
(headers_for_type):
* Source/WebKit/Shared/API/APIObject.h:
* Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h:

Add the new private headers to the umbrella.

* Source/WebKit/Shared/Cocoa/APIObject.mm:
(API::Object::newObject):
* Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in:
* Source/WebKit/Sources.txt:
* Source/WebKit/SourcesCocoa.txt:
* Source/WebKit/UIProcess/API/APITargetedElementInfo.cpp: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.
(API::TargetedElementInfo::TargetedElementInfo):

Add an API object to represent a single targeted element retrieved via the new SPI, which is wrapped
by the Objective-C SPI object `_WKTargetedElementInfo`.

(API::TargetedElementInfo::isSameElement const):
(API::TargetedElementInfo::boundsInWebView const):
* Source/WebKit/UIProcess/API/APITargetedElementInfo.h: Added.
* Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm:
(-[WKWebView _requestTargetedElementInfo:completionHandler:]):
* Source/WebKit/UIProcess/API/Cocoa/WKWebViewPrivate.h:
* Source/WebKit/UIProcess/API/Cocoa/_WKRectEdge.h: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.

Move the existing macOS-specific declaration of `_WKRectEdge` out to a separate header, and make it
available on iOS as well. This allows us to reuse this type for the new property on
`_WKTargetedElementInfo`.

* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementInfo.h: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementInfo.mm: Added.
(-[_WKTargetedElementInfo dealloc]):
(-[_WKTargetedElementInfo _apiObject]):
(-[_WKTargetedElementInfo positionType]):
(-[_WKTargetedElementInfo bounds]):
(-[_WKTargetedElementInfo selectors]):
(-[_WKTargetedElementInfo renderedText]):
(-[_WKTargetedElementInfo offsetEdges]):
(-[_WKTargetedElementInfo hasBottomOffset]):
(-[_WKTargetedElementInfo hasLeftOffset]):
(-[_WKTargetedElementInfo hasRightOffset]):
(-[_WKTargetedElementInfo isSameElement:]):
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementInfoInternal.h: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementRequest.h: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementRequest.mm: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.

I opted to avoid making a new API object to represent the request object, since it's only a single
`CGPoint` (and may include another boolean parameter in the future). If these request options become
more complex, we can consider introducing an intermediate `API::TargetedElementRequest` class to
represent this.

* Source/WebKit/UIProcess/Cocoa/SOAuthorization/SubFrameSOAuthorizationSession.mm:
* Source/WebKit/UIProcess/Cocoa/VideoPresentationManagerProxy.mm:

Some build fixes here, due to unified source order changes.

* Source/WebKit/UIProcess/PageClient.h:
(WebKit::PageClient::rootViewToWebView const):

Add a client helper method to map the given rect from root view coordinates to web view coordinates.
On iOS, this maps from `WKContentView`'s coordinate space to `WKWebView`.

* Source/WebKit/UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::requestTargetedElement):
* Source/WebKit/UIProcess/WebPageProxy.h:
* Source/WebKit/UIProcess/ios/PageClientImplIOS.h:
* Source/WebKit/UIProcess/ios/PageClientImplIOS.mm:
(WebKit::PageClientImpl::rootViewToWebView const):
* Source/WebKit/WebKit.xcodeproj/project.pbxproj:
* Source/WebKit/WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::requestTargetedElement):
* Source/WebKit/WebProcess/WebPage/WebPage.h:
* Source/WebKit/WebProcess/WebPage/WebPage.messages.in:

Canonical link: https://commits.webkit.org/276475@main

a181cfb

Misc iOS, tvOS & watchOS macOS Linux Windows
❌ πŸ§ͺ style βœ… πŸ›  ios βœ… πŸ›  mac βœ… πŸ›  wpe βœ… πŸ›  wincairo
βœ… πŸ§ͺ bindings βœ… πŸ›  ios-sim βœ… πŸ›  mac-AS-debug βœ… πŸ§ͺ wpe-wk2
βœ… πŸ§ͺ webkitperl βœ… πŸ§ͺ ios-wk2 βœ… πŸ§ͺ api-mac βœ… πŸ§ͺ api-wpe
βœ… πŸ§ͺ webkitpy βœ… πŸ§ͺ ios-wk2-wpt βœ… πŸ§ͺ mac-wk1 βœ… πŸ›  wpe-skia
βœ… πŸ§ͺ api-ios βœ… πŸ§ͺ mac-wk2 βœ… πŸ›  gtk
βœ… πŸ›  tv βœ… πŸ§ͺ mac-AS-debug-wk2 ❌ πŸ§ͺ gtk-wk2
βœ… πŸ›  tv-sim βœ… πŸ§ͺ api-gtk
βœ… πŸ›  πŸ§ͺ merge βœ… πŸ›  watch
βœ… πŸ›  watch-sim

@whsieh whsieh self-assigned this Mar 20, 2024
@whsieh whsieh added the Platform Portability improvements and other general platform improvements not driven directly by site bugs. label Mar 20, 2024
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Mar 20, 2024
@whsieh whsieh removed the merging-blocked Applied to prevent a change from being merged label Mar 20, 2024
@whsieh whsieh marked this pull request as ready for review March 20, 2024 22:35
@whsieh whsieh requested a review from cdumez as a code owner March 20, 2024 22:35
Source/WebCore/page/ElementTargeting.cpp Outdated Show resolved Hide resolved
Source/WebCore/page/ElementTargetingTypes.h Outdated Show resolved Hide resolved
Source/WebKit/UIProcess/API/APITargetedElementInfo.cpp Outdated Show resolved Hide resolved
@whsieh whsieh changed the title [Cocoa] Add WebKit API scaffolding to request targeted elements for remote inspection Add WebKit API scaffolding to request targeted elements for remote inspection Mar 21, 2024
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Mar 21, 2024
@whsieh whsieh removed the merging-blocked Applied to prevent a change from being merged label Mar 21, 2024
@whsieh whsieh added the merge-queue Applied to send a pull request to merge-queue label Mar 21, 2024
@whsieh
Copy link
Member Author

whsieh commented Mar 21, 2024

Thanks for the reviews!

…spection

https://bugs.webkit.org/show_bug.cgi?id=271333

Reviewed by Aditya Keerthi.

Add WebKit API objects and SPI stubs to hit-test DOM elements for improved element targeting when
the web view is in element selection mode. See below for more details.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/page/ElementTargeting.cpp: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.
(WebCore::findTargetedElements):
* Source/WebCore/page/ElementTargeting.h: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.
* Source/WebCore/page/ElementTargetingTypes.h: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.

Add a header declaring the new core types in WebCore: `TargetedElementInfo` (which corresponds to
the API object of the same name in WebKit) and `TargetedElementRequest` (which corresponds to
`_WKTargetedElementRequest`).

* Source/WebKit/Scripts/webkit/messages.py:
(headers_for_type):
* Source/WebKit/Shared/API/APIObject.h:
* Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h:

Add the new private headers to the umbrella.

* Source/WebKit/Shared/Cocoa/APIObject.mm:
(API::Object::newObject):
* Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in:
* Source/WebKit/Sources.txt:
* Source/WebKit/SourcesCocoa.txt:
* Source/WebKit/UIProcess/API/APITargetedElementInfo.cpp: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.
(API::TargetedElementInfo::TargetedElementInfo):

Add an API object to represent a single targeted element retrieved via the new SPI, which is wrapped
by the Objective-C SPI object `_WKTargetedElementInfo`.

(API::TargetedElementInfo::isSameElement const):
(API::TargetedElementInfo::boundsInWebView const):
* Source/WebKit/UIProcess/API/APITargetedElementInfo.h: Added.
* Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm:
(-[WKWebView _requestTargetedElementInfo:completionHandler:]):
* Source/WebKit/UIProcess/API/Cocoa/WKWebViewPrivate.h:
* Source/WebKit/UIProcess/API/Cocoa/_WKRectEdge.h: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.

Move the existing macOS-specific declaration of `_WKRectEdge` out to a separate header, and make it
available on iOS as well. This allows us to reuse this type for the new property on
`_WKTargetedElementInfo`.

* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementInfo.h: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementInfo.mm: Added.
(-[_WKTargetedElementInfo dealloc]):
(-[_WKTargetedElementInfo _apiObject]):
(-[_WKTargetedElementInfo positionType]):
(-[_WKTargetedElementInfo bounds]):
(-[_WKTargetedElementInfo selectors]):
(-[_WKTargetedElementInfo renderedText]):
(-[_WKTargetedElementInfo offsetEdges]):
(-[_WKTargetedElementInfo hasBottomOffset]):
(-[_WKTargetedElementInfo hasLeftOffset]):
(-[_WKTargetedElementInfo hasRightOffset]):
(-[_WKTargetedElementInfo isSameElement:]):
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementInfoInternal.h: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementRequest.h: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.
* Source/WebKit/UIProcess/API/Cocoa/_WKTargetedElementRequest.mm: Copied from Source/WebKit/Shared/API/Cocoa/WebKitPrivate.h.

I opted to avoid making a new API object to represent the request object, since it's only a single
`CGPoint` (and may include another boolean parameter in the future). If these request options become
more complex, we can consider introducing an intermediate `API::TargetedElementRequest` class to
represent this.

* Source/WebKit/UIProcess/Cocoa/SOAuthorization/SubFrameSOAuthorizationSession.mm:
* Source/WebKit/UIProcess/Cocoa/VideoPresentationManagerProxy.mm:

Some build fixes here, due to unified source order changes.

* Source/WebKit/UIProcess/PageClient.h:
(WebKit::PageClient::rootViewToWebView const):

Add a client helper method to map the given rect from root view coordinates to web view coordinates.
On iOS, this maps from `WKContentView`'s coordinate space to `WKWebView`.

* Source/WebKit/UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::requestTargetedElement):
* Source/WebKit/UIProcess/WebPageProxy.h:
* Source/WebKit/UIProcess/ios/PageClientImplIOS.h:
* Source/WebKit/UIProcess/ios/PageClientImplIOS.mm:
(WebKit::PageClientImpl::rootViewToWebView const):
* Source/WebKit/WebKit.xcodeproj/project.pbxproj:
* Source/WebKit/WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::requestTargetedElement):
* Source/WebKit/WebProcess/WebPage/WebPage.h:
* Source/WebKit/WebProcess/WebPage/WebPage.messages.in:

Canonical link: https://commits.webkit.org/276475@main
@webkit-commit-queue
Copy link
Collaborator

Committed 276475@main (3aef7bc): https://commits.webkit.org/276475@main

Reviewed commits have been landed. Closing PR #26207 and removing active labels.

@webkit-commit-queue webkit-commit-queue merged commit 3aef7bc into WebKit:main Mar 21, 2024
@webkit-commit-queue webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Mar 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Platform Portability improvements and other general platform improvements not driven directly by site bugs.
Projects
None yet
7 participants