Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[macOS] Add support for QR code detection when showing a context menu #10709

Merged
merged 1 commit into from
Feb 27, 2023

Conversation

pxlcoder
Copy link
Member

@pxlcoder pxlcoder commented Feb 27, 2023

b8d16fc

[macOS] Add support for QR code detection when showing a context menu
https://bugs.webkit.org/show_bug.cgi?id=252971
rdar://105953011

Reviewed by Wenson Hsieh.

Safari currently implements support for QR code detection when showing a context
menu using an injected bundle. This patch works towards two goals by adding
equivalent support at the WebKit layer:

1. Reducing the surface of Safari's injected bundle.

2. Enabling iokit blocking in the Web Process. Safari's QR code detection is
   implemented using the Vision framework in the Web Process. With iokit blocking
   enabled, this approach will fail, as Vision fails to create `CIImage`s.

This patch adds initial support for QR code detection when showing a context menu
in WebKit. Support is implemented by using the Vision framework in the UIProcess
and is controlled by new SPI on `WKWebViewConfiguration`, as QR code detection is
not inexpensive.

Note that Safari's heuristics for detecting QR codes outside of <img> elements
are currently unsupported. These heuristics will be implemented in a subsequent
patch.

* Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml:

Add an off-by-default setting for QR code detection when showing a context menu.

* Source/WTF/wtf/PlatformEnableCocoa.h:
* Source/WTF/wtf/PlatformHave.h:

The Vision framework is available on all Cocoa platforms other than watchOS.

* Source/WebCore/PAL/PAL.xcodeproj/project.pbxproj:
* Source/WebCore/PAL/pal/PlatformMac.cmake:
* Source/WebCore/PAL/pal/cocoa/VisionSoftLink.h: Added.
* Source/WebCore/PAL/pal/cocoa/VisionSoftLink.mm: Added.
* Source/WebKit/Platform/cocoa/ImageAnalysisUtilities.h:
* Source/WebKit/Platform/cocoa/ImageAnalysisUtilities.mm:
(WebKit::imageFilledWithWhiteBackground):

Adjust the image used to perform QR code detection to have a white background
to support transparent images.

(WebKit::requestPayloadForQRCode):

Add a helper method to obtain QR code data from a `CGImageRef`.

* Source/WebKit/Shared/ContextMenuContextData.h:
(WebKit::ContextMenuContextData::qrCodePayloadString const):
(WebKit::ContextMenuContextData::setQRCodePayloadString):
* Source/WebKit/UIProcess/API/APIContextMenuClient.h:
(API::ContextMenuClient::menuFromProposedMenu):

Adjust the method to take a `ContextMenuContextData` rather than a `WebHitTestResultData`
to include additional context.

* Source/WebKit/UIProcess/API/APIContextMenuElementInfoMac.h:
* Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm:
(-[WKWebView _setupPageConfiguration:]):
* Source/WebKit/UIProcess/API/Cocoa/WKWebViewConfiguration.mm:
(-[WKWebViewConfiguration init]):
(-[WKWebViewConfiguration copyWithZone:]):
(-[WKWebViewConfiguration _contextMenuQRCodeDetectionEnabled]):
(-[WKWebViewConfiguration _setContextMenuQRCodeDetectionEnabled:]):
* Source/WebKit/UIProcess/API/Cocoa/WKWebViewConfigurationPrivate.h:

Add SPI to control whether QR code detection is performed when showing a context menu.
Detection is not performed by default.

* Source/WebKit/UIProcess/API/Cocoa/_WKContextMenuElementInfo.h:

Expose the detection result via SPI on `_WKContextMenuElementInfo`.

* Source/WebKit/UIProcess/API/Cocoa/_WKContextMenuElementInfo.mm:
(-[_WKContextMenuElementInfo qrCodePayloadString]):
* Source/WebKit/UIProcess/Cocoa/UIDelegate.h:
* Source/WebKit/UIProcess/Cocoa/UIDelegate.mm:
(WebKit::UIDelegate::ContextMenuClient::menuFromProposedMenu):
* Source/WebKit/UIProcess/WebContextMenuProxy.h:

Remove the `const` qualifier from the `ContextMenuContextData` member, as it can
be modified once QR code detection is completed.

* Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.h:
* Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.mm:
(WebKit::WebContextMenuProxyMac::show):

Display the context menu asynchronously if any post-processing (QR code detection)
needs to be performed.

(WebKit::WebContextMenuProxyMac::showAfterPostProcessingContextData):

QR code detection is only performed if the setting is enabled, and there is no
link information in the hit test result. This mirrors the existing behavior in
Safari.

(WebKit::WebContextMenuProxyMac::useContextMenuItems):
* Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* Tools/TestWebKitAPI/Tests/WebKit/qr-code.png: Added.
* Tools/TestWebKitAPI/Tests/mac/ContextMenuTests.mm:
(TestWebKitAPI::TEST):

Added tests for QR code detection when showing a context menu.

Canonical link: https://commits.webkit.org/260897@main

4867e83

Misc iOS, tvOS & watchOS macOS Linux Windows
❌ πŸ§ͺ style βœ… πŸ›  ios βœ… πŸ›  mac βœ… πŸ›  wpe   πŸ›  wincairo
βœ… πŸ§ͺ bindings βœ… πŸ›  ios-sim βœ… πŸ›  mac-AS-debug βœ… πŸ›  gtk
βœ… πŸ§ͺ webkitperl   πŸ§ͺ ios-wk2   πŸ§ͺ api-mac   πŸ§ͺ gtk-wk2
  πŸ§ͺ api-ios   πŸ§ͺ mac-wk1   πŸ§ͺ api-gtk
  πŸ›  πŸ§ͺ jsc βœ… πŸ›  tv   πŸ§ͺ mac-wk2 βœ… πŸ›  jsc-armv7
βœ… πŸ›  πŸ§ͺ jsc-arm64   πŸ›  tv-sim   πŸ§ͺ mac-AS-debug-wk2 βœ… πŸ§ͺ jsc-armv7-tests
  πŸ›  watch   πŸ§ͺ mac-wk2-stress βœ… πŸ›  jsc-mips
βœ… πŸ›  πŸ§ͺ merge βœ… πŸ›  watch-sim βœ… πŸ§ͺ jsc-mips-tests

@pxlcoder pxlcoder self-assigned this Feb 27, 2023
@pxlcoder pxlcoder added the Platform Portability improvements and other general platform improvements not driven directly by site bugs. label Feb 27, 2023
Source/WebKit/Platform/cocoa/ImageAnalysisUtilities.mm Outdated Show resolved Hide resolved
Source/WebKit/Platform/cocoa/ImageAnalysisUtilities.mm Outdated Show resolved Hide resolved
Source/WebKit/Platform/cocoa/ImageAnalysisUtilities.mm Outdated Show resolved Hide resolved
@pxlcoder
Copy link
Member Author

Thanks for the review!

@pxlcoder pxlcoder added the merge-queue Applied to send a pull request to merge-queue label Feb 27, 2023
https://bugs.webkit.org/show_bug.cgi?id=252971
rdar://105953011

Reviewed by Wenson Hsieh.

Safari currently implements support for QR code detection when showing a context
menu using an injected bundle. This patch works towards two goals by adding
equivalent support at the WebKit layer:

1. Reducing the surface of Safari's injected bundle.

2. Enabling iokit blocking in the Web Process. Safari's QR code detection is
   implemented using the Vision framework in the Web Process. With iokit blocking
   enabled, this approach will fail, as Vision fails to create `CIImage`s.

This patch adds initial support for QR code detection when showing a context menu
in WebKit. Support is implemented by using the Vision framework in the UIProcess
and is controlled by new SPI on `WKWebViewConfiguration`, as QR code detection is
not inexpensive.

Note that Safari's heuristics for detecting QR codes outside of <img> elements
are currently unsupported. These heuristics will be implemented in a subsequent
patch.

* Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml:

Add an off-by-default setting for QR code detection when showing a context menu.

* Source/WTF/wtf/PlatformEnableCocoa.h:
* Source/WTF/wtf/PlatformHave.h:

The Vision framework is available on all Cocoa platforms other than watchOS.

* Source/WebCore/PAL/PAL.xcodeproj/project.pbxproj:
* Source/WebCore/PAL/pal/PlatformMac.cmake:
* Source/WebCore/PAL/pal/cocoa/VisionSoftLink.h: Added.
* Source/WebCore/PAL/pal/cocoa/VisionSoftLink.mm: Added.
* Source/WebKit/Platform/cocoa/ImageAnalysisUtilities.h:
* Source/WebKit/Platform/cocoa/ImageAnalysisUtilities.mm:
(WebKit::imageFilledWithWhiteBackground):

Adjust the image used to perform QR code detection to have a white background
to support transparent images.

(WebKit::requestPayloadForQRCode):

Add a helper method to obtain QR code data from a `CGImageRef`.

* Source/WebKit/Shared/ContextMenuContextData.h:
(WebKit::ContextMenuContextData::qrCodePayloadString const):
(WebKit::ContextMenuContextData::setQRCodePayloadString):
* Source/WebKit/UIProcess/API/APIContextMenuClient.h:
(API::ContextMenuClient::menuFromProposedMenu):

Adjust the method to take a `ContextMenuContextData` rather than a `WebHitTestResultData`
to include additional context.

* Source/WebKit/UIProcess/API/APIContextMenuElementInfoMac.h:
* Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm:
(-[WKWebView _setupPageConfiguration:]):
* Source/WebKit/UIProcess/API/Cocoa/WKWebViewConfiguration.mm:
(-[WKWebViewConfiguration init]):
(-[WKWebViewConfiguration copyWithZone:]):
(-[WKWebViewConfiguration _contextMenuQRCodeDetectionEnabled]):
(-[WKWebViewConfiguration _setContextMenuQRCodeDetectionEnabled:]):
* Source/WebKit/UIProcess/API/Cocoa/WKWebViewConfigurationPrivate.h:

Add SPI to control whether QR code detection is performed when showing a context menu.
Detection is not performed by default.

* Source/WebKit/UIProcess/API/Cocoa/_WKContextMenuElementInfo.h:

Expose the detection result via SPI on `_WKContextMenuElementInfo`.

* Source/WebKit/UIProcess/API/Cocoa/_WKContextMenuElementInfo.mm:
(-[_WKContextMenuElementInfo qrCodePayloadString]):
* Source/WebKit/UIProcess/Cocoa/UIDelegate.h:
* Source/WebKit/UIProcess/Cocoa/UIDelegate.mm:
(WebKit::UIDelegate::ContextMenuClient::menuFromProposedMenu):
* Source/WebKit/UIProcess/WebContextMenuProxy.h:

Remove the `const` qualifier from the `ContextMenuContextData` member, as it can
be modified once QR code detection is completed.

* Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.h:
* Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.mm:
(WebKit::WebContextMenuProxyMac::show):

Display the context menu asynchronously if any post-processing (QR code detection)
needs to be performed.

(WebKit::WebContextMenuProxyMac::showAfterPostProcessingContextData):

QR code detection is only performed if the setting is enabled, and there is no
link information in the hit test result. This mirrors the existing behavior in
Safari.

(WebKit::WebContextMenuProxyMac::useContextMenuItems):
* Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* Tools/TestWebKitAPI/Tests/WebKit/qr-code.png: Added.
* Tools/TestWebKitAPI/Tests/mac/ContextMenuTests.mm:
(TestWebKitAPI::TEST):

Added tests for QR code detection when showing a context menu.

Canonical link: https://commits.webkit.org/260897@main
@webkit-commit-queue
Copy link
Collaborator

Committed 260897@main (b8d16fc): https://commits.webkit.org/260897@main

Reviewed commits have been landed. Closing PR #10709 and removing active labels.

@webkit-early-warning-system webkit-early-warning-system merged commit b8d16fc into WebKit:main Feb 27, 2023
@webkit-commit-queue webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Feb 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Platform Portability improvements and other general platform improvements not driven directly by site bugs.
Projects
None yet
4 participants