-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Avoid triggering image analysis when interrupting scroll deceleration…
… by long pressing on an image https://bugs.webkit.org/show_bug.cgi?id=258929 rdar://111506756 Reviewed by Dean Jackson. On iOS, when long pressing over an image, the `WKImageAnalysisGestureRecognizer` activates and begins analyzing images for Live Text and visual look up. This is not desirable for power and performance in the case where the user is simply trying to scroll through a webpage with many images, and frequently interrupts scroll deceleration by long pressing on images. Mitigate this by adding an early return to `-imageAnalysisGestureDidBegin:`, to avoid incurring any extra work in the case where the gesture is being activated while panning within a scroll view. To do this, we add a new `lastTouchedScrollView` property on `WKImageAnalysisGestureRecognizer` that tracks the last `UIScrollView` where touches began. This code is the same as existing logic in `WKHighlightLongPressGestureRecognizer` and `WKSyntheticTapGestureRecognizer`, but it's somewhat tricky to share the implementation between these three classes since their superclasses are all different; for now, we can at least de-duplicate a bit of shared code by adding a new standalone helper function to find a scroll view for a series of touches in the now-renamed `UIKitUtilities.h`, and deploying that helper function in these three places. We also add more testing infrastructure to keep track of a monotonically increasing ID representing the number of times we trigger VisionKit APIs for image analysis by swizzling out `VKCImageAnalyzer` methods; see below for more details. Test: fast/images/text-recognition/ios/no-image-analysis-when-interrupting-scrolling.html * LayoutTests/fast/images/text-recognition/ios/no-image-analysis-when-interrupting-scrolling-expected.txt: Added. * LayoutTests/fast/images/text-recognition/ios/no-image-analysis-when-interrupting-scrolling.html: Added. Add a layout test to verify that interrupting momentum scrolling via a long press doesn't trigger any additional image analyzer requests in VisionKit. * LayoutTests/resources/ui-helper.js: (window.UIHelper.currentImageAnalysisRequestID): Add a helper method to return the current (monotonically increasing) image analysis request ID. * Source/WebKit/SourcesCocoa.txt: * Source/WebKit/UIProcess/API/ios/WKWebViewIOS.mm: * Source/WebKit/UIProcess/Cocoa/MediaPermissionUtilities.mm: * Source/WebKit/UIProcess/Cocoa/WKStorageAccessAlert.mm: Adjust `UIAlertControllerUtilities.h` -> `UIKitUtilities.h`; see below. * Source/WebKit/UIProcess/ios/UIKitUtilities.h: Renamed from Source/WebKit/UIProcess/ios/UIAlertControllerUtilities.h. * Source/WebKit/UIProcess/ios/UIKitUtilities.mm: Renamed from Source/WebKit/UIProcess/ios/UIAlertControllerUtilities.mm. Add a new helper method to this file, to return a `UIScrollView` corresponding to a set of `UITouch` objects; also rename `UIAlertControllerUtilities.*` to `UIKitUtilities.*`, since it now contains helper functions related to UIKit (not just `UIAlertController`). (WebKit::createUIAlertController): (WebKit::scrollViewForTouches): * Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm: (-[WKContentView imageAnalysisGestureDidBegin:]): Implement the actual fix here by checking if the `WKImageAnalysisGestureRecognizer`'s last touched scroll view is being panned. * Source/WebKit/UIProcess/ios/WKHighlightLongPressGestureRecognizer.mm: (-[WKHighlightLongPressGestureRecognizer touchesBegan:withEvent:]): Deduplicate some code, using the new helper function in `UIKitUtilities.h`. * Source/WebKit/UIProcess/ios/WKImageAnalysisGestureRecognizer.h: * Source/WebKit/UIProcess/ios/WKImageAnalysisGestureRecognizer.mm: (-[WKImageAnalysisGestureRecognizer reset]): (-[WKImageAnalysisGestureRecognizer touchesBegan:withEvent:]): Implement `-lastTouchedScrollView` on the image analysis gesture. * Source/WebKit/UIProcess/ios/WKPasswordView.mm: Rename `UIAlertControllerUtilities.h` -> `UIKitUtilities.h` (see above). * Source/WebKit/UIProcess/ios/WKSyntheticTapGestureRecognizer.mm: (-[WKSyntheticTapGestureRecognizer touchesBegan:withEvent:]): Deduplicate some more code, using the new helper function in `UIKitUtilities.h`. * Source/WebKit/UIProcess/ios/WKWebGeolocationPolicyDeciderIOS.mm: * Source/WebKit/UIProcess/ios/fullscreen/WKFullScreenViewController.mm: Rename `UIAlertControllerUtilities.h` -> `UIKitUtilities.h` (see above). * Source/WebKit/WebKit.xcodeproj/project.pbxproj: * Tools/TestRunnerShared/UIScriptContext/Bindings/UIScriptController.idl: Add a new script controller hook to return the number of times we triggered image analyzer requests. * Tools/TestRunnerShared/UIScriptContext/UIScriptController.h: (WTR::UIScriptController::currentImageAnalysisRequestID const): * Tools/WebKitTestRunner/Configurations/WebKitTestRunnerApp.xcconfig: * Tools/WebKitTestRunner/Configurations/WebKitTestRunnerLibrary.xcconfig: Link against PAL, so that we can use extant soft-linking macros in `VisionKitCoreSoftLink.h`. * Tools/WebKitTestRunner/TestController.h: * Tools/WebKitTestRunner/cocoa/TestControllerCocoa.mm: (swizzledProcessImageAnalysisRequest): Swizzle out calls to `-processRequest:progressHandler:completionHandler:` to always return a consistent result (error), and increment the global `gCurrentImageAnalysisRequestID` count returned by the new script controller attribute. (WTR::TestController::cocoaPlatformInitialize): (WTR::TestController::currentImageAnalysisRequestID): * Tools/WebKitTestRunner/cocoa/UIScriptControllerCocoa.h: * Tools/WebKitTestRunner/cocoa/UIScriptControllerCocoa.mm: (WTR::UIScriptControllerCocoa::currentImageAnalysisRequestID const): Canonical link: https://commits.webkit.org/265824@main
- Loading branch information
Showing
26 changed files
with
190 additions
and
29 deletions.
There are no files selected for viewing
10 changes: 10 additions & 0 deletions
10
...st/images/text-recognition/ios/no-image-analysis-when-interrupting-scrolling-expected.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
Verifies that long pressing an image while stopping momentum scrolling does not trigger spurious image analysis requests. This test requires WebKitTestRunner. | ||
|
||
On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE". | ||
|
||
|
||
PASS imageAnalysisRequestIDBefore is imageAnalysisRequestIDAfter | ||
PASS successfullyParsed is true | ||
|
||
TEST COMPLETE | ||
|
54 changes: 54 additions & 0 deletions
54
...Tests/fast/images/text-recognition/ios/no-image-analysis-when-interrupting-scrolling.html
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
<!DOCTYPE html> <!-- webkit-test-runner [ useFlexibleViewport=true ] --> | ||
<html> | ||
<head> | ||
<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no"> | ||
<script src="../../../../resources/js-test.js"></script> | ||
<script src="../../../../resources/ui-helper.js"></script> | ||
<style> | ||
html, body { | ||
width: 100%; | ||
height: 100%; | ||
margin: 0px; | ||
padding: 0px; | ||
} | ||
|
||
img { | ||
width: 100%; | ||
height: 100%; | ||
-webkit-touch-callout: none; | ||
} | ||
</style> | ||
<body onload="runTest()"> | ||
<img src="../../resources/dice.png"> | ||
<img src="../../resources/dice.png"> | ||
<img src="../../resources/dice.png"> | ||
<img src="../../resources/dice.png"> | ||
<img src="../../resources/dice.png"> | ||
<img src="../../resources/dice.png"> | ||
<img src="../../resources/dice.png"> | ||
<img src="../../resources/dice.png"> | ||
<img src="../../resources/dice.png"> | ||
<img src="../../resources/dice.png"> | ||
</body> | ||
<script> | ||
jsTestIsAsync = true; | ||
|
||
async function runTest() | ||
{ | ||
description("Verifies that long pressing an image while stopping momentum scrolling does not trigger spurious image analysis requests. This test requires WebKitTestRunner."); | ||
|
||
if (!window.testRunner) | ||
return; | ||
|
||
await UIHelper.dragFromPointToPoint(160, 500, 160, 50, 0.2); | ||
await UIHelper.ensureVisibleContentRectUpdate(); | ||
imageAnalysisRequestIDBefore = await UIHelper.currentImageAnalysisRequestID(); | ||
|
||
await UIHelper.longPressAtPoint(160, document.scrollingElement.scrollTop + 250); | ||
imageAnalysisRequestIDAfter = await UIHelper.currentImageAnalysisRequestID(); | ||
|
||
shouldBe("imageAnalysisRequestIDBefore", "imageAnalysisRequestIDAfter"); | ||
finishJSTest(); | ||
} | ||
</script> | ||
</html> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.