-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[iOS] Image analysis menu items for images with machine-readable code…
…s sometimes fail to appear https://bugs.webkit.org/show_bug.cgi?id=265544 rdar://118540193 Reviewed by Megan Gardner. Our current logic for dynamically inserting image analysis menu items (such as "Look Up" and any menu items for machine-readable codes) works like this: 1. We kick off image analysis for machine-readable codes and visual search when the context menu is about to be presented. The menu is presented with a hidden placeholder menu item, which represents the set of dynamically-inserted image analysis items we might insert in the future. 2. When the image analysis started by (1) is done, we use `-updateVisibleMenuWithBlock:` on the context menu interaction to replace the placeholder item above with the final set of image analysis items (only if the WebKit client didn't remove the placeholder menu item). However, there's a subtle race here, where image analysis may complete before the context menu has a visible menu. In this scenario, the call to `-updateVisibleMenuWithBlock:` returns early without calling the update block, and we end up missing our only chance to swap out the placeholder item for the actual image analysis items. To fix this, we detect the case where our image analysis completes before the menu is visible, and try again in `-contextMenuInteraction:willDisplayMenuForConfiguration:animator:`, after the menu is guaranteed to be visible. Test: fast/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items.html * LayoutTests/fast/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items-expected.txt: Added. * LayoutTests/fast/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items.html: Added. Add a new layout test, along with some additional test infrastructure to exercise this fix. * LayoutTests/resources/ui-helper.js: Add a way to simulate MRC menu item results when long-pressing on an image in layout tests, on iOS. (window.UIHelper.contextMenuItems): (window.UIHelper.installFakeMachineReadableCodeResultsForImageAnalysis): * Source/WebKit/UIProcess/ios/WKContentViewInteraction.h: * Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm: (-[WKContentView _setUpImageAnalysis]): (-[WKContentView _tearDownImageAnalysis]): (-[WKContentView imageAnalysisGestureDidBegin:]): Replace the extant boolean flag `_waitingForDynamicImageAnalysisContextMenuActions` with a tri-state enum type, `_dynamicImageAnalysisContextMenuState`, which allows us to appropriately deal with the corner case where image analysis completes before the context menu has any visible items. See above for more details. (-[WKContentView _completeImageAnalysisRequestForContextMenu:requestIdentifier:hasTextResults:]): (-[WKContentView _insertDynamicImageAnalysisContextMenuItemsIfPossible]): Pull the existing logic for updating visible context menu items to include the dynamically-inserted image analysis items into a separate helper method, so that we can invoke it after the context menu is done presenting. If the update block runs successfully (i.e. the menu is visible), then we transition directly to the `NotWaiting` state and we're done. However, if the update block never runs when calling into `-updateVisibleMenuWithBlock:`, we instead transition to `WaitingForVisibleMenu` and wait until the menu is done presenting, before trying to perform the update again. (-[WKContentView placeholderForDynamicallyInsertedImageAnalysisActions]): (-[WKContentView contextMenuInteraction:willDisplayMenuForConfiguration:animator:]): Update the menu to include MRC menu items (and other dynamically-inserted image analysis items) if needed — note that we'll only perform the update here, in the case where we failed to run the update block at all, in `-_completeImageAnalysisRequestForContextMenu:requestIdentifier:hasTextResults:`. * Tools/TestRunnerShared/UIScriptContext/Bindings/UIScriptController.idl: * Tools/TestRunnerShared/UIScriptContext/UIScriptController.h: (WTR::UIScriptController::installFakeMachineReadableCodeResultsForImageAnalysis): * Tools/WebKitTestRunner/TestController.h: * Tools/WebKitTestRunner/cocoa/TestControllerCocoa.mm: (fakeMachineReadableCodeMenuForTesting): (-[FakeMachineReadableCodeImageAnalysis allLines]): (-[FakeMachineReadableCodeImageAnalysis hasResultsForAnalysisTypes:]): (-[FakeMachineReadableCodeImageAnalysis mrcMenu]): Add a helper class that represents a fake VisionKit image analysis result that contains MRC menu items (for simplicity, there's only one hard-coded item titled "QR code"). (swizzledProcessImageAnalysisRequest): (WTR::TestController::installFakeMachineReadableCodeResultsForImageAnalysis): (WTR::TestController::shouldUseFakeMachineReadableCodeResultsForImageAnalysis const): (WTR::TestController::cocoaResetStateToConsistentValues): * Tools/WebKitTestRunner/cocoa/UIScriptControllerCocoa.h: * Tools/WebKitTestRunner/cocoa/UIScriptControllerCocoa.mm: (WTR::UIScriptControllerCocoa::installFakeMachineReadableCodeResultsForImageAnalysis): Canonical link: https://commits.webkit.org/271317@main
- Loading branch information
Showing
11 changed files
with
221 additions
and
50 deletions.
There are no files selected for viewing
11 changes: 11 additions & 0 deletions
11
...t/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items-expected.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
Verifies that dynamically-inserted image analysis context menu items show up after long-pressing on an image. This test requires WebKitTestRunner. | ||
|
||
On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE". | ||
|
||
|
||
PASS Found QR code menu item | ||
PASS Finished dismissing context menu | ||
PASS successfullyParsed is true | ||
|
||
TEST COMPLETE | ||
|
39 changes: 39 additions & 0 deletions
39
...ests/fast/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items.html
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
<!DOCTYPE html> <!-- webkit-test-runner [ useFlexibleViewport=true ] --> | ||
<meta charset="utf-8"> | ||
<html> | ||
<head> | ||
<meta name="viewport" content="initial-scale=1"> | ||
<script src="../../../../resources/ui-helper.js"></script> | ||
<script src="../../../../resources/js-test.js"></script> | ||
<script> | ||
jsTestIsAsync = true; | ||
|
||
addEventListener("load", async () => { | ||
description("Verifies that dynamically-inserted image analysis context menu items show up after long-pressing on an image. This test requires WebKitTestRunner."); | ||
|
||
if (!window.testRunner || !testRunner.runUIScript) | ||
return; | ||
|
||
await UIHelper.installFakeMachineReadableCodeResultsForImageAnalysis(); | ||
await UIHelper.longPressElement(document.querySelector("img")); | ||
|
||
let foundQRCodeItem = false; | ||
do { | ||
await UIHelper.delayFor(200); | ||
let contextMenu = await UIHelper.contextMenuItems(); | ||
if (contextMenu?.items?.includes("QR Code")) | ||
foundQRCodeItem = true; | ||
} while (!foundQRCodeItem); | ||
|
||
testPassed("Found QR code menu item"); | ||
await UIHelper.activateAt(0, 0); | ||
await UIHelper.waitForContextMenuToHide(); | ||
testPassed("Finished dismissing context menu"); | ||
finishJSTest(); | ||
}); | ||
</script> | ||
</head> | ||
<body> | ||
<img src="../../../images/resources/dice.png" width="320" height="240" alt="Dice"> | ||
</body> | ||
</html> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.