[iOS] Image analysis menu items for images with machine-readable code…

…s sometimes fail to appear

https://bugs.webkit.org/show_bug.cgi?id=265544
rdar://118540193

Reviewed by Megan Gardner.

Our current logic for dynamically inserting image analysis menu items (such as "Look Up" and any
menu items for machine-readable codes) works like this:

1.  We kick off image analysis for machine-readable codes and visual search when the context menu is
    about to be presented. The menu is presented with a hidden placeholder menu item, which
    represents the set of dynamically-inserted image analysis items we might insert in the future.

2.  When the image analysis started by (1) is done, we use `-updateVisibleMenuWithBlock:` on the
    context menu interaction to replace the placeholder item above with the final set of image
    analysis items (only if the WebKit client didn't remove the placeholder menu item).

However, there's a subtle race here, where image analysis may complete before the context menu has a
visible menu. In this scenario, the call to `-updateVisibleMenuWithBlock:` returns early without
calling the update block, and we end up missing our only chance to swap out the placeholder item for
the actual image analysis items.

To fix this, we detect the case where our image analysis completes before the menu is visible, and
try again in `-contextMenuInteraction:willDisplayMenuForConfiguration:animator:`, after the menu
is guaranteed to be visible.

Test: fast/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items.html

* LayoutTests/fast/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items-expected.txt: Added.
* LayoutTests/fast/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items.html: Added.

Add a new layout test, along with some additional test infrastructure to exercise this fix.

* LayoutTests/resources/ui-helper.js:

Add a way to simulate MRC menu item results when long-pressing on an image in layout tests, on iOS.

(window.UIHelper.contextMenuItems):
(window.UIHelper.installFakeMachineReadableCodeResultsForImageAnalysis):
* Source/WebKit/UIProcess/ios/WKContentViewInteraction.h:
* Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm:
(-[WKContentView _setUpImageAnalysis]):
(-[WKContentView _tearDownImageAnalysis]):
(-[WKContentView imageAnalysisGestureDidBegin:]):

Replace the extant boolean flag `_waitingForDynamicImageAnalysisContextMenuActions` with a tri-state
enum type, `_dynamicImageAnalysisContextMenuState`, which allows us to appropriately deal with the
corner case where image analysis completes before the context menu has any visible items. See above
for more details.

(-[WKContentView _completeImageAnalysisRequestForContextMenu:requestIdentifier:hasTextResults:]):
(-[WKContentView _insertDynamicImageAnalysisContextMenuItemsIfPossible]):

Pull the existing logic for updating visible context menu items to include the dynamically-inserted
image analysis items into a separate helper method, so that we can invoke it after the context menu
is done presenting.

If the update block runs successfully (i.e. the menu is visible), then we transition directly to the
`NotWaiting` state and we're done. However, if the update block never runs when calling into
`-updateVisibleMenuWithBlock:`, we instead transition to `WaitingForVisibleMenu` and wait until the
menu is done presenting, before trying to perform the update again.

(-[WKContentView placeholderForDynamicallyInsertedImageAnalysisActions]):
(-[WKContentView contextMenuInteraction:willDisplayMenuForConfiguration:animator:]):

Update the menu to include MRC menu items (and other dynamically-inserted image analysis items) if
needed — note that we'll only perform the update here, in the case where we failed to run the update
block at all, in `-_completeImageAnalysisRequestForContextMenu:requestIdentifier:hasTextResults:`.

* Tools/TestRunnerShared/UIScriptContext/Bindings/UIScriptController.idl:
* Tools/TestRunnerShared/UIScriptContext/UIScriptController.h:
(WTR::UIScriptController::installFakeMachineReadableCodeResultsForImageAnalysis):
* Tools/WebKitTestRunner/TestController.h:
* Tools/WebKitTestRunner/cocoa/TestControllerCocoa.mm:
(fakeMachineReadableCodeMenuForTesting):
(-[FakeMachineReadableCodeImageAnalysis allLines]):
(-[FakeMachineReadableCodeImageAnalysis hasResultsForAnalysisTypes:]):
(-[FakeMachineReadableCodeImageAnalysis mrcMenu]):

Add a helper class that represents a fake VisionKit image analysis result that contains MRC menu
items (for simplicity, there's only one hard-coded item titled "QR code").

(swizzledProcessImageAnalysisRequest):
(WTR::TestController::installFakeMachineReadableCodeResultsForImageAnalysis):
(WTR::TestController::shouldUseFakeMachineReadableCodeResultsForImageAnalysis const):
(WTR::TestController::cocoaResetStateToConsistentValues):
* Tools/WebKitTestRunner/cocoa/UIScriptControllerCocoa.h:
* Tools/WebKitTestRunner/cocoa/UIScriptControllerCocoa.mm:
(WTR::UIScriptControllerCocoa::installFakeMachineReadableCodeResultsForImageAnalysis):

Canonical link: https://commits.webkit.org/271317@main

LayoutTests/fast/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items-expected.txt

@@ -0,0 +1,11 @@
+Verifies that dynamically-inserted image analysis context menu items show up after long-pressing on an image. This test requires WebKitTestRunner.
+
+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
+
+
+PASS Found QR code menu item
+PASS Finished dismissing context menu
+PASS successfullyParsed is true
+
+TEST COMPLETE
+

LayoutTests/fast/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items.html

@@ -0,0 +1,39 @@
+<!DOCTYPE html> <!-- webkit-test-runner [ useFlexibleViewport=true ] -->
+<meta charset="utf-8">
+<html>
+<head>
+<meta name="viewport" content="initial-scale=1">
+<script src="../../../../resources/ui-helper.js"></script>
+<script src="../../../../resources/js-test.js"></script>
+<script>
+jsTestIsAsync = true;
+
+addEventListener("load", async () => {
+    description("Verifies that dynamically-inserted image analysis context menu items show up after long-pressing on an image. This test requires WebKitTestRunner.");
+
+    if (!window.testRunner || !testRunner.runUIScript)
+        return;
+
+    await UIHelper.installFakeMachineReadableCodeResultsForImageAnalysis();
+    await UIHelper.longPressElement(document.querySelector("img"));
+
+    let foundQRCodeItem = false;
+    do {
+        await UIHelper.delayFor(200);
+        let contextMenu = await UIHelper.contextMenuItems();
+        if (contextMenu?.items?.includes("QR Code"))
+            foundQRCodeItem = true;
+    } while (!foundQRCodeItem);
+
+    testPassed("Found QR code menu item");
+    await UIHelper.activateAt(0, 0);
+    await UIHelper.waitForContextMenuToHide();
+    testPassed("Finished dismissing context menu");
+    finishJSTest();
+});
+</script>
+</head>
+<body>
+<img src="../../../images/resources/dice.png" width="320" height="240" alt="Dice">
+</body>
+</html>

LayoutTests/resources/ui-helper.js

@@ -1079,6 +1079,15 @@ window.UIHelper = class UIHelper {
         });
     }
 
+    static contextMenuItems()
+    {
+        return new Promise(resolve => {
+            testRunner.runUIScript(`(() => {
+                uiController.uiScriptComplete(JSON.stringify(uiController.contentsOfUserInterfaceItem('contextMenu')))
+            })()`, result => resolve(result ? JSON.parse(result).contextMenu : null));
+        });
+    }
+
     static setSelectedColorForColorPicker(red, green, blue)
     {
         const selectColorScript = `uiController.setSelectedColorForColorPicker(${red}, ${green}, ${blue})`;
@@ -1930,6 +1939,16 @@ window.UIHelper = class UIHelper {
         });
     }
 
+    static installFakeMachineReadableCodeResultsForImageAnalysis()
+    {
+        if (!this.isWebKit2())
+            return Promise.resolve();
+
+        return new Promise(resolve => {
+            testRunner.runUIScript("uiController.installFakeMachineReadableCodeResultsForImageAnalysis()", resolve);
+        });
+    }
+
     static moveToNextByKeyboardAccessoryBar()
     {
         return new Promise((resolve) => {

Source/WebKit/UIProcess/ios/WKContentViewInteraction.h

@@ -272,6 +272,12 @@ struct RemoveBackgroundData {
 
 enum class ProceedWithTextSelectionInImage : bool { No, Yes };
 
+enum class DynamicImageAnalysisContextMenuState : uint8_t {
+    NotWaiting,
+    WaitingForImageAnalysis,
+    WaitingForVisibleMenu,
+};
+
 enum class ImageAnalysisRequestIdentifierType { };
 using ImageAnalysisRequestIdentifier = ObjectIdentifier<ImageAnalysisRequestIdentifierType>;
 
@@ -565,7 +571,7 @@ struct ImageAnalysisContextMenuActionData {
     RetainPtr<NSString> _visualSearchPreviewTitle;
     CGRect _visualSearchPreviewImageBounds;
 #endif // USE(QUICK_LOOK)
-    BOOL _waitingForDynamicImageAnalysisContextMenuActions;
+    WebKit::DynamicImageAnalysisContextMenuState _dynamicImageAnalysisContextMenuState;
     std::optional<WebKit::ImageAnalysisContextMenuActionData> _imageAnalysisContextMenuActionData;
 #endif // ENABLE(IMAGE_ANALYSIS)
     uint32_t _fullscreenVideoImageAnalysisRequestIdentifier;

Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm

@@ -11840,7 +11840,7 @@ - (void)_setUpImageAnalysis
 #if ENABLE(IMAGE_ANALYSIS_ENHANCEMENTS)
     _removeBackgroundData = std::nullopt;
 #endif
-    _waitingForDynamicImageAnalysisContextMenuActions = NO;
+    _dynamicImageAnalysisContextMenuState = WebKit::DynamicImageAnalysisContextMenuState::NotWaiting;
     _imageAnalysisContextMenuActionData = std::nullopt;
 }
 
@@ -11861,7 +11861,7 @@ - (void)_tearDownImageAnalysis
     [self uninstallImageAnalysisInteraction];
     _removeBackgroundData = std::nullopt;
 #endif
-    _waitingForDynamicImageAnalysisContextMenuActions = NO;
+    _dynamicImageAnalysisContextMenuState = WebKit::DynamicImageAnalysisContextMenuState::NotWaiting;
     _imageAnalysisContextMenuActionData = std::nullopt;
 }
 
@@ -11954,7 +11954,7 @@ - (void)imageAnalysisGestureDidBegin:(WKImageAnalysisGestureRecognizer *)gesture
     _pendingImageAnalysisRequestIdentifier = requestIdentifier;
     _isProceedingWithTextSelectionInImage = NO;
     _elementPendingImageAnalysis = std::nullopt;
-    _waitingForDynamicImageAnalysisContextMenuActions = NO;
+    _dynamicImageAnalysisContextMenuState = WebKit::DynamicImageAnalysisContextMenuState::NotWaiting;
     _imageAnalysisContextMenuActionData = std::nullopt;
 
     WebKit::InteractionInformationRequest request { WebCore::roundedIntPoint([gestureRecognizer locationInView:self]) };
@@ -12058,7 +12058,7 @@ static BOOL shouldUseMachineReadableCodeMenuFromImageAnalysisResult(CocoaImageAn
 
 - (void)_completeImageAnalysisRequestForContextMenu:(CGImageRef)image requestIdentifier:(WebKit::ImageAnalysisRequestIdentifier)requestIdentifier hasTextResults:(BOOL)hasTextResults
 {
-    _waitingForDynamicImageAnalysisContextMenuActions = YES;
+    _dynamicImageAnalysisContextMenuState = WebKit::DynamicImageAnalysisContextMenuState::WaitingForImageAnalysis;
     _imageAnalysisContextMenuActionData = std::nullopt;
     [self _invokeAllActionsToPerformAfterPendingImageAnalysis:WebKit::ProceedWithTextSelectionInImage::No];
 
@@ -12068,52 +12068,11 @@ - (void)_completeImageAnalysisRequestForContextMenu:(CGImageRef)image requestIde
     auto weakSelf = WeakObjCPtr<WKContentView>(self);
     auto aggregator = MainRunLoopCallbackAggregator::create([weakSelf, data]() mutable {
         auto strongSelf = weakSelf.get();
-        if (!strongSelf || !strongSelf->_waitingForDynamicImageAnalysisContextMenuActions)
+        if (!strongSelf || strongSelf->_dynamicImageAnalysisContextMenuState == WebKit::DynamicImageAnalysisContextMenuState::NotWaiting)
             return;
 
-        strongSelf->_waitingForDynamicImageAnalysisContextMenuActions = NO;
         strongSelf->_imageAnalysisContextMenuActionData = WTFMove(*data);
-        [[strongSelf contextMenuInteraction] updateVisibleMenuWithBlock:makeBlockPtr([strongSelf](UIMenu *menu) -> UIMenu * {
-            __block auto indexOfPlaceholder = NSNotFound;
-            __block BOOL foundCopyItem = NO;
-            auto revealImageIdentifier = elementActionTypeToUIActionIdentifier(_WKElementActionTypeRevealImage);
-            auto copyIdentifier = elementActionTypeToUIActionIdentifier(_WKElementActionTypeCopy);
-            [menu.children enumerateObjectsUsingBlock:^(UIMenuElement *child, NSUInteger index, BOOL* stop) {
-                auto *action = dynamic_objc_cast<UIAction>(child);
-                if ([action.identifier isEqualToString:revealImageIdentifier] && (action.attributes & UIMenuElementAttributesHidden))
-                    indexOfPlaceholder = index;
-                else if ([action.identifier isEqualToString:copyIdentifier])
-                    foundCopyItem = YES;
-            }];
-
-            if (indexOfPlaceholder == NSNotFound)
-                return menu;
-
-            auto adjustedChildren = adoptNS(menu.children.mutableCopy);
-            auto replacements = adoptNS([NSMutableArray<UIMenuElement *> new]);
-
-            auto *elementInfo = [_WKActivatedElementInfo activatedElementInfoWithInteractionInformationAtPosition:strongSelf->_positionInformation userInfo:nil];
-            auto addAction = [&](_WKElementActionType action) {
-                auto *elementAction = [_WKElementAction _elementActionWithType:action info:elementInfo assistant:strongSelf->_actionSheetAssistant.get()];
-                [replacements addObject:[elementAction uiActionForElementInfo:elementInfo]];
-            };
-
-            if (foundCopyItem && [strongSelf copySubjectResultForImageContextMenu])
-                addAction(_WKElementActionTypeCopyCroppedImage);
-
-            if ([strongSelf hasSelectableTextForImageContextMenu])
-                addAction(_WKElementActionTypeImageExtraction);
-
-            if ([strongSelf hasVisualSearchResultsForImageContextMenu])
-                addAction(_WKElementActionTypeRevealImage);
-
-            if (UIMenu *subMenu = [strongSelf machineReadableCodeSubMenuForImageContextMenu])
-                [replacements addObject:subMenu];
-
-            RELEASE_LOG(ImageAnalysis, "Dynamically inserting %zu context menu action(s)", [replacements count]);
-            [adjustedChildren replaceObjectsInRange:NSMakeRange(indexOfPlaceholder, 1) withObjectsFromArray:replacements.get()];
-            return [menu menuByReplacingChildren:adjustedChildren.get()];
-        }).get()];
+        [strongSelf _insertDynamicImageAnalysisContextMenuItemsIfPossible];
     });
 
     auto request = [self createImageAnalyzerRequest:VKAnalysisTypeVisualSearch | VKAnalysisTypeMachineReadableCode | VKAnalysisTypeAppClip image:image];
@@ -12156,6 +12115,55 @@ - (void)_completeImageAnalysisRequestForContextMenu:(CGImageRef)image requestIde
 #endif
 }
 
+- (void)_insertDynamicImageAnalysisContextMenuItemsIfPossible
+{
+    bool updated = false;
+    [self.contextMenuInteraction updateVisibleMenuWithBlock:makeBlockPtr([&](UIMenu *menu) -> UIMenu * {
+        updated = true;
+        __block auto indexOfPlaceholder = NSNotFound;
+        __block BOOL foundCopyItem = NO;
+        auto revealImageIdentifier = elementActionTypeToUIActionIdentifier(_WKElementActionTypeRevealImage);
+        auto copyIdentifier = elementActionTypeToUIActionIdentifier(_WKElementActionTypeCopy);
+        [menu.children enumerateObjectsUsingBlock:^(UIMenuElement *child, NSUInteger index, BOOL* stop) {
+            auto *action = dynamic_objc_cast<UIAction>(child);
+            if ([action.identifier isEqualToString:revealImageIdentifier] && (action.attributes & UIMenuElementAttributesHidden))
+                indexOfPlaceholder = index;
+            else if ([action.identifier isEqualToString:copyIdentifier])
+                foundCopyItem = YES;
+        }];
+
+        if (indexOfPlaceholder == NSNotFound)
+            return menu;
+
+        auto adjustedChildren = adoptNS(menu.children.mutableCopy);
+        auto replacements = adoptNS([NSMutableArray<UIMenuElement *> new]);
+
+        auto *elementInfo = [_WKActivatedElementInfo activatedElementInfoWithInteractionInformationAtPosition:_positionInformation userInfo:nil];
+        auto addAction = [&](_WKElementActionType action) {
+            auto *elementAction = [_WKElementAction _elementActionWithType:action info:elementInfo assistant:_actionSheetAssistant.get()];
+            [replacements addObject:[elementAction uiActionForElementInfo:elementInfo]];
+        };
+
+        if (foundCopyItem && self.copySubjectResultForImageContextMenu)
+            addAction(_WKElementActionTypeCopyCroppedImage);
+
+        if (self.hasSelectableTextForImageContextMenu)
+            addAction(_WKElementActionTypeImageExtraction);
+
+        if (self.hasVisualSearchResultsForImageContextMenu)
+            addAction(_WKElementActionTypeRevealImage);
+
+        if (UIMenu *subMenu = self.machineReadableCodeSubMenuForImageContextMenu)
+            [replacements addObject:subMenu];
+
+        RELEASE_LOG(ImageAnalysis, "Dynamically inserting %zu context menu action(s)", [replacements count]);
+        [adjustedChildren replaceObjectsInRange:NSMakeRange(indexOfPlaceholder, 1) withObjectsFromArray:replacements.get()];
+        return [menu menuByReplacingChildren:adjustedChildren.get()];
+    }).get()];
+
+    _dynamicImageAnalysisContextMenuState = updated ? WebKit::DynamicImageAnalysisContextMenuState::NotWaiting : WebKit::DynamicImageAnalysisContextMenuState::WaitingForVisibleMenu;
+}
+
 - (void)imageAnalysisGestureDidFail:(WKImageAnalysisGestureRecognizer *)gestureRecognizer
 {
     [self _endImageAnalysisGestureDeferral:WebKit::ShouldPreventGestures::No];
@@ -13312,14 +13320,19 @@ - (void)_internalContextMenuInteraction:(UIContextMenuInteraction *)interaction
 - (UIAction *)placeholderForDynamicallyInsertedImageAnalysisActions
 {
 #if ENABLE(IMAGE_ANALYSIS)
-    if (_waitingForDynamicImageAnalysisContextMenuActions) {
+    switch (_dynamicImageAnalysisContextMenuState) {
+    case WebKit::DynamicImageAnalysisContextMenuState::NotWaiting:
+        return nil;
+    case WebKit::DynamicImageAnalysisContextMenuState::WaitingForImageAnalysis:
+    case WebKit::DynamicImageAnalysisContextMenuState::WaitingForVisibleMenu: {
         // FIXME: This placeholder item warrants its own unique item identifier; however, to ensure that internal clients
         // that already allow image analysis items (e.g. Mail) continue to allow this placeholder item, we reuse an existing
         // image analysis identifier.
         RetainPtr placeholder = [UIAction actionWithTitle:@"" image:nil identifier:elementActionTypeToUIActionIdentifier(_WKElementActionTypeRevealImage) handler:^(UIAction *) { }];
         [placeholder setAttributes:UIMenuElementAttributesHidden];
         return placeholder.autorelease();
     }
+    }
 #endif // ENABLE(IMAGE_ANALYSIS)
     return nil;
 }
@@ -13578,6 +13591,14 @@ - (void)contextMenuInteraction:(UIContextMenuInteraction *)interaction willDispl
             ASSERT_IMPLIES(strongSelf->_isDisplayingContextMenuWithAnimation, [strongSelf->_contextMenuHintContainerView window]);
             strongSelf->_isDisplayingContextMenuWithAnimation = NO;
             [strongSelf->_webView _didShowContextMenu];
+            switch (strongSelf->_dynamicImageAnalysisContextMenuState) {
+            case WebKit::DynamicImageAnalysisContextMenuState::NotWaiting:
+            case WebKit::DynamicImageAnalysisContextMenuState::WaitingForImageAnalysis:
+                break;
+            case WebKit::DynamicImageAnalysisContextMenuState::WaitingForVisibleMenu:
+                [strongSelf _insertDynamicImageAnalysisContextMenuItemsIfPossible];
+                break;
+            }
         }
     }];
 

Tools/TestRunnerShared/UIScriptContext/Bindings/UIScriptController.idl

@@ -396,4 +396,5 @@ interface UIScriptController {
     readonly attribute unsigned long countOfUpdatesWithLayerChanges;
 
     readonly attribute unsigned long long currentImageAnalysisRequestID;
+    undefined installFakeMachineReadableCodeResultsForImageAnalysis();
 };

Tools/TestRunnerShared/UIScriptContext/UIScriptController.h

@@ -392,6 +392,7 @@ class UIScriptController : public JSWrappable {
     // Image Analysis
 
     virtual uint64_t currentImageAnalysisRequestID() const { return 0; }
+    virtual void installFakeMachineReadableCodeResultsForImageAnalysis() { }
 
 protected:
     explicit UIScriptController(UIScriptContext&);

Tools/WebKitTestRunner/TestController.h

@@ -423,6 +423,8 @@ class TestController {
 
 #if ENABLE(IMAGE_ANALYSIS)
     static uint64_t currentImageAnalysisRequestID();
+    void installFakeMachineReadableCodeResultsForImageAnalysis();
+    bool shouldUseFakeMachineReadableCodeResultsForImageAnalysis() const;
 #endif
 
 private:
@@ -654,6 +656,10 @@ class TestController {
     Vector<std::unique_ptr<InstanceMethodSwizzler>> m_presentPopoverSwizzlers;
 #endif
 
+#if ENABLE(IMAGE_ANALYSIS)
+    bool m_useFakeMachineReadableCodeResultsForImageAnalysis { false };
+#endif
+
     enum State {
         Initial,
         Resetting,