-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[iOS] Image analysis menu items for images with machine-readable codes sometimes fail to appear #21068
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
EWS run on current version of this PR (hash 86ac200) |
megangardner
approved these changes
Nov 29, 2023
aprotyas
reviewed
Nov 29, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
…s sometimes fail to appear https://bugs.webkit.org/show_bug.cgi?id=265544 rdar://118540193 Reviewed by Megan Gardner. Our current logic for dynamically inserting image analysis menu items (such as "Look Up" and any menu items for machine-readable codes) works like this: 1. We kick off image analysis for machine-readable codes and visual search when the context menu is about to be presented. The menu is presented with a hidden placeholder menu item, which represents the set of dynamically-inserted image analysis items we might insert in the future. 2. When the image analysis started by (1) is done, we use `-updateVisibleMenuWithBlock:` on the context menu interaction to replace the placeholder item above with the final set of image analysis items (only if the WebKit client didn't remove the placeholder menu item). However, there's a subtle race here, where image analysis may complete before the context menu has a visible menu. In this scenario, the call to `-updateVisibleMenuWithBlock:` returns early without calling the update block, and we end up missing our only chance to swap out the placeholder item for the actual image analysis items. To fix this, we detect the case where our image analysis completes before the menu is visible, and try again in `-contextMenuInteraction:willDisplayMenuForConfiguration:animator:`, after the menu is guaranteed to be visible. Test: fast/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items.html * LayoutTests/fast/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items-expected.txt: Added. * LayoutTests/fast/events/touch/ios/long-press-on-image-with-machine-readable-code-menu-items.html: Added. Add a new layout test, along with some additional test infrastructure to exercise this fix. * LayoutTests/resources/ui-helper.js: Add a way to simulate MRC menu item results when long-pressing on an image in layout tests, on iOS. (window.UIHelper.contextMenuItems): (window.UIHelper.installFakeMachineReadableCodeResultsForImageAnalysis): * Source/WebKit/UIProcess/ios/WKContentViewInteraction.h: * Source/WebKit/UIProcess/ios/WKContentViewInteraction.mm: (-[WKContentView _setUpImageAnalysis]): (-[WKContentView _tearDownImageAnalysis]): (-[WKContentView imageAnalysisGestureDidBegin:]): Replace the extant boolean flag `_waitingForDynamicImageAnalysisContextMenuActions` with a tri-state enum type, `_dynamicImageAnalysisContextMenuState`, which allows us to appropriately deal with the corner case where image analysis completes before the context menu has any visible items. See above for more details. (-[WKContentView _completeImageAnalysisRequestForContextMenu:requestIdentifier:hasTextResults:]): (-[WKContentView _insertDynamicImageAnalysisContextMenuItemsIfPossible]): Pull the existing logic for updating visible context menu items to include the dynamically-inserted image analysis items into a separate helper method, so that we can invoke it after the context menu is done presenting. If the update block runs successfully (i.e. the menu is visible), then we transition directly to the `NotWaiting` state and we're done. However, if the update block never runs when calling into `-updateVisibleMenuWithBlock:`, we instead transition to `WaitingForVisibleMenu` and wait until the menu is done presenting, before trying to perform the update again. (-[WKContentView placeholderForDynamicallyInsertedImageAnalysisActions]): (-[WKContentView contextMenuInteraction:willDisplayMenuForConfiguration:animator:]): Update the menu to include MRC menu items (and other dynamically-inserted image analysis items) if needed — note that we'll only perform the update here, in the case where we failed to run the update block at all, in `-_completeImageAnalysisRequestForContextMenu:requestIdentifier:hasTextResults:`. * Tools/TestRunnerShared/UIScriptContext/Bindings/UIScriptController.idl: * Tools/TestRunnerShared/UIScriptContext/UIScriptController.h: (WTR::UIScriptController::installFakeMachineReadableCodeResultsForImageAnalysis): * Tools/WebKitTestRunner/TestController.h: * Tools/WebKitTestRunner/cocoa/TestControllerCocoa.mm: (fakeMachineReadableCodeMenuForTesting): (-[FakeMachineReadableCodeImageAnalysis allLines]): (-[FakeMachineReadableCodeImageAnalysis hasResultsForAnalysisTypes:]): (-[FakeMachineReadableCodeImageAnalysis mrcMenu]): Add a helper class that represents a fake VisionKit image analysis result that contains MRC menu items (for simplicity, there's only one hard-coded item titled "QR code"). (swizzledProcessImageAnalysisRequest): (WTR::TestController::installFakeMachineReadableCodeResultsForImageAnalysis): (WTR::TestController::shouldUseFakeMachineReadableCodeResultsForImageAnalysis const): (WTR::TestController::cocoaResetStateToConsistentValues): * Tools/WebKitTestRunner/cocoa/UIScriptControllerCocoa.h: * Tools/WebKitTestRunner/cocoa/UIScriptControllerCocoa.mm: (WTR::UIScriptControllerCocoa::installFakeMachineReadableCodeResultsForImageAnalysis): Canonical link: https://commits.webkit.org/271317@main
86ac200
to
48c8c31
Compare
Committed 271317@main (48c8c31): https://commits.webkit.org/271317@main Reviewed commits have been landed. Closing PR #21068 and removing active labels. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Platform
Portability improvements and other general platform improvements not driven directly by site bugs.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
48c8c31
86ac200
🧪 wpe-wk2🛠 watch