Skip to content
Permalink
Browse files
[Cocoa] Web Automation: add SPI to tell whether the automation sessio…
…n is currently simulating user interactions

https://bugs.webkit.org/show_bug.cgi?id=178616

Reviewed by Joseph Pecoraro.

This is needed to disambiguate whether an action (such as selectAll:) came from
a user clicking on "Edit > Select All" in a menu or whether it was produced by
simulating the keystrokes to produce the chord for "Command + a". Some clients,
such as Safari, would allow the latter but not the former during automation.

* UIProcess/API/Cocoa/_WKAutomationSession.h:
* UIProcess/API/Cocoa/_WKAutomationSession.mm:
(-[_WKAutomationSession isSimulatingUserInteraction]):
Add new SPI property that's backed by the same WebAutomationSession method.

* UIProcess/Automation/WebAutomationSession.h:
* UIProcess/Automation/WebAutomationSession.cpp:
(WebKit::WebAutomationSession::keyboardEventsFlushedForPage):
(WebKit::WebAutomationSession::performMouseInteraction):
(WebKit::WebAutomationSession::performKeyboardInteractions):
Set m_simulatingUserInteraction prior to sending the synthesized events. It will
be cleared when keyboardEventsFlushedForPage() is called by WebPageProxy.

* UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::didReceiveEvent):
Notify the automation session that the key event queue was flushed *after* giving
delegates a chance to do something with the key event. This is necessary so that
any actions that are created from the NSEvent by the delegates are handled prior
to the automation session finishing its keyboard interaction command.


Canonical link: https://commits.webkit.org/194876@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@223880 268f45cc-cd09-0410-ab3c-d52691b4dbfc
  • Loading branch information
burg committed Oct 24, 2017
1 parent d097976 commit 092026a823a1c71c2c4548433505921e891076e6
Showing 6 changed files with 60 additions and 10 deletions.
@@ -1,3 +1,35 @@
2017-10-23 Brian Burg <bburg@apple.com>

[Cocoa] Web Automation: add SPI to tell whether the automation session is currently simulating user interactions
https://bugs.webkit.org/show_bug.cgi?id=178616

Reviewed by Joseph Pecoraro.

This is needed to disambiguate whether an action (such as selectAll:) came from
a user clicking on "Edit > Select All" in a menu or whether it was produced by
simulating the keystrokes to produce the chord for "Command + a". Some clients,
such as Safari, would allow the latter but not the former during automation.

* UIProcess/API/Cocoa/_WKAutomationSession.h:
* UIProcess/API/Cocoa/_WKAutomationSession.mm:
(-[_WKAutomationSession isSimulatingUserInteraction]):
Add new SPI property that's backed by the same WebAutomationSession method.

* UIProcess/Automation/WebAutomationSession.h:
* UIProcess/Automation/WebAutomationSession.cpp:
(WebKit::WebAutomationSession::keyboardEventsFlushedForPage):
(WebKit::WebAutomationSession::performMouseInteraction):
(WebKit::WebAutomationSession::performKeyboardInteractions):
Set m_simulatingUserInteraction prior to sending the synthesized events. It will
be cleared when keyboardEventsFlushedForPage() is called by WebPageProxy.

* UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::didReceiveEvent):
Notify the automation session that the key event queue was flushed *after* giving
delegates a chance to do something with the key event. This is necessary so that
any actions that are created from the NSEvent by the delegates are handled prior
to the automation session finishing its keyboard interaction command.

2017-10-23 Brian Burg <bburg@apple.com>

[Mac] Web Automation: key modifiers for synthesized NSEvents are incorrect
@@ -43,6 +43,8 @@ WK_CLASS_AVAILABLE(macosx(10.12), ios(10.0))
@property (nonatomic, weak) id <_WKAutomationSessionDelegate> delegate;
@property (nonatomic, readonly, getter=isPaired) BOOL paired;

@property (nonatomic, readonly, getter=isSimulatingUserInteraction) BOOL simulatingUserInteraction WK_API_AVAILABLE(macosx(WK_MAC_TBA), ios(WK_IOS_TBA));

- (instancetype)initWithConfiguration:(_WKAutomationSessionConfiguration *)configuration NS_DESIGNATED_INITIALIZER;

#if !TARGET_OS_IPHONE
@@ -97,6 +97,11 @@ - (BOOL)isPaired
return _session->isPaired();
}

- (BOOL)isSimulatingUserInteraction
{
return _session->isSimulatingUserInteraction();
}

#if PLATFORM(MAC)
- (BOOL)wasEventSynthesizedForAutomation:(NSEvent *)event
{
@@ -663,8 +663,12 @@ void WebAutomationSession::inspectorFrontendLoaded(const WebPageProxy& page)

void WebAutomationSession::keyboardEventsFlushedForPage(const WebPageProxy& page)
{
if (auto callback = m_pendingKeyboardEventsFlushedCallbacksPerPage.take(page.pageID()))
if (auto callback = m_pendingKeyboardEventsFlushedCallbacksPerPage.take(page.pageID())) {
callback->sendSuccess(InspectorObject::create());

if (m_pendingKeyboardEventsFlushedCallbacksPerPage.isEmpty())
m_simulatingUserInteraction = false;
}
}

void WebAutomationSession::willClosePage(const WebPageProxy& page)
@@ -1284,7 +1288,7 @@ static WebEvent::Modifiers protocolModifierToWebEventModifier(Inspector::Protoco

RELEASE_ASSERT_NOT_REACHED();
}
#endif // USE(APPKIT)
#endif // USE(APPKIT) || PLATFORM(GTK)

void WebAutomationSession::performMouseInteraction(Inspector::ErrorString& errorString, const String& handle, const Inspector::InspectorObject& requestedPositionObject, const String& mouseButtonString, const String& mouseInteractionString, const Inspector::InspectorArray& keyModifierStrings, Ref<PerformMouseInteractionCallback>&& callback)
{
@@ -1338,7 +1342,7 @@ void WebAutomationSession::performMouseInteraction(Inspector::ErrorString& error
.setY(y - page->topContentInset())
.release());
});
#endif // USE(APPKIT)
#endif // USE(APPKIT) || PLATFORM(GTK)
}

void WebAutomationSession::performKeyboardInteractions(ErrorString& errorString, const String& handle, const Inspector::InspectorArray& interactions, Ref<PerformKeyboardInteractionsCallback>&& callback)
@@ -1411,9 +1415,12 @@ void WebAutomationSession::performKeyboardInteractions(ErrorString& errorString,
callbackInMap->sendFailure(STRING_FOR_PREDEFINED_ERROR_NAME(Timeout));
callbackInMap = WTFMove(callback);

// This is cleared when all keyboard events are flushed.
m_simulatingUserInteraction = true;

for (auto& action : actionsToPerform)
action();
#endif // PLATFORM(COCOA)
#endif // PLATFORM(COCOA) || PLATFORM(GTK)
}

void WebAutomationSession::takeScreenshot(ErrorString& errorString, const String& handle, const String* optionalFrameHandle, const String* optionalNodeHandle, const bool* optionalScrollIntoViewIfNeeded, Ref<TakeScreenshotCallback>&& callback)
@@ -156,6 +156,7 @@ class WebAutomationSession final : public API::ObjectImpl<API::Object::Type::Aut
#endif

// Event Simulation Support.
bool isSimulatingUserInteraction() const { return m_simulatingUserInteraction; }
#if PLATFORM(MAC)
bool wasEventSynthesizedForAutomation(NSEvent *);
void markEventAsSynthesizedForAutomation(NSEvent *);
@@ -259,6 +260,8 @@ class WebAutomationSession final : public API::ObjectImpl<API::Object::Type::Aut

bool m_permissionForGetUserMedia { true };

// True if a synthesized key event is still being processed.
bool m_simulatingUserInteraction { false };

// Keep track of currently active modifiers across multiple keystrokes.
// Most platforms do not track current modifiers from synthesized events.
@@ -5085,20 +5085,21 @@ void WebPageProxy::didReceiveEvent(uint32_t opaqueType, bool handled)
if (!m_keyEventQueue.isEmpty()) {
LOG(KeyHandling, " UI process: sent keyEvent from didReceiveEvent");
m_process->send(Messages::WebPage::KeyEvent(m_keyEventQueue.first()), m_pageID);
} else {
if (auto* automationSession = process().processPool().automationSession())
automationSession->keyboardEventsFlushedForPage(*this);
}

// The call to doneWithKeyEvent may close this WebPage.
// Protect against this being destroyed.
Ref<WebPageProxy> protect(*this);

m_pageClient.doneWithKeyEvent(event, handled);
if (handled)
break;
if (!handled)
m_uiClient->didNotHandleKeyEvent(this, event);

m_uiClient->didNotHandleKeyEvent(this, event);
// Notify the session after -[NSApp sendEvent:] has a crack at turning the event into an action.
if (m_keyEventQueue.isEmpty()) {
if (auto* automationSession = process().processPool().automationSession())
automationSession->keyboardEventsFlushedForPage(*this);
}
break;
}
#if ENABLE(MAC_GESTURE_EVENTS)

0 comments on commit 092026a

Please sign in to comment.