Avoid UIA event flooding in NvDA by moving UIA event handler COM obje…

…cts into c++ (nvaccess#14888) If NVDA is flooded with UI Automation events, such as textChange events from Windows consoles, NVDA may seem to freeze, and or UIA focus tracking completely stops. Description of user facing changes NVDA should remain responsive when being flooded with many UI Automation events, E.g. large amounts of text being written to a Windows console. Description of development approach Implement a new UI Automation event handler COM object in c++ which avoids calling into Python, and instead stores the events, and requests NvDA's main thread to later wake and read off the events when it can. This COM object also coalesces duplicate automation and propertyChange events for the same element, so that NVDA is no longer flooded with duplicate events, E.g. textChange events in Windows consoles. The path for a UI Automation event is now as follows: • In an MTA thread, the new c++ UIA event handler COM object receives the event via its handle*Event method from UI Automation core. • The event is stored in a list of events. • If the event type supports coalescing (automation and propertyChange events), An existing duplicate event is looked up in a map, and if one exists, the existing event is removed from the list, and the map entry is then pointed to the newest event. • If there was no existing event, a new entry is added to the map for this event. • In other words: the events are stored in the list in order they were received, but any duplicate events are removed, leaving only the latest ones. Ecentually an ordered dictionary. • If this event type does not support coalescing (E.g. focusChange event), NVDA is requested to wake and flush the event queue on its main thread as soon as it can. • If this event does support coalescing, and this is the first event since the last flush, then NVDA is requested to wake and flush the event queue in around 30 ms, giving time for more events to possibly be received and coalesced. • When NvDA wakes on its main thread, It requests the rate limited UIA event handler to flush its event list, emitting all the stored events out to our original Python UIA event handler COM object for normal handling. These changes mean that UIA core in the MTA is never blocked waiting for the Python GIL, and from its perspective, event handling is instantanious. Thus UIA core should never feel the need to kill off events, including focus change events. This pr also provides a couple of extra optimizations for UIA handling in Python: • UIAHandler events: avoid needlessly creating an NVDAObject when the event is for the focus object. • UIA event handlers: pass windowHandle into NvDAObject constructor if we already have it.
josephsl · Nov 27, 2023 · dd8a44c · dd8a44c
1 parent 251b0b0
commit dd8a44c
Show file tree

Hide file tree

Showing 14 changed files with 728 additions and 67 deletions.
diff --git a/nvdaHelper/local/UIAEventLimiter/api.cpp b/nvdaHelper/local/UIAEventLimiter/api.cpp
@@ -0,0 +1,34 @@
+/*
+This file is a part of the NVDA project.
+URL: http://github.com/nvaccess/nvda/
+Copyright 2023 NV Access Limited.
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License version 2.0, as published by
+    the Free Software Foundation.
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+This license can be found at:
+http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+*/
+
+#include <windows.h>
+#include <common/log.h>
+#include "rateLimitedEventHandler.h"
+
+HRESULT rateLimitedUIAEventHandler_create(IUnknown* pExistingHandler, RateLimitedEventHandler** ppRateLimitedEventHandler) {
+	LOG_DEBUG(L"rateLimitedUIAEventHandler_create called");
+	if(!pExistingHandler || !ppRateLimitedEventHandler) {
+		LOG_ERROR(L"rateLimitedUIAEventHandler_create: one or more NULL arguments"); 
+		return E_INVALIDARG;
+	}
+
+	// Create the RateLimitedEventHandler instance
+	*ppRateLimitedEventHandler = new RateLimitedEventHandler(pExistingHandler);
+	if (!(*ppRateLimitedEventHandler)) {
+		LOG_ERROR(L"rateLimitedUIAEventHandler_create: Could not create RateLimitedUIAEventHandler. Returning");
+		return E_OUTOFMEMORY;
+	}
+	LOG_DEBUG(L"rateLimitedUIAEventHandler_create: done");
+	return S_OK;
+}
diff --git a/nvdaHelper/local/UIAEventLimiter/eventRecord.h b/nvdaHelper/local/UIAEventLimiter/eventRecord.h
@@ -0,0 +1,107 @@
+/*
+This file is a part of the NVDA project.
+URL: http://github.com/nvaccess/nvda/
+Copyright 2023 NV Access Limited.
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License version 2.0, as published by
+    the Free Software Foundation.
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+This license can be found at:
+http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+*/
+
+#pragma once
+
+#include <vector>
+#include <variant>
+#include <atlcomcli.h>
+#include <UIAutomation.h>
+#include "utils.h"
+
+// The following structs are used to represent UI Automation event params.
+// Apart from holding the params,
+// each struct also contains a method for generating a comparison key
+// which is used to detect and remove duplicate events.
+// The key is made up of the element's runtime ID,
+// plus any extra event params that make the event unique,
+// E.g. event ID, property ID etc. 
+
+struct AutomationEventRecord_t {
+	CComPtr<IUIAutomationElement> sender;
+	EVENTID eventID;
+	std::vector<int> generateCoalescingKey() const {
+		auto key = getRuntimeIDFromElement(sender);
+		key.push_back(eventID);
+		return key;
+	}
+};
+
+struct PropertyChangedEventRecord_t {
+	CComPtr<IUIAutomationElement> sender;
+	PROPERTYID propertyID;
+	CComVariant newValue;
+	std::vector<int> generateCoalescingKey() const {
+		auto key = getRuntimeIDFromElement(sender);
+		key.push_back(UIA_AutomationPropertyChangedEventId);
+		key.push_back(propertyID);
+		return key;
+	}
+};
+
+struct FocusChangedEventRecord_t {
+	CComPtr<IUIAutomationElement> sender;
+	std::vector<int> generateCoalescingKey() const {
+		auto key = getRuntimeIDFromElement(sender);
+		key.push_back(UIA_AutomationFocusChangedEventId);
+		return key;
+	}
+};
+
+struct NotificationEventRecord_t {
+	CComPtr<IUIAutomationElement> sender;
+	NotificationKind notificationKind;
+	NotificationProcessing notificationProcessing;
+	CComBSTR displayString;
+	CComBSTR activityID;
+	std::vector<int> generateCoalescingKey() const {
+		auto key = getRuntimeIDFromElement(sender);
+		key.push_back(UIA_NotificationEventId);
+		key.push_back(notificationKind);
+		key.push_back(notificationProcessing);
+		// Include the activity ID in the key also,
+		// by converting it to a sequence of ints.
+		if(activityID.m_str) {
+			for(int c: std::wstring_view(activityID.m_str)) {
+				key.push_back(c);
+			}
+		} else {
+			key.push_back(0);
+		}
+		return key;
+	}
+};
+
+struct ActiveTextPositionChangedEventRecord_t {
+	CComPtr<IUIAutomationElement> sender;
+	CComPtr<IUIAutomationTextRange> range;
+	std::vector<int> generateCoalescingKey() const {
+		auto key = getRuntimeIDFromElement(sender);
+		key.push_back(UIA_ActiveTextPositionChangedEventId);
+		return key;
+	}
+};
+
+// @brief a variant type that holds all possible UI Automation event records we support.
+using EventRecordVariant_t = std::variant<AutomationEventRecord_t, FocusChangedEventRecord_t, PropertyChangedEventRecord_t, NotificationEventRecord_t, ActiveTextPositionChangedEventRecord_t>;
+
+// @brief A concept to be used with the above event record types
+// that ensures the type has a generateCoalescingKey method,
+// and that the type appears in the EventRecordVariant_t type. 
+template<typename T>
+concept EventRecordConstraints = requires(T t) {
+	{ t.generateCoalescingKey() } -> std::same_as<std::vector<int>>;
+	// The type must be supported by the EventRecordVariant_t variant type
+	 requires supports_alternative<T, EventRecordVariant_t>;
+};
diff --git a/nvdaHelper/local/UIAEventLimiter/rateLimitedEventHandler.cpp b/nvdaHelper/local/UIAEventLimiter/rateLimitedEventHandler.cpp
@@ -0,0 +1,248 @@
+/*
+This file is a part of the NVDA project.
+URL: http://github.com/nvaccess/nvda/
+Copyright 2023 NV Access Limited.
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License version 2.0, as published by
+    the Free Software Foundation.
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+This license can be found at:
+http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+*/
+
+#include <windows.h>
+#include <uiautomation.h>
+#include <numeric>
+#include <variant>
+#include <map>
+#include <vector>
+#include <functional>
+#include <mutex>
+#include <atomic>
+#include <atlcomcli.h>
+#include <comutil.h>
+#include <common/log.h>
+#include "eventRecord.h"
+#include "rateLimitedEventHandler.h"
+
+template<EventRecordConstraints EventRecordClass, typename... EventRecordArgTypes>
+HRESULT RateLimitedEventHandler::queueEvent(EventRecordArgTypes&&... args) {
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::queueEvent called");
+	bool needsFlush = false;
+	{ // scoped lock
+		std::lock_guard lock(m_mtx);
+		// work out whether we need to request a flush after inserting this event.
+		needsFlush = m_eventRecords.empty();
+		LOG_DEBUG(L"RateLimitedUIAEventHandler::queueEvent: Inserting new event");
+		auto& recordVar = m_eventRecords.emplace_back(std::in_place_type_t<EventRecordClass>{}, args...);
+		auto recordVarIter = m_eventRecords.end();
+		recordVarIter--;
+		auto& record = std::get<EventRecordClass>(recordVar);
+		auto coalescingKey = record.generateCoalescingKey();
+		auto existingKeyIter = m_eventRecordsByKey.find(coalescingKey);
+		if(existingKeyIter != m_eventRecordsByKey.end()) {
+			LOG_DEBUG(L"RateLimitedUIAEventHandler::queueEvent: found existing event with same key"); 
+			auto& [existingRecordVarIter,existingCoalesceCount] = existingKeyIter->second;
+			LOG_DEBUG(L"RateLimitedUIAEventHandler::queueEvent: updating key and count to "<<(existingCoalesceCount+1));
+			existingKeyIter->second = {recordVarIter, existingCoalesceCount + 1};
+			LOG_DEBUG(L"RateLimitedUIAEventHandler::queueEvent: erasing old item"); 
+			m_eventRecords.erase(existingRecordVarIter);
+		} else {
+			LOG_DEBUG(L"RateLimitedUIAEventHandler::queueEvent: Adding key");
+			m_eventRecordsByKey.insert_or_assign(coalescingKey, std::pair(recordVarIter, 1));
+		}
+		if(needsFlush) {
+			LOG_DEBUG(L"RateLimitedUIAEventHandler::queueEvent: requesting flush");
+			m_needsFlush = true;
+			m_flushConditionVar.notify_one();
+		}
+	} // m_mtx released. 
+	return S_OK;
+}
+
+void RateLimitedEventHandler::flusherThreadFunc(std::stop_token stopToken) {
+	LOG_DEBUG(L"flusherThread started");
+	// If this thread is requested to stop, we need to ensure we wake up.
+	std::stop_callback stopCallback{stopToken, [this](){
+		this->m_flushConditionVar.notify_all();
+	}};
+	do { // thread main loop
+		LOG_DEBUG(L"flusherThreadFunc sleeping...");
+		{ std::unique_lock lock(m_mtx);
+			// sleep until a flush is needed or this thread should stop.
+			m_flushConditionVar.wait(lock, [this, stopToken](){
+				return this->m_needsFlush || stopToken.stop_requested();
+			});
+			LOG_DEBUG(L"flusherThread woke up");
+			if(stopToken.stop_requested()) {
+				LOG_DEBUG(L"flusherThread returning as stop requested"); 
+				return;
+			}
+			m_needsFlush = false;
+		} // m_mtx released here.
+		flushEvents();
+	} while(!stopToken.stop_requested());
+	LOG_DEBUG(L"flusherThread returning");
+}
+
+HRESULT RateLimitedEventHandler::emitEvent(const AutomationEventRecord_t& record) const {
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::emitAutomationEvent called");
+	if(!m_pExistingAutomationEventHandler) {
+		LOG_ERROR(L"RateLimitedUIAEventHandler::emitAutomationEvent: interface not supported.");
+		return E_NOINTERFACE;
+	}
+	LOG_DEBUG(L"Emitting automationEvent for eventID "<<record.eventID);
+	return m_pExistingAutomationEventHandler->HandleAutomationEvent(record.sender, record.eventID);
+}
+
+HRESULT RateLimitedEventHandler::emitEvent(const FocusChangedEventRecord_t& record) const {
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::emitFocusChangedEvent called");
+	if(!m_pExistingFocusChangedEventHandler) {
+		LOG_ERROR(L"RateLimitedUIAEventHandler::emitFocusChangedEvent: interface not supported.");
+		return E_NOINTERFACE;
+	}
+	LOG_DEBUG(L"Emitting focus changed event");
+	return m_pExistingFocusChangedEventHandler->HandleFocusChangedEvent(record.sender);
+}
+
+HRESULT RateLimitedEventHandler::emitEvent(const PropertyChangedEventRecord_t& record) const {
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::emitPropertyChangedEvent called");
+	if(!m_pExistingPropertyChangedEventHandler) {
+		LOG_ERROR(L"RateLimitedUIAEventHandler::emitPropertyChangedEvent: interface not supported.");
+		return E_NOINTERFACE;
+	}
+	LOG_DEBUG(L"Emitting property changed event for property "<<(record.propertyID));
+	return m_pExistingPropertyChangedEventHandler->HandlePropertyChangedEvent(record.sender, record.propertyID, record.newValue);
+}
+
+HRESULT RateLimitedEventHandler::emitEvent(const NotificationEventRecord_t& record) const {
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::emitNotificationEvent called");
+	if(!m_pExistingNotificationEventHandler) {
+		LOG_ERROR(L"RateLimitedUIAEventHandler::emitNotificationChangedEvent: interface not supported.");
+		return E_NOINTERFACE;
+	}
+	LOG_DEBUG(L"Emitting notification event");
+	return m_pExistingNotificationEventHandler->HandleNotificationEvent(record.sender, record.notificationKind, record.notificationProcessing, record.displayString, record.activityID);
+}
+
+HRESULT RateLimitedEventHandler::emitEvent(const ActiveTextPositionChangedEventRecord_t& record) const {
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::emitActiveTextPositionChangedEvent called");
+	if(!m_pExistingActiveTextPositionChangedEventHandler) {
+		LOG_ERROR(L"RateLimitedUIAEventHandler::emitActiveTextPositionChangedEvent: interface not supported.");
+		return E_NOINTERFACE;
+	}
+	LOG_DEBUG(L"Emitting active text position changed event");
+	return m_pExistingActiveTextPositionChangedEventHandler->HandleActiveTextPositionChangedEvent(record.sender, record.range);
+}
+
+RateLimitedEventHandler::~RateLimitedEventHandler() {
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::~RateLimitedUIAEventHandler called");
+}
+
+RateLimitedEventHandler::RateLimitedEventHandler(IUnknown* pExistingHandler):
+	m_pExistingAutomationEventHandler(pExistingHandler),
+	m_pExistingFocusChangedEventHandler(pExistingHandler), 
+	m_pExistingPropertyChangedEventHandler(pExistingHandler),
+	m_pExistingNotificationEventHandler(pExistingHandler), 
+	m_pExistingActiveTextPositionChangedEventHandler(pExistingHandler),
+	m_flusherThread([this](std::stop_token st){ this->flusherThreadFunc(st); })
+{
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::RateLimitedUIAEventHandler called");
+}
+
+// IUnknown methods
+ULONG STDMETHODCALLTYPE RateLimitedEventHandler::AddRef() {
+	auto refCount = InterlockedIncrement(&m_refCount);
+	LOG_DEBUG(L"AddRef: "<<refCount)
+	return refCount;
+}
+
+ULONG STDMETHODCALLTYPE RateLimitedEventHandler::Release() {
+	auto refCount = InterlockedDecrement(&m_refCount);
+	if (refCount == 0) {
+		delete this;
+	}
+	LOG_DEBUG(L"Release: "<<refCount)
+	return refCount;
+}
+
+HRESULT STDMETHODCALLTYPE RateLimitedEventHandler::QueryInterface(REFIID riid, void** ppInterface) {
+	if (riid == __uuidof(IUnknown)) {
+		*ppInterface = static_cast<IUIAutomationEventHandler*>(this);
+		AddRef();
+		return S_OK;
+	} else if (riid == __uuidof(IUIAutomationEventHandler)) {
+		*ppInterface = static_cast<IUIAutomationEventHandler*>(this);
+		AddRef();
+		return S_OK;
+	} else if (riid == __uuidof(IUIAutomationFocusChangedEventHandler)) {
+		*ppInterface = static_cast<IUIAutomationFocusChangedEventHandler*>(this);
+		AddRef();
+		return S_OK;
+	} else if (riid == __uuidof(IUIAutomationPropertyChangedEventHandler)) {
+		*ppInterface = static_cast<IUIAutomationPropertyChangedEventHandler*>(this);
+		AddRef();
+		return S_OK;
+	} else if (riid == __uuidof(IUIAutomationNotificationEventHandler)) {
+		*ppInterface = static_cast<IUIAutomationNotificationEventHandler*>(this);
+		AddRef();
+		return S_OK;
+	} else if (riid == __uuidof(IUIAutomationActiveTextPositionChangedEventHandler)) {
+		*ppInterface = static_cast<IUIAutomationActiveTextPositionChangedEventHandler*>(this);
+		AddRef();
+		return S_OK;
+	}
+	*ppInterface = nullptr;
+	return E_NOINTERFACE;
+}
+
+// IUIAutomationEventHandler method
+HRESULT STDMETHODCALLTYPE RateLimitedEventHandler::HandleAutomationEvent(IUIAutomationElement* sender, EVENTID eventID) {
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::HandleAutomationEvent called");
+	LOG_DEBUG(L"Queuing automationEvent for eventID "<<eventID);
+	return queueEvent<AutomationEventRecord_t>(sender, eventID);
+}
+
+// IUIAutomationFocusEventHandler method
+HRESULT STDMETHODCALLTYPE RateLimitedEventHandler::HandleFocusChangedEvent(IUIAutomationElement* sender) {
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::HandleFocusChangedEvent called");
+	return queueEvent<FocusChangedEventRecord_t>(sender);
+}
+
+// IUIAutomationPropertyChangedEventHandler method
+HRESULT STDMETHODCALLTYPE RateLimitedEventHandler::HandlePropertyChangedEvent(IUIAutomationElement* sender, PROPERTYID propertyID, VARIANT newValue) {
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::HandlePropertyChangedEvent called");
+	return queueEvent<PropertyChangedEventRecord_t>(sender, propertyID, newValue);
+}
+
+// IUIAutomationNotificationEventHandler method
+HRESULT STDMETHODCALLTYPE RateLimitedEventHandler::HandleNotificationEvent(IUIAutomationElement* sender, NotificationKind notificationKind, NotificationProcessing notificationProcessing, BSTR displayString, BSTR activityID) {
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::HandleNotificationEvent called");
+	return queueEvent<NotificationEventRecord_t>(sender, notificationKind, notificationProcessing, displayString, activityID);
+}
+
+// IUIAutomationActiveTextPositionchangedEventHandler method
+HRESULT STDMETHODCALLTYPE RateLimitedEventHandler::HandleActiveTextPositionChangedEvent(IUIAutomationElement* sender, IUIAutomationTextRange* range) {
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::HandleActiveTextPositionChangedEvent called");
+	return queueEvent<ActiveTextPositionChangedEventRecord_t>(sender, range);
+}
+
+void RateLimitedEventHandler::flushEvents() {
+	LOG_DEBUG(L"RateLimitedEventHandler::flushEvents called");
+	decltype(m_eventRecords) eventRecordsCopy;
+	decltype(m_eventRecordsByKey) eventRecordsByKeyCopy;
+	{ std::lock_guard lock{m_mtx};
+		eventRecordsCopy.swap(m_eventRecords);
+		eventRecordsByKeyCopy.swap(m_eventRecordsByKey);
+	} // m_mtx released here.
+	// Emit events
+	LOG_DEBUG(L"RateLimitedUIAEventHandler::flusherThreadFunc: Emitting events...");
+	for(const auto& recordVar: eventRecordsCopy) {
+		std::visit([this](const auto& record) {
+			this->emitEvent(record);
+		}, recordVar);
+	}
+		LOG_DEBUG(L"RateLimitedUIAEventHandler::flusherThreadFunc: Done emitting events");
+}