Use IME/accessibility to find out about editable fields

[The-Compiler]

Currently we use a heuristic to find out whether editable elements are clicked. We might be able to hook into a QInputMethodEvent instead (though Qt 5.8 with QtWebEngine has some issues there...)

[The-Compiler]

From @amosbird

        focusObject = QApplication.focusObject()
        if not focusObject:
            log.misc.info('bad')
            return

        query = QInputMethodQueryEvent(Qt.ImCursorRectangle);
        QApplication.sendEvent(focusObject, query);
        r = query.value(Qt.ImCursorRectangle);
        if r:
            log.misc.info('good')
        else:
            log.misc.info('bad')

If that works reliably, it might be able to replace a lot of custom logic with better results. Ideally, we'd even have a way to get a signal on changes without having to turn on accessibility (which IIRC results in QtWebEngine slowing down significantly and might not even be exposed to PyQt/qutebrowser).

Certainly invites some more investigation sooner rather than later.

[The-Compiler]

I digged into this a bit more today, and probably found the holy grail 🎉

Turns out, we don't actually have to implement a custom IME to get events from Qt when a cursor appears/disappears! Instead, we can access the existing QApplication::inputMethod() and use its cursorRectangleChanged signal. In the debug console:

from PyQt5.QtWidgets import QApplication
QApplication.inputMethod().cursorRectangleChanged.connect(lambda: print(QApplication.inputMethod().cursorRectangle()))

This is likely going to cause a couple of false-positives (when the cursor is moved due to relayouting and such?), but I suppose we could do something like queryFocusObject(Qt.ImHints) and only continue switching insert mode if those hints changed or something. Qt.ImEnabled sounds good but I think that isn't set on password input fields. It also triggers in any Qt widgets, so it probably needs an additional check to see whether we have a web view focused.

Either way: This would probably allow us to have insert mode entered as soon as a blinking cursor appears somewhere, while getting rid of a lot of heuristics!

Open questions:

• Does this work with older Qt versions? I remember seeing some IME bugs in Qt's bugtracker.
• Does this work with custom IMEs (ibus/fcitx/...) enabled?
• Does this work on Windows and macOS?

Resources:

• IME handling in RenderWidgetHostViewQt
  • Note: Has been refactored a bit
  • Refactored IME handling in RenderWidgetHostViewQtDelegateClient
• IME handling in RenderWidgetHostViewQtDelegateWidget
• Qt Input Method - In Depth
• QInputMethod

[The-Compiler]

10:54 <amosbird> The-Compiler: Hi, I've been playing around with that IME trick for a while. It works in almost all cases, even for password fields (I'm using fcitx). There is only one issue remaining: When some input fields get focused (cursor blinks), there isn't any output of PyQt5.QtCore.QRectF, or just PyQt5.QtCore.QRectF(). It's only after a character is typed in then something like
10:54 <amosbird> PyQt5.QtCore.QRectF(725.0, 317.0, 1.0, 22.0) is printed.
10:54 <amosbird> For instance, when opening baidu.com, cursor starts blinking directly, but there is no output
10:55 <amosbird> also, when in google.com, using "gi" to focus the search field, qb only prints PyQt5.QtCore.QRectF()

Might be possible to just use cursorRectangleChanged as an event, but then use queryFocusObject to get e.g. the flags to find out more?

[toofar]

I've been testing a few of my absolute favourite web assets with the cursor rectangle changed event. It works pretty well. Beyond that I have two heuristics.

1. For preventing clicking on widgets not on the page affecting insert mode; checking the anchorRectangle() seems to work so far because proper widgets set it but webengine doesn't (I would want to compare with a "examine the focused widget" and compare across versions before proposing it for real though).
2. There are a couple of cases where I found that querying the ImEnabled InputMethodQueryEvent before leaving insert mode helped: 1. pressing enter while in a text field on confluence and jira (including in comments on the QT jira instance) and 2. tabbing between table cells in confluence and https://html-online.com/editor/ (maybe related to "rogue tab keypresses"? I tested on 5.14.2. The reported focus object for that event is the same as when you click on a non-editable element)

very WIP patch

diff --git i/qutebrowser/browser/eventfilter.py w/qutebrowser/browser/eventfilter.py
index 002949a2b5ca..84911d343cf1 100644
--- i/qutebrowser/browser/eventfilter.py
+++ w/qutebrowser/browser/eventfilter.py
@@ -235,17 +235,6 @@ class TabEventFilter(QObject):
             self._check_insertmode_on_release = True
             return
 
-        if elem.is_editable():
-            log.mouse.debug("Clicked editable element!")
-            if config.val.input.insert_mode.auto_enter:
-                modeman.enter(self._tab.win_id, usertypes.KeyMode.insert,
-                              'click', only_if_normal=True)
-        else:
-            log.mouse.debug("Clicked non-editable element!")
-            if config.val.input.insert_mode.auto_leave:
-                modeman.leave(self._tab.win_id, usertypes.KeyMode.insert,
-                              'click', maybe=True)
-
     def _mouserelease_insertmode(self):
         """If we have an insertmode check scheduled, handle it."""
         if not self._check_insertmode_on_release:
diff --git i/qutebrowser/keyinput/eventfilter.py w/qutebrowser/keyinput/eventfilter.py
index 6ef0dd201ba0..75bac48e0a97 100644
--- i/qutebrowser/keyinput/eventfilter.py
+++ w/qutebrowser/keyinput/eventfilter.py
@@ -21,13 +21,14 @@
 
 import typing
 
-from PyQt5.QtCore import pyqtSlot, QObject, QEvent
-from PyQt5.QtGui import QKeyEvent, QWindow
+from PyQt5.QtCore import pyqtSlot, QObject, QEvent, Qt
+from PyQt5.QtGui import QKeyEvent, QWindow, QInputMethodQueryEvent
 from PyQt5.QtWidgets import QApplication
 
+from qutebrowser.config import config
 from qutebrowser.keyinput import modeman
 from qutebrowser.misc import quitter
-from qutebrowser.utils import objreg
+from qutebrowser.utils import objreg, usertypes, objreg, log
 
 
 class EventFilter(QObject):
@@ -110,8 +111,82 @@ class EventFilter(QObject):
             raise
 
 
+from PyQt5.QtCore import QTimer
+
+class IMEEventHandler(QObject):
+
+    def __init__(self, parent=None):
+        super().__init__(parent)
+        self._input_method = QApplication.inputMethod()
+        print("adding ime handler")
+        self._input_method.cursorRectangleChanged.connect(
+            self.cursor_rectangle_changed
+        )
+        self._last_seen_rect = None
+
+    @pyqtSlot()
+    def cursor_rectangle_changed(self):
+        # todo:
+        #   clear last_seen_rect on mode exit so that you can click on
+        #     focused input field and re-enter
+        #   last seen rect per window? tab? tab might work better with
+        #      remembering focus for tabs
+        #   anything to unregister? saw some hangs on crash but might be
+        #      because of being a temp basedir
+        # some input examples here https://www.javatpoint.com/html-form-input-types
+        #  <input type="date">: doesn't report as having an input method enabled,
+        #  although the existing heuristics pick it up
+        # if insert_mode_auto_load is false but there is a blinking cursor on
+        # load clicking the scroll bar will enter insert mode
+
+        new_rect = self._input_method.cursorRectangle()
+        if self._last_seen_rect and self._last_seen_rect.contains(new_rect):
+            print("contains")
+            return
+
+        self._last_seen_rect = new_rect
+
+        # implementation detail: qtwebengine doesn't set anchor for input
+        # fields in a web page, qt widgets do, I haven't found any cases where
+        # it doesn't work yet. Would like to compare with a "get focused thing
+        # and examine" check first and compare across versions.
+        anchor_rect = self._input_method.anchorRectangle()
+        if anchor_rect:
+            print("Not handling because anchor rect is set")
+            return
+
+        focused_window = objreg.last_focused_window()
+        focusObject = QApplication.focusObject()
+        query = None
+
+        if not new_rect and focusObject:
+            # sometimes we get a rectangle changed event and the queried
+            # rectangle is empty but we are still in an editable element. For
+            # instance when pressing enter in a text box on confluence or jira
+            # (including comment on the Qt instance) and tabbing between cells
+            # on https://html-online.com/editor/
+            # Checking ImEnabled helps in these cases.
+            query = QInputMethodQueryEvent(Qt.ImEnabled);
+            QApplication.sendEvent(focusObject, query);
+
+        if new_rect or (query and query.value(Qt.ImEnabled)):
+            log.mouse.debug("Clicked editable element!")
+            if config.val.input.insert_mode.auto_enter:
+                modeman.enter(focused_window.win_id, usertypes.KeyMode.insert,
+                              'click', only_if_normal=True)
+        else:
+            log.mouse.debug("Clicked non-editable element!")
+            if config.val.input.insert_mode.auto_leave:
+                modeman.leave(focused_window.win_id, usertypes.KeyMode.insert,
+                              'click', maybe=True)
+
+
+_ime_event_handler_instance = None
+
 def init() -> None:
     """Initialize the global EventFilter instance."""
     event_filter = EventFilter(parent=QApplication.instance())
     event_filter.install()
     quitter.instance.shutting_down.connect(event_filter.shutdown)
+    def dothing():
+        _ime_event_handler_instance = IMEEventHandler(parent=QApplication.instance())
+    QTimer.singleShot(1000, dothing)

Decoupling the entry of insert mode from clicking is quite the difference in behaviour in some cases. For instance slack focuses the message entry box whenever the webpage is focused, including when exiting command mode, and when switching between channels. Perhaps doing the ImEnabled query on click would help with the "clicking on editable things" detection and there could be a new setting to enable this mode. The auto enter/exit mode ones could use per-url enabling too.

I've had good results with the existing is editable detection by getting rid of the button press handling path and just always doing it on button release. This helps with sites that create an editable element from a button like with jira comments where by the time the JS runs for the release event the new text box is focused. Changing the existing is_editable check for querying ImEnabled may require adding an extra delay to get that benefit (the webelem click code would have to be changed to generate real mouse events to focus the element so that hinting non-editable elements would change mode correctly). Presumably we would still want to inspect elements for editableness without focusing them in some cases?

[andys8]

Hey there,

I've noticed that this issue (where there is a blinking cursor, but the browser isn't in insert mode) is happening so often to me, that my muscle memory has lost any trust in the browser and I'm clicking / firing commands multiple times for any input.

I'm wondering if people struggle with input fields (faked / shadow dom / js) not triggering insert mode, and if there are workarounds or ways to improve it.

[The-Compiler]

@andys8 Sorry for the delay, I was taking a break from things for a bit. I suppose you could try playing with the input.insert_mode.auto_* settings, and see how things feel with e.g. qutebrowser not entering insert mode automatically at all (and pressing i manually instead).

[chadkouse]

Here's an updated version of the patch @toofar posted above.

Updated Patch for 3.x

diff --git i/qutebrowser/browser/eventfilter.py w/qutebrowser/browser/eventfilter.py
index 1cff11ac4..77f018282 100644
--- i/qutebrowser/browser/eventfilter.py
+++ w/qutebrowser/browser/eventfilter.py
@@ -193,16 +193,16 @@ class TabEventFilter(QObject):
             self._check_insertmode_on_release = True
             return
 
-        if elem.is_editable():
-            log.mouse.debug("Clicked editable element!")
-            if config.val.input.insert_mode.auto_enter:
-                modeman.enter(self._tab.win_id, usertypes.KeyMode.insert,
-                              'click', only_if_normal=True)
-        else:
-            log.mouse.debug("Clicked non-editable element!")
-            if config.val.input.insert_mode.auto_leave:
-                modeman.leave(self._tab.win_id, usertypes.KeyMode.insert,
-                              'click', maybe=True)
+        # if elem.is_editable():
+        #     log.mouse.debug("Clicked editable element!")
+        #     if config.val.input.insert_mode.auto_enter:
+        #         modeman.enter(self._tab.win_id, usertypes.KeyMode.insert,
+        #                       'click', only_if_normal=True)
+        # else:
+        #     log.mouse.debug("Clicked non-editable element!")
+        #     if config.val.input.insert_mode.auto_leave:
+        #         modeman.leave(self._tab.win_id, usertypes.KeyMode.insert,
+        #                       'click', maybe=True)
 
     def _mouserelease_insertmode(self):
         """If we have an insertmode check scheduled, handle it."""
diff --git i/qutebrowser/keyinput/eventfilter.py w/qutebrowser/keyinput/eventfilter.py
index 306d4405b..9fc33e980 100644
--- i/qutebrowser/keyinput/eventfilter.py
+++ w/qutebrowser/keyinput/eventfilter.py
@@ -6,12 +6,14 @@
 
 from typing import cast, Optional
 
-from qutebrowser.qt.core import pyqtSlot, QObject, QEvent, qVersion
-from qutebrowser.qt.gui import QKeyEvent, QWindow
+from qutebrowser.qt.core import pyqtSlot, QObject, QEvent, qVersion, Qt, QTimer
+from qutebrowser.qt.gui import QKeyEvent, QWindow, QInputMethodQueryEvent
+from qutebrowser.qt.widgets import QApplication
 
+from qutebrowser.config import config
 from qutebrowser.keyinput import modeman
 from qutebrowser.misc import quitter, objects
-from qutebrowser.utils import objreg, debug, log
+from qutebrowser.utils import objreg, debug, log, usertypes
 
 
 class EventFilter(QObject):
@@ -116,6 +118,76 @@ class EventFilter(QObject):
             # activated, we'll get an infinite loop and a stack overflow.
             self._activated = False
             raise
+class IMEEventHandler(QObject):
+
+    def __init__(self, parent=None):
+        super().__init__(parent)
+        self._input_method = QApplication.inputMethod()
+        print("adding ime handler")
+        self._input_method.cursorRectangleChanged.connect(
+            self.cursor_rectangle_changed
+        )
+        self._last_seen_rect = None
+
+    @pyqtSlot()
+    def cursor_rectangle_changed(self):
+        # todo:
+        #   clear last_seen_rect on mode exit so that you can click on
+        #     focused input field and re-enter
+        #   last seen rect per window? tab? tab might work better with
+        #      remembering focus for tabs
+        #   anything to unregister? saw some hangs on crash but might be
+        #      because of being a temp basedir
+        # some input examples here https://www.javatpoint.com/html-form-input-types
+        #  <input type="date">: doesn't report as having an input method enabled,
+        #  although the existing heuristics pick it up
+        # if insert_mode_auto_load is false but there is a blinking cursor on
+        # load clicking the scroll bar will enter insert mode
+
+        new_rect = self._input_method.cursorRectangle()
+        if self._last_seen_rect and self._last_seen_rect.contains(new_rect):
+            print("contains")
+            return
+
+        self._last_seen_rect = new_rect
+
+        # implementation detail: qtwebengine doesn't set anchor for input
+        # fields in a web page, qt widgets do, I haven't found any cases where
+        # it doesn't work yet. Would like to compare with a "get focused thing
+        # and examine" check first and compare across versions.
+        anchor_rect = self._input_method.anchorRectangle()
+        if anchor_rect:
+            print("Not handling because anchor rect is set")
+            return
+
+        focused_window = objreg.last_focused_window()
+        focusObject = QApplication.focusObject()
+        query = None
+
+        if not new_rect and focusObject:
+            # sometimes we get a rectangle changed event and the queried
+            # rectangle is empty but we are still in an editable element. For
+            # instance when pressing enter in a text box on confluence or jira
+            # (including comment on the Qt instance) and tabbing between cells
+            # on https://html-online.com/editor/
+            # Checking ImEnabled helps in these cases.
+            query = QInputMethodQueryEvent(Qt.InputMethodQuery.ImEnabled);
+            QApplication.sendEvent(focusObject, query);
+
+        if new_rect or (query and query.value(Qt.InputMethodQuery.ImEnabled)):
+            log.mouse.debug("Clicked editable element!")
+            if config.val.input.insert_mode.auto_enter:
+                modeman.enter(focused_window.win_id, usertypes.KeyMode.insert,
+                              'click', only_if_normal=True)
+        else:
+            log.mouse.debug("Clicked non-editable element!")
+            if config.val.input.insert_mode.auto_leave:
+                modeman.leave(focused_window.win_id, usertypes.KeyMode.insert,
+                              'click', maybe=True)
+
+
+_ime_event_handler_instance = None
+
 
 
 def init() -> None:
@@ -123,3 +195,6 @@ def init() -> None:
     event_filter = EventFilter(parent=objects.qapp)
     event_filter.install()
     quitter.instance.shutting_down.connect(event_filter.shutdown)
+    def dothing():
+        _ime_event_handler_instance = IMEEventHandler(parent=QApplication.instance())
+    QTimer.singleShot(1000, dothing)

Seems to work for me with the 3.x branch