Add feature allowing key and mouse events to be sent via Natlink

Re: #372.

Natlink's `playEvents' function can be used to simulate keystrokes
and mouse events with Dragon NaturallySpeaking.  Dragonfly's Key,
Text and Mouse action objects are now able to send events this way,
allowing Dragonfly commands to interact with administrative applic-
ations on Windows.

This feature is disabled by default because the `playEvents' inter-
face is not a perfect substitute for the Windows `SendInput' funct-
ion.  Dragonfly falls back on the default implementation if Natlink
cannot be used.

Documentation on this feature has been added in the relevant files.

documentation/faq.txt

@@ -486,6 +486,20 @@ It is not normally possible to interact with applications running in
 elevated mode.  The recommended way to do so is to use Dragon's built-in
 commands, which can interact with such applications.
 
+The :code:`Key`, :code:`Text` and :code:`Mouse` action classes may be
+configured to send input events via Natlink to Dragon.  To enable this
+feature, run the following code, or add it into one of your command modules:
+
+  .. code-block:: python
+
+     # Enable sending keyboard events via Natlink to Dragon.
+     from dragonfly.actions.keyboard import Keyboard
+     Keyboard.try_natlink = True
+
+     # Enable sending mouse events via Natlink to Dragon.
+     from dragonfly.actions.mouse import ButtonEvent
+     ButtonEvent.try_natlink = True
+
 If you are not using Dragon, or if you want to use your own commands instead
 of the built-ins, try running the appropriate command-module loader script
 in `dragonfly/examples`_ as the administrator.

dragonfly/actions/action_key.py

@@ -199,8 +199,8 @@
 applications do not accept the keystrokes at all.  On X11, this feature is
 always on, since there are no caveats.
 
-To enable this feature on Windows, run the following code, or add it into
-one of your command modules:
+To enable this feature globally on Windows, run the following code, or add
+it into one of your command modules:
 
 .. code:: python
 
@@ -231,6 +231,41 @@
 used on Windows.  They have no effect on other platforms.
 
 
+.. _RefNatlinkKeyboardInput:
+
+Using Natlink for keyboard input on Windows (Key)
+............................................................................
+
+On Windows, the :class:`Key` action object may be configured to send events
+via Natlink.  This allows one to make use of Dragon NaturallySpeaking's
+ability to control applications running in elevated mode, i.e.,
+administrative applications.
+
+This feature is disabled by default, primarily because modifier keys are not
+always held down when simulated this way.  To (globally) enable the feature
+anyway, run the following code, or add it into one of your command
+modules: ::
+
+   from dragonfly.actions.keyboard import Keyboard
+   Keyboard.try_natlink = True
+
+In order for this to work, Natlink must be available and Dragon
+NaturallySpeaking must be running.  Dragonfly will fallback on the default
+keyboard implementation for Windows if it is unable to send input via
+Natlink.
+
+The following keyboard events cannot be sent via Natlink and will always be
+sent normally:
+
+ * Unicode character keystrokes (if enabled)
+
+As noted above, modifier keys are not always held down when simulated via
+Natlink.
+
+This feature may be enabled for mouse input events too.  See
+:ref:`RefNatlinkMouseInput`.
+
+
 X11 keys
 ............................................................................
 

dragonfly/actions/action_mouse.py

@@ -171,6 +171,34 @@
    #Mouse("left, left, left").execute()
 
 
+.. _RefNatlinkMouseInput:
+
+Using Natlink for mouse input on Windows
+............................................................................
+
+On Windows, the :class:`Mouse` action object may be configured to send
+events via Natlink.  This allows one to make use of Dragon
+NaturallySpeaking's ability to control applications running in elevated
+mode, i.e., administrative applications.
+
+This feature is disabled by default, primary because some events cannot be
+simulated this way (see below).  To (globally) enable this feature anyway,
+run the following code, or add it into one of your command modules: ::
+
+   from dragonfly.actions.mouse import ButtonEvent
+   ButtonEvent.try_natlink = True
+
+In order for this to work, Natlink must be available and Dragon
+NaturallySpeaking must be running.  Dragonfly will fallback on the default
+mouse implementation for Windows if it is unable to send input via Natlink.
+
+The following mouse events cannot be sent via Natlink and will always be
+sent normally:
+
+ * Button *four* and *five* events
+ * Scroll wheel events (*wheelup*, *wheeldown*, etc.)
+
+
 Mouse class reference
 ............................................................................
 

dragonfly/actions/action_text.py

@@ -49,6 +49,17 @@
 subject.  Most of it applies to :class:`Text` action objects too.
 
 
+Using Natlink for keyboard input on Windows (Text)
+............................................................................
+
+On Windows, the :class:`Key` action object may be configured to send events
+via Natlink.  This allows one to make use of Dragon NaturallySpeaking's
+ability to control applications running in elevated mode, i.e.,
+administrative applications.
+
+Please see :ref:`RefNatlinkKeyboardInput` for documentation on this subject.
+
+
 Text class reference
 ............................................................................
 

dragonfly/actions/keyboard/_win32.py

@@ -39,6 +39,7 @@
 from dragonfly.actions.sendinput       import (KeyboardInput,
                                                make_input_array,
                                                send_input_array)
+from dragonfly.actions.natlinkinput    import send_input_array_natlink
 
 
 class Win32KeySymbols(BaseKeySymbols):
@@ -246,6 +247,9 @@ class Win32Keyboard(BaseKeyboard):
         "tvnviewer.exe", "vncviewer.exe", "mstsc.exe", "virtualbox.exe"
     ]
 
+    #: Use Natlink to send keyboard events, if possible.
+    try_natlink = False
+
     @classmethod
     def get_current_layout(cls):
         # Get the current window's keyboard layout.
@@ -287,13 +291,14 @@ def send_keyboard_events(cls, events):
                     at a window or control that accepts Unicode text.
         """
         layout = cls.get_current_layout()
+        try_natlink = cls.try_natlink
 
         # Process and send keyboard events.
         items = []
         for event in events:
             if len(event) == 3:
-                # Pass scancode=0 to press keys via virtual-key codes
-                #  instead of scancodes.
+                # scancode=0 may be given below to send keystrokes as
+                #  virtual-keys instead of scancodes.
                 keycode, down, timeout = event
                 input_structure = KeyboardInput(keycode, down,
                                                 # scancode=0,
@@ -307,11 +312,15 @@ def send_keyboard_events(cls, events):
             if timeout:
                 array = make_input_array(items)
                 items = []
-                send_input_array(array)
+                # Try using Natlink to send input, if appropriate.
+                #  Otherwise, use SendInput.
+                if not (try_natlink and send_input_array_natlink(array)):
+                    send_input_array(array)
                 time.sleep(timeout)
         if items:
             array = make_input_array(items)
-            send_input_array(array)
+            if not (try_natlink and send_input_array_natlink(array)):
+                send_input_array(array)
             if timeout: time.sleep(timeout)
 
     @classmethod

dragonfly/actions/mouse/_win32.py

@@ -21,13 +21,15 @@
 # pylint: disable=E0401
 # This file imports Win32-only symbols.
 
-from ctypes import windll, pointer, c_long, c_ulong, Structure
+from ctypes                          import (windll, pointer, c_long,
+                                             c_ulong, Structure)
 
 import win32con
 
-from dragonfly.actions.sendinput import (MouseInput, make_input_array,
-                                         send_input_array)
-from dragonfly.actions.mouse._base import BaseButtonEvent, MoveEvent
+from dragonfly.actions.sendinput     import (MouseInput, make_input_array,
+                                             send_input_array)
+from dragonfly.actions.natlinkinput  import send_input_array_natlink
+from dragonfly.actions.mouse._base   import BaseButtonEvent, MoveEvent
 
 
 #---------------------------------------------------------------------------
@@ -104,6 +106,9 @@ def set_position(cls, x, y):
 
 class ButtonEvent(BaseButtonEvent):
 
+    #: Use Natlink to send mouse events, if possible.
+    try_natlink = False
+
     def execute(self, window):
         # Ensure that the primary mouse button is the *left* button before
         #  sending events.
@@ -114,7 +119,11 @@ def execute(self, window):
             inputs = [MouseInput(0, 0, flag[1], flag[0], 0, zero)
                       for flag in self._flags]
             array = make_input_array(inputs)
-            send_input_array(array)
+
+            # Try using Natlink to send input, if appropriate.  Otherwise,
+            #  use SendInput.
+            if not (self.try_natlink and send_input_array_natlink(array)):
+                send_input_array(array)
         finally:
             # Swap the primary mouse button back if it was previously set
             #  to *right*.

dragonfly/actions/natlinkinput.py

@@ -0,0 +1,104 @@
+#
+# This file is part of Dragonfly.
+# (c) Copyright 2022 by Dane Finlay
+# Licensed under the LGPL.
+#
+#   Dragonfly is free software: you can redistribute it and/or modify it
+#   under the terms of the GNU Lesser General Public License as published
+#   by the Free Software Foundation, either version 3 of the License, or
+#   (at your option) any later version.
+#
+#   Dragonfly is distributed in the hope that it will be useful, but
+#   WITHOUT ANY WARRANTY; without even the implied warranty of
+#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#   Lesser General Public License for more details.
+#
+#   You should have received a copy of the GNU Lesser General Public
+#   License along with Dragonfly.  If not, see
+#   <http://www.gnu.org/licenses/>.
+#
+
+"""
+Natlink input wrapper functions.
+
+This file implements an interface to natlink's playEvents function
+for simulating keyboard and mouse events via NatSpeak.
+"""
+
+import logging
+from ctypes import sizeof, pointer
+
+import win32con
+
+try:
+    import natlink
+except:
+    natlink = None
+
+
+_log = logging.getLogger("keyboard")
+
+def send_input_array_natlink(input_array):
+    # Check if Natlink can be used.
+    if natlink is None or not natlink.isNatSpeakRunning(): return False
+
+    # Take the same argument as the equivalent SendInput function.
+    length = len(input_array)
+    if length == 0: return True
+    size = sizeof(input_array[0])
+    ptr = pointer(input_array)
+
+    # Translate each input struct into a tuple that can be processed by
+    #  NatSpeak.  Return False if any incompatible event is found.
+    input_list = []
+    for input in input_array:
+        if input.type == win32con.INPUT_KEYBOARD:
+            keybdinput = input.ii.ki
+            keycode = keybdinput.wVk
+            dwFlags = keybdinput.dwFlags
+            if keycode == 0: return False  # Implies KEYEVENTF_UNICODE.
+            msg = 0x101 if dwFlags & win32con.KEYEVENTF_KEYUP else 0x100
+            event = (msg, keycode, 1)
+        elif input.type == win32con.INPUT_MOUSE:
+            mouseinput = input.ii.mi
+            dwFlags = mouseinput.dwFlags
+            if   dwFlags & win32con.MOUSEEVENTF_LEFTDOWN:   msg = 0x201
+            elif dwFlags & win32con.MOUSEEVENTF_LEFTUP:     msg = 0x202
+            elif dwFlags & win32con.MOUSEEVENTF_RIGHTDOWN:  msg = 0x204
+            elif dwFlags & win32con.MOUSEEVENTF_RIGHTUP:    msg = 0x205
+            elif dwFlags & win32con.MOUSEEVENTF_MIDDLEDOWN: msg = 0x207
+            elif dwFlags & win32con.MOUSEEVENTF_MIDDLEUP:   msg = 0x208
+            else: return False
+            # Note: In Dragonfly the mouse is not moved via SendInput, so
+            #  we just use the current cursor position.
+            x, y = natlink.getCursorPos()
+            event = (msg, x, y)
+        else:
+            return False
+        input_list.append(event)
+
+    # Attempt to send input via Natlink.
+    connected = None
+    try:
+        natlink.playEvents(input_list)
+    except natlink.NatError as err:
+        # If this failed because we're not connected yet, attempt to
+        #  connect to natlink and retry.
+        if (str(err) == "Calling playEvents is not allowed before"
+                        " calling natConnect"):
+            natlink.natConnect()
+            connected = True
+        else:
+            _log.exception("Exception sending input via Natlink: %s", err)
+            return False
+
+    # Retry, and disconnect afterward, if necessary.
+    if connected:
+        try:
+            natlink.playEvents(input_list)
+        except natlink.NatError as err:
+            _log.exception("Exception sending input via Natlink: %s", err)
+            return False
+        finally:
+            natlink.natDisconnect()
+    return True