Add HUD #773

dmakarov · 2020-03-15T19:25:05Z

Title

HUD to display recognized utterances and corresponding actions

Description

This adds the HUD GUI. This is work in progress. Initial PR to reveal any possible issues.

Related Issue

Motivation and Context

It is hoped to improve the user experience.

How Has This Been Tested

Ran on macOS and Windows

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue or bug)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Renamed existing command phrases (we discourage this without a strong rationale).

Checklist

I have read the CONTRIBUTING document.
My code follows the code style of this project.
I have checked that my code does not duplicate functionality elsewhere in Caster.
I have checked for and utilized existing command phrases from within Caster (delete if not applicable).
My code implements all the features I wish to merge in this pull request.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have added tests to cover my changes.
All new and existing tests pass.

Maintainer/Reviewer Checklist

Basic functionality has been tested and works as claimed.
New documentation is clear and complete.
Code is clear and readable.

dmakarov · 2020-03-15T20:40:48Z

LexiconCode · 2020-03-15T22:09:59Z

It's been great coordinating chat make this possible! Thank you for your work on this as it's invaluable. Looking forward to testing!

_caster.py

kendonB · 2020-04-30T10:40:06Z

This one isn't working for me with kaldi:

ERROR:action:Exception from function show_hud:
Traceback (most recent call last):
  File "C:\Python27_64\lib\site-packages\dragonfly\actions\action_function.py", line 145, in _execute
    self._function(**arguments)
  File "castervoice\lib\utilities.py", line 361, in show_hud
    hud.show_hud()
  File "C:\Python27_64\lib\xmlrpclib.py", line 1243, in __call__
    return self.__send(self.__name, args)
  File "C:\Python27_64\lib\xmlrpclib.py", line 1602, in __request
    verbose=self.__verbose
  File "C:\Python27_64\lib\xmlrpclib.py", line 1283, in request
    return self.single_request(host, handler, request_body, verbose)
  File "C:\Python27_64\lib\xmlrpclib.py", line 1311, in single_request
    self.send_content(h, request_body)
  File "C:\Python27_64\lib\xmlrpclib.py", line 1459, in send_content
    connection.endheaders(request_body)
  File "C:\Python27_64\lib\httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "C:\Python27_64\lib\httplib.py", line 882, in _send_output
    self.send(msg)
  File "C:\Python27_64\lib\httplib.py", line 844, in send
    self.connect()
  File "C:\Python27_64\lib\httplib.py", line 821, in connect
    self.timeout, self.source_address)
  File "C:\Python27_64\lib\socket.py", line 575, in create_connection
    raise err
error: [Errno 10061] No connection could be made because the target machine actively refused it
ERROR:action.exec:Execution failed: 'show_hud'(): [Errno 10061] No connection could be made because the target machine actively refused it
```

dmakarov · 2020-04-30T10:45:07Z

@kendonB this requires python3.

kendonB · 2020-04-30T20:05:04Z

@dmakarov is the idea here that we'd hold off on merging this until natlink is up and running with python 3? Or this would just be available for people with Kaldi setups?

dmakarov · 2020-05-01T05:03:16Z

@kendonB as far as I know natlink works on python3. This PR is still WIP, but I should finish it soon. I didn’t intend HUD to be available only with kaldi engine, although I test it only with kaldi and test engines, and don’t know whether it would be useful with other setups. There’s nothing engine specific in HUD’s code though. Overall the idea was to consolidate all separate Caster’s GUI under one app and single RPC server.

caspark · 2020-05-01T05:27:15Z

castervoice/asynch/hud.py

+    def closeEvent(self, event):
+        event.accept()
+
+    def setup_xmlrpc_server(self):


Apologies for the drive-by comment, but might this need a security token type of thing here, like what dragonfly & aenea use? https://github.com/dictation-toolbox/dragonfly/pull/61/files#diff-6219d462fba21473f0a2404097943f75R29

The danger case would be something like: user is tricked into browsing to evil website, evil website asks web browser to make xmlhttprequests to http://localhost:1338, evil website can now ask this server to execute any registered function. Obviously from a practical perspective this is only going to be a concern if a caster user is actively being targeted (and the blast radius in this server seems small right now - basically it'd be annoying to have the rules UI show up or have caster quit unexpectedly), but mentioning it here anyway.

Thanks. I will consider adding the security token. Probably, not in this PR, however.

Is adding the security token difficult? I wouldn't want any extra vulnerabilities introduced to master.

It's actually a vulnerability that exists already with the current implementation.

The current master you mean? Can we fix that?

@kendonB That's correct and in the current master. Basically someone would have to set up a malicious website specifically for caster and a caster user would have to go to that website. In that case they then could control RPC commands that are defined. Mostly emulating mouse now this is none that I know of control keyboard input.

To protect against this a secret would be randomly generated and be used for authentication for the client/server to protect against such targeting.

dmakarov mentioned he would implement it in another pull request.

Yes, sorry, I should have mentioned that this is an existing problem so it's not meaningfully worse than before (just a couple new commands available to an attacker - but none of them seem to allow arbitrary code execution).

@caspark would you be able to open an issue on this (for the current vulnerabilities)?

Thanks for the reminder @caspark
Documented in #809

Thanks. I will consider adding the security token. Probably, not in this PR, however.

Is this something you are willing to take on?

LexiconCode · 2021-08-05T13:50:00Z

This will work with everything but Python 32-bit 2.7.x. This is a limitation of QT 5 via pyside2. Really only affects DNS with Natlink on 32-bit 2.7.x.

- Redirect dragonfly observers and logging to printer.out The RPC timeout with the hud can interfere with the dragonfly speech recognition loop. This appears as a stutter recognition latency. Exceptions are tracked so this stutter only happens to end user once. An exception occurs if the hud is not available/python 2/running text engine. If an exception in handle_message, print is managed by SimplePrintMessageHandler

- Removed LoggingHandler - Re-factored RecognitionObserver to configure_engine.py

bluecamel · 2021-08-22T07:15:38Z

@dmakarov This is wonderful! FWIW, I made a couple of tweaks to help with when it was crashing caster.

@dmakarov still getting this error with python 3

I had this issue at first too. There was another error earlier from when the hud tried to start, but that was hitting a Shiboken error. I upgraded PyQt5 and PySide2 and it's working great now.

LexiconCode · 2021-10-18T05:13:19Z

@dmakarov This is wonderful! FWIW, I made a couple of tweaks to help with when it was crashing caster.

@dmakarov still getting this error with python 3

I had this issue at first too. There was another error earlier from when the hud tried to start, but that was hitting a Shiboken error. I upgraded PyQt5 and PySide2 and it's working great now.

I've included some of your enhancements catching connection error. This PR will be merged Directly but will be included in a new all request combines most of dmakarov work and drop support for Python 2.

- grammar should be loaded before registering observers - cleanup imports and pylint

commit a5303dd Author: Brian Vancil <1388508+bvancil@users.noreply.github.com> Date: Wed Dec 21 16:40:34 2022 -0600 Added vscode terminal pane toggle commit 1de7195 Author: LexiconCode <CasterVoice@protonmail.com> Date: Sun Oct 30 08:25:03 2022 -0500 Hud use TK and multiprocessing and Queue commit 992fd88 Author: LexiconCode <CasterVoice@protonmail.com> Date: Tue Jul 5 13:50:08 2022 -0500 add missing extras and defaults `Your_First_Rule.md` and example rules commit e5f047f Author: LexiconCode <CasterVoice@protonmail.com> Date: Mon Feb 14 03:54:59 2022 -0600 Update collections import commit bca040d Author: LexiconCode <CasterVoice@protonmail.com> Date: Sat Feb 12 02:21:41 2022 -0600 Clean up hud state commit de91635 Author: LexiconCode <CasterVoice@protonmail.com> Date: Sat Feb 12 01:51:22 2022 -0600 Only find DPI/DNS if isNatSpeakRunningnatlink. commit 1ec2775 Author: LexiconCode <CasterVoice@protonmail.com> Date: Thu Feb 10 20:20:30 2022 -0600 remove settings window manual path handling commit d0c3cce Author: LexiconCode <CasterVoice@protonmail.com> Date: Thu Feb 10 20:10:51 2022 -0600 Rename WXTYPE_SETTINGS to QTTYPE_SETTINGS commit bf2be01 Author: LexiconCode <CasterVoice@protonmail.com> Date: Thu Feb 10 19:41:21 2022 -0600 removed availability_message function PIL is cross platform https://pillow.readthedocs.io/en/stable/installation.html commit 556e6dc Author: LexiconCode <CasterVoice@protonmail.com> Date: Thu Feb 10 19:27:35 2022 -0600 remove wxpython commit e821be9 Author: LexiconCode <CasterVoice@protonmail.com> Date: Thu Feb 10 19:05:28 2022 -0600 Clear hud when restarting Caster commit dd61366 Author: LexiconCode <CasterVoice@protonmail.com> Date: Tue Feb 8 15:23:04 2022 -0600 "clear caster hud" command clears the "Caster: Status Window" if hud is not available commit e0a5b1d Author: LexiconCode <CasterVoice@protonmail.com> Date: Tue Feb 8 15:21:32 2022 -0600 Rename and add hud commands to docs - rename hud commands to be less ambiguous commit d31fe8c Author: LexiconCode <CasterVoice@protonmail.com> Date: Mon Jan 31 23:02:33 2022 -0600 add PySide2 to windows os commit f868f01 Author: Brian Vancil <1388508+bvancil@users.noreply.github.com> Date: Wed Dec 21 16:38:18 2022 -0600 dragonfly2 version bump to 0.34.0 - Fix various bugs with keyboard/text input actions on Windows - Fix problems caused by relative import statements in the library by using the absolute form instead. - edited by bvancil to resolve merge conflicts commit 9e07efd Author: LexiconCode <CasterVoice@protonmail.com> Date: Mon Jan 31 22:47:44 2022 -0600 rename work flow commit ec459fc Author: Kendon Bell <kendonB@users.noreply.github.com> Date: Tue Jan 25 13:46:45 2022 +1300 PySide2 version I got this error with an old PySide2 Manjaro-WebDad/jak#67 commit bdf04ca Author: Kendon Bell <kendonB@users.noreply.github.com> Date: Fri Jan 21 11:20:33 2022 +1300 update docs for keyboard Adding the "release modifier keys" command that replaced "release <keyname>" commit 31d1e89 Author: LexiconCode <CasterVoice@protonmail.com> Date: Wed Jan 19 12:29:47 2022 -0600 Remove API command grammars examples commit 9c67920 Author: Dmitri Makarov <dmakarov@users.noreply.github.com> Date: Tue Oct 19 06:15:45 2021 -0700 Convert HMC to QT on python 3 (dictation-toolbox#786) commit 2e573c7 Author: Dmitri Makarov <dmakarov@users.noreply.github.com> Date: Mon Oct 18 17:19:10 2021 -0700 Add HUD (dictation-toolbox#773) commit 5dd65fb Author: Brian Vancil <1388508+bvancil@users.noreply.github.com> Date: Wed Dec 21 16:31:28 2022 -0600 utilized py launcher with explicit 3.8-32 python version -edited by bvancil to resolve merge conflicts commit ded7e05 Author: LexiconCode <CasterVoice@protonmail.com> Date: Sun Oct 3 01:09:27 2021 -0500 Miscellaneous cleanup commit 1530d58 Author: Brian Vancil <1388508+bvancil@users.noreply.github.com> Date: Wed Dec 21 16:26:25 2022 -0600 remove py2 action -edited by bvancil to remove merge conflicts commit c4e0d28 Author: Brian Vancil <1388508+bvancil@users.noreply.github.com> Date: Wed Dec 21 16:23:08 2022 -0600 remove imports for py2 - remove default encoding utf-8 for py2 - remove six code for py2 - fix natlink imports for py3 - edited by bvancil to resolve merge dependencies commit c041a90 Author: LexiconCode <CasterVoice@protonmail.com> Date: Fri Oct 1 19:46:17 2021 -0500 Drop py2 pathlib port commit 2ed2487 Author: LexiconCode <CasterVoice@protonmail.com> Date: Fri Oct 1 19:02:43 2021 -0500 Doc Natlink Py3

LexiconCode added GUI issues related to graphics user interface New Feature A new feature that is not currently implemented. WIP An work in progress Python 3 Issues related to Python 3 labels Mar 15, 2020

LexiconCode assigned dmakarov Mar 15, 2020

dmakarov force-pushed the qt branch 8 times, most recently from 934db13 to 4142b85 Compare March 21, 2020 18:16

dmakarov force-pushed the qt branch from 4142b85 to bdda383 Compare April 1, 2020 12:48

LexiconCode reviewed Apr 4, 2020

View reviewed changes

_caster.py Outdated Show resolved Hide resolved

dmakarov force-pushed the qt branch 3 times, most recently from 68d2db9 to 6642785 Compare April 10, 2020 18:25

dmakarov force-pushed the qt branch from 6642785 to f18c2e9 Compare April 19, 2020 14:29

caspark reviewed May 1, 2020

View reviewed changes

dmakarov force-pushed the qt branch 2 times, most recently from 039874c to 8145a6b Compare May 3, 2020 06:12

LexiconCode mentioned this pull request Jun 7, 2020

generate grammer documentation dictation-toolbox/dragonfly#259

Closed

Fix hmc on python 3

de64f74

cleanup

7d0e18e

This comment has been minimized.

Sign in to view

LexiconCode added 2 commits August 2, 2021 13:15

refactor to hud_support

eb74858

update hud_support imports

25f7e36

LexiconCode mentioned this pull request Aug 3, 2021

make printer.out more flexible #909

Merged

17 tasks

LexiconCode added 3 commits August 4, 2021 12:22

Merge branch 'master' into qt

3beaec4

Integrate HudPrintMessageHandler

38e703b

added clear_hud command and increased hud auto clear 50 commands

00ca3b6

LexiconCode force-pushed the qt branch from 8ebed95 to 00ca3b6 Compare August 5, 2021 13:44

LexiconCode removed the Python 3 Issues related to Python 3 label Aug 5, 2021

LexiconCode added 3 commits August 5, 2021 16:43

Suppress recognition observers when in sleep mode

b3ceaba

- Added additional text formatting for HUD

0386ef9

- Removed LoggingHandler - Re-factored RecognitionObserver to configure_engine.py

LexiconCode added 2 commits October 18, 2021 00:10

Handle exception if HUD is not available post HUD initialization

61e5da3

Increase port range of RPC

3309858

LexiconCode changed the base branch from master to DropPy2 October 18, 2021 21:19

LexiconCode added 4 commits October 18, 2021 18:37

Merge branch 'DropPy2' into qt

386f5a2

Corrected initializing get_engine

2907b2a

Re-factor registering observer to EngineConfigLate

fb87aa8

- grammar should be loaded before registering observers - cleanup imports and pylint

remove six py2 compatibility from h_launch

c47118c

LexiconCode merged commit eb803ed into dictation-toolbox:DropPy2 Oct 19, 2021

LexiconCode pushed a commit to LexiconCode/Caster that referenced this pull request Jan 14, 2022

Add HUD (dictation-toolbox#773)

734d1db

LexiconCode pushed a commit to LexiconCode/Caster that referenced this pull request Feb 22, 2022

Add HUD (dictation-toolbox#773)

c4c8495

LexiconCode pushed a commit to LexiconCode/Caster that referenced this pull request Mar 10, 2022

Add HUD (dictation-toolbox#773)

087ae2a

bvancil pushed a commit to bvancil/Caster that referenced this pull request Dec 21, 2022

Add HUD (dictation-toolbox#773)

2e573c7

kendonB added a commit to kendonB/Caster that referenced this pull request Feb 23, 2023

Add HUD (dictation-toolbox#773)

f711a15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add HUD #773

Add HUD #773

dmakarov commented Mar 15, 2020 •

edited by LexiconCode

dmakarov commented Mar 15, 2020

LexiconCode commented Mar 15, 2020

kendonB commented Apr 30, 2020

dmakarov commented Apr 30, 2020

kendonB commented Apr 30, 2020

dmakarov commented May 1, 2020

caspark May 1, 2020

dmakarov May 1, 2020

kendonB May 1, 2020 •

edited

LexiconCode May 1, 2020

kendonB May 1, 2020

LexiconCode May 1, 2020 •

edited

caspark May 2, 2020

kendonB May 2, 2020

LexiconCode May 2, 2020

LexiconCode Aug 8, 2021

This comment has been minimized.

LexiconCode commented Aug 5, 2021

bluecamel commented Aug 22, 2021

LexiconCode commented Oct 18, 2021

Add HUD #773

Add HUD #773

Conversation

dmakarov commented Mar 15, 2020 • edited by LexiconCode

Title

Description

Related Issue

Motivation and Context

How Has This Been Tested

Types of changes

Checklist

Maintainer/Reviewer Checklist

dmakarov commented Mar 15, 2020

LexiconCode commented Mar 15, 2020

kendonB commented Apr 30, 2020

dmakarov commented Apr 30, 2020

kendonB commented Apr 30, 2020

dmakarov commented May 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kendonB May 1, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LexiconCode May 1, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

LexiconCode commented Aug 5, 2021

bluecamel commented Aug 22, 2021

LexiconCode commented Oct 18, 2021

dmakarov commented Mar 15, 2020 •

edited by LexiconCode

kendonB May 1, 2020 •

edited

LexiconCode May 1, 2020 •

edited