Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If UIA is enabled in Word, the "Scrolls the braille display back" and "Scrolls the braille display forward" will not work in focus mode #12855

Closed
cary-rowen opened this issue Sep 16, 2021 · 12 comments · Fixed by #12868
Milestone

Comments

@cary-rowen
Copy link
Contributor

Test.docx

Steps to reproduce:

  1. Download the provided test.docx.
  2. Open the document.
  3. Open the UIA in Word in the advanced settings panel of NVDA.
  4. Connect your braille display to make it work.
  5. In MS Word, press NVDA + space to make sure to switch to focus mode.
  6. Read the content of the document and try to execute "Scrolls the braille display back / forward".

Actual behavior:

The content on the braille display changes quickly and then reverts to the original content. Did not finish scrolling.

Expected behavior:

Display scrolled content correctly.

System configuration

NVDA installed/portable/running from source:

installed and portable

NVDA version:

NVDA2021.1

Windows version:

Windows 10 21H1 (x64) build 19043.1237

Name and version of other software in use when reproducing the issue:

Microsoft 365 MSO (16.0.14326.20324) 64 bit

Other information about your system:

None

Other questions

Does the issue still occur after restarting your computer?

Yes

Have you tried any other versions of NVDA? If so, please report their behaviors.

NVDA2021.1, 2020.4 and Alpha-23765 Can be reproduced

If add-ons are disabled, is your problem still occurring?

Yes

Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?

Yes

@feerrenrut
Copy link
Contributor

I can't reproduce this with the braille viewer. Which braille device are you using?
Can anyone else confirm with their braille device?

@cary-rowen
Copy link
Contributor Author

Hello, I used orbit Reader 20 Plus, Seaka Mini24 and metec40 to reproduce this.
Of course, Metec40 is a third-party additional driver, which may not be representative, but the other two devices can also reproduce this problem.

@feerrenrut
Copy link
Contributor

@LeonarddeR @michaelDCurran Can you reproduce this issue?

@LeonarddeR
Copy link
Collaborator

Nope, I can not, though I'm testing with alpha currently and stuff might have changed.
@cary-rowen Could you please test with NVDA 2021.2 and 2021.3 Alpha?

@cary-rowen
Copy link
Contributor Author

Hello, I upgraded NVDA to alpha-23799, and then I installed the English version of word, and the issue can still be reproduced.
Of course NVDA2021.2 is the same.

The following is a log snippet (where fk2 is the key used for scrolling):

IO - inputCore.InputManager.executeGesture (23:38:11.228) - winInputHook (12292):
Input: kb(laptop):upArrow
DEBUG - editableText.EditableText._hasCaretMoved (23:38:11.277) - MainThread (17984):
Caret move detected using event. Elapsed: 10 ms
IO - braille.BrailleBuffer.update (23:38:11.300) - MainThread (17984):
Braille regions text: ['文档 ', ' ']
IO - braille.BrailleHandler.update (23:38:11.300) - MainThread (17984):
Braille window dots: -
IO - braille.BrailleHandler.update (23:38:11.301) - MainThread (17984):
Braille window dots: -
IO - speech.speech.speak (23:38:11.318) - MainThread (17984):
Speaking ['空白']
IO - braille.BrailleBuffer.update (23:38:11.347) - MainThread (17984):
Braille regions text: ['文档 ', ' ']
IO - braille.BrailleHandler.update (23:38:11.347) - MainThread (17984):
Braille window dots: -
IO - braille.BrailleHandler.update (23:38:11.348) - MainThread (17984):
Braille window dots: -
IO - inputCore.InputManager.executeGesture (23:38:11.674) - winInputHook (12292):
Input: kb(laptop):downArrow
DEBUG - editableText.EditableText._hasCaretMoved (23:38:11.732) - MainThread (17984):
Caret move detected using bookmarks. Elapsed: 0 ms
IO - braille.BrailleBuffer.update (23:38:11.757) - MainThread (17984):
Braille regions text: ['Several Japanese politicians have announced their decisions to seek the leadership of the ']
IO - braille.BrailleHandler.update (23:38:11.757) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleHandler.update (23:38:11.759) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - speech.speech.speak (23:38:11.780) - MainThread (17984):
Speaking ['Several Japanese politicians have announced their decisions to seek the leadership of the ']
IO - braille.BrailleBuffer.update (23:38:11.817) - MainThread (17984):
Braille regions text: ['Several Japanese politicians have announced their decisions to seek the leadership of the ']
IO - braille.BrailleHandler.update (23:38:11.818) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleHandler.update (23:38:11.820) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleBuffer.update (23:38:13.012) - MainThread (17984):
Braille regions text: ['Several Japanese politicians have announced their decisions to seek the leadership of the ']
IO - braille.BrailleHandler.update (23:38:13.012) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleHandler.update (23:38:13.013) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleBuffer.update (23:38:14.060) - MainThread (17984):
Braille regions text: ['Several Japanese politicians have announced their decisions to seek the leadership of the ']
IO - braille.BrailleHandler.update (23:38:14.060) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleHandler.update (23:38:14.061) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleBuffer.update (23:38:15.191) - MainThread (17984):
Braille regions text: ['Several Japanese politicians have announced their decisions to seek the leadership of the ']
IO - braille.BrailleHandler.update (23:38:15.191) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleHandler.update (23:38:15.192) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleBuffer.update (23:38:16.314) - MainThread (17984):
Braille regions text: ['Several Japanese politicians have announced their decisions to seek the leadership of the ']
IO - braille.BrailleHandler.update (23:38:16.314) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleHandler.update (23:38:16.315) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
DEBUG - external:brailleDisplayDrivers.MetecBD._do_key (23:38:16.920) - MainThread (17984):
Metec key 0x10ff
IO - inputCore.InputManager.executeGesture (23:38:16.921) - MainThread (17984):
Input: br(MetecBD):fk2
IO - braille.BrailleHandler.update (23:38:16.935) - MainThread (17984):
Braille window dots: 1345 14 15 145 - 2345 125 15 24 1235 - 145 15 14 24 234 24 135 1345 234 - 2345 135 - 234 15 15 13 - 2345 125 15 - 123 15 1 145 15 1235 234
IO - braille.BrailleBuffer.update (23:38:17.423) - MainThread (17984):
Braille regions text: ['Several Japanese politicians have announced their decisions to seek the leadership of the ']
IO - braille.BrailleHandler.update (23:38:17.423) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleHandler.update (23:38:17.424) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
DEBUG - external:brailleDisplayDrivers.MetecBD._do_key (23:38:18.117) - MainThread (17984):
Metec key 0x10ff
IO - inputCore.InputManager.executeGesture (23:38:18.117) - MainThread (17984):
Input: br(MetecBD):fk2
IO - braille.BrailleHandler.update (23:38:18.148) - MainThread (17984):
Braille window dots: 1345 14 15 145 - 2345 125 15 24 1235 - 145 15 14 24 234 24 135 1345 234 - 2345 135 - 234 15 15 13 - 2345 125 15 - 123 15 1 145 15 1235 234
IO - braille.BrailleBuffer.update (23:38:18.567) - MainThread (17984):
Braille regions text: ['Several Japanese politicians have announced their decisions to seek the leadership of the ']
IO - braille.BrailleHandler.update (23:38:18.567) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleHandler.update (23:38:18.568) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleBuffer.update (23:38:19.677) - MainThread (17984):
Braille regions text: ['Several Japanese politicians have announced their decisions to seek the leadership of the ']
IO - braille.BrailleHandler.update (23:38:19.678) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleHandler.update (23:38:19.679) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
DEBUG - external:brailleDisplayDrivers.MetecBD._do_key (23:38:19.929) - MainThread (17984):
Metec key 0x10ff
IO - inputCore.InputManager.executeGesture (23:38:19.929) - MainThread (17984):
Input: br(MetecBD):fk2
IO - braille.BrailleHandler.update (23:38:19.960) - MainThread (17984):
Braille window dots: 1345 14 15 145 - 2345 125 15 24 1235 - 145 15 14 24 234 24 135 1345 234 - 2345 135 - 234 15 15 13 - 2345 125 15 - 123 15 1 145 15 1235 234
IO - braille.BrailleBuffer.update (23:38:20.804) - MainThread (17984):
Braille regions text: ['Several Japanese politicians have announced their decisions to seek the leadership of the ']
IO - braille.BrailleHandler.update (23:38:20.805) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleHandler.update (23:38:20.806) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleBuffer.update (23:38:21.930) - MainThread (17984):
Braille regions text: ['Several Japanese politicians have announced their decisions to seek the leadership of the ']
IO - braille.BrailleHandler.update (23:38:21.930) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - braille.BrailleHandler.update (23:38:21.932) - MainThread (17984):
Braille window dots: 2347 15 1236 15 1235 1 123 - 6 2457 1 1234 1 1345 15 234 15 - 1234 135 123 24 2345 24 14 24 1 1345 234 - 125 1 1236 15 - 1 1345 1345 135 136
IO - inputCore.InputManager.executeGesture (23:38:22.824) - winInputHook (12292):

@cary-rowen
Copy link
Contributor Author

I tested the seaka V6 braille display on other computers with office2016 installed, and this problem can also be reproduced.

@LeonarddeR
Copy link
Collaborator

Is UIA enabled when the advanced setting "Always use UI Automation to access Microsoft Word document controls when available" is disabled in alpha builds? If not, it is likely that you're on a version of Office that has limited support for UIA that's not mature enough.

@cary-rowen
Copy link
Contributor Author

Microsoft 365 MSO (16.0.14326.20384) 64-bit
Both of the following options are enabled:
Enable selective registration for UI Automation events and property changes
Always use UI Automation to access Microsoft Word document controls when available

@cary-rowen
Copy link
Contributor Author

Don't forget this issue. All people around me who use braille displays can reproduce this.

@michaelDCurran
Copy link
Member

I was finally able to reproduce this by installing and switching to a Chinese input method such as Taiwan - Microsoft Quick.
then, when scrolling forward, the display keeps jumping back to the caret. This keeps happening for about 15 seconds after I last moved the caret with the arrow keys. After this 15 seconds, it behaves normally again.
Also switching back to English US the problem disappears.
I am investigating further.
@cary-rowen are you using a chinese input method when reading the document?

@michaelDCurran
Copy link
Member

While Chinese Microsoft Quick is runningin Microsoft Word, it seems that we get legacy locationchange / show events for the caret from MSAA. Once per second for about 28 seconds or so.
In NVDA we currently route this caret event straight on toe the focus, whether it is MSAA or not. We certainly should not be doing this if the focus is UI Automation at very least.
I will come up with a pr for this.

michaelDCurran added a commit that referenced this issue Sep 22, 2021
…ents to the currently focused object if the currently focused object is not an IAccessible (MSAA) object.

Fixes #12855 as the Chinese Microsoft Quick input method when used in Microsoft Word was firing MSAA caret events on the Word document, even though the Word document was UIA.
@cary-rowen
Copy link
Contributor Author

当中文 Microsoft Quick 在 Microsoft Word 中运行时,似乎我们从 MSAA 获得了插入符号的遗留位置更改/显示事件。每秒一次,持续约 28 秒左右。
在 NVDA 中,我们目前直接将这个插入符号事件路由到焦点,无论它是否是 MSAA。如果重点至少是 UI 自动化,我们当然不应该这样做。
我会为此提出一个公关。

Hi, Yes, I am using the Simplified Chinese Microsoft Pinyin input method. I tested the build #12868 and it can solve this issue.

michaelDCurran added a commit that referenced this issue Sep 23, 2021
…onger incorrectly jumps back (#12868)

Fixes #12855

Summary of the issue:
When in Microsoft Word, and using a Chinese input method such as Taiwan - Microsoft Quick, and NVDA is accessing Microsoft Word via UI Automation, trying to scroll forward through the document with a braille display causes the braille display to keep jumping back to the original caret position.
This is due to the fact that while the Chinese input method is enabled, legacy MSAA caret events are fired on the Word document window. NVDA is not ignoring these, even though NVDA is configured to access Microsoft word with UIA.

Description of how this pull request fixes the issue:
In IAccessibleHandler.processGenericWinEvent: if an MSAA caret event is identified, before we route this onto the focused object, we first check if the focused object is in deed IAccessible (MSAA), and if it is not (E.g. it is UIA) then we drop the event completely.
This pr also improves MSAA debug logging a little by including the winEvent info in the message about the caret being routed to the focus, and also routes generic events for the focus to the existing focus if their event object is not already the existing focus. This really only stops some extra redundant logging.
@nvaccessAuto nvaccessAuto added this to the 2021.3 milestone Sep 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants