Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Qt 5.11+ accessibility requires updates on the NVDA side #8604

Closed
ghost opened this issue Aug 7, 2018 · 53 comments
Closed

Qt 5.11+ accessibility requires updates on the NVDA side #8604

ghost opened this issue Aug 7, 2018 · 53 comments
Labels
Abandoned requested reports or updates are missing since more than 1 year, author or users are not available. component/qt Qt fraphical framework p3 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority

Comments

@ghost
Copy link

ghost commented Aug 7, 2018

Qt 5.11 replaced its legacy MSAA-based Windows accessibility backend with a new implementation based on UI Automation. This seems to be causing some issues with NVDA. I'm currently investigating the following bug report, which includes a detailed description of the issues, as well as a sample project to reproduce them:

https://bugreports.qt.io/browse/QTBUG-69472

While some of the issues may result from bugs in our implementation, some others may require updates in the NVDA side, which uses a custom app module for Qt applications, which may no longer apply since Qt 5.11.

I took a look at NVDA sources and saw that it supports MSAA and UI Automation, and has app modules that add custom behavior and fixes for many applications and frameworks, including Qt. The Qt app module in NVDA is based on MSAA/IAccessible, used in previous Qt versions. When used with a Qt 5.11 application, NVDA is recognizing that it supports UI Automation and using its UI Automation module.

However, this module seems to be lacking some functionality. For instance, the automatic detection of changes in text selection seems to have been disabled for some reason. The Qt framework is raising "UIA_Text_TextSelectionChangedEventId" notifications to inform UIA clients of changes in selection. However, NVDA seems to be purposely ignoring it (the mapping from UIA_Text_TextSelectionChangedEventId to the "caret" event is commented out in the _UIAHandler.py source). Also, NVDA's UIA module uses the "EditableTextWithoutAutoSelectDetection" class, instead of the "EditableTextWithAutoSelectDetection" used on the IAccessible module. I tried enabling the UIA_Text_TextSelectionChangedEventId mapping and changing the UIA module to use EditableTextWithAutoSelectDetection, and some of the compatibility issues reported, like NVDA failing to speak the mouse selection, or the lack of updates in the Braille display where fixed. However, I'm not familiar with the NVDA code, and that change will likely break other stuff.

I would suggest keeping the current custom app module in place for supporting Qt versions prior to 5.11 and using the UIA module for >= 5.11 (i.e. whenever UIA support is detected), while enabling the necessary features to make it work properly. I can help with changes on the Qt side.

Also, please instead of creating a new custom app module for Qt 5.11+ to work around bugs in our implementation or behavior unexpected for a proper UI Automation server, please report them to us so we can have them fixed on our side instead (you can assign bug reports to me).

Note: the new accessibility backend is a pure UI Automation implementation. Legacy MSAA clients are supported through the UIA-to-MSAA bridge provided by the MS UIA framework in the middle. However, the use of the bridge was broken in the initial 5.11 release, but will be fixed in 5.11.2 (the fix is already in our 5.11 branch, if you build Qt yourself). However, the use through the bridge does not provide all functionality that may be required. In particular, it does not provide IAccessible2 support, only IAccessible. Some features like caret movement detection are in IAccessible2, so I think a proper UI Automation interfacing is the way to go, instead of trying to use it in MSAA mode (also, even going through this legacy mode would also require changes in NVDA, as it detects the UI Automation support in Qt 5.11 and seems to ignore the MSAA-based app module).

@josephsl
Copy link
Collaborator

josephsl commented Aug 7, 2018 via email

@LeonarddeR
Copy link
Collaborator

@anrocqt commented on 7 Aug 2018, 16:42 CEST:

... as well as a sample project to reproduce them:

I might be missing something, but could you please provide a executable binary of this application? That will make testing a whole lot easier I think.

@ghost
Copy link
Author

ghost commented Aug 10, 2018

SimpleBox.zip

@LeonarddeR: Added a binary of the sample application statically built against the current code in our 5.11 branch.

@ghost ghost closed this as completed Aug 10, 2018
@ghost ghost reopened this Aug 10, 2018
@LeonarddeR
Copy link
Collaborator

@anrocqt commented on 7 Aug 2018, 16:42 CEST:

... which uses a custom app module for Qt applications, which may no longer apply since Qt 5.11.

Strictly spoken, NVDA uses special NVDAObject types for some QT widgets that recide in the NVDAObjects.IAccessible package. These will not apply to the UIA implementation at all, as UIA normally has preference over MSAA/IAccessible.

For instance, the automatic detection of changes in text selection seems to have been disabled for some reason. The Qt framework is raising "UIA_Text_TextSelectionChangedEventId" notifications to inform UIA clients of changes in selection. However, NVDA seems to be purposely ignoring it (the mapping from UIA_Text_TextSelectionChangedEventId to the "caret" event is commented out in the _UIAHandler.py source).

@jcsteh: Could you elaborate on why you commented this out, according to git blame?

As a side note, I noticed that NVDA is unable to detect bold and itallic text in the text browser. @anrocqt: could this implementation be incomplete on the QT side?

@jcsteh
Copy link
Contributor

jcsteh commented Aug 10, 2018 via email

@derekriemer
Copy link
Collaborator

just want to make sure this doesn't stall, because QT is hugely important.

@LeonarddeR
Copy link
Collaborator

@anrocqt: Testing with JAWS 2018 revealed that the formatting changes are not detected within the text browser. The font size is reported as 0 and the bold and italic states are not reported.

@jcsteh commented on 10 aug. 2018 23:12 CEST:

Because there's no way we can currently selectively blacklist certain events in specific apps at the UIA core level, we had to disable them altogether.

Do you recall whether these freezes in NVDA were occurring just because of the flood of text selection events, or because every text selection event executed an event_caret internally?

@LeonarddeR
Copy link
Collaborator

@anrocqt: Is there anything to add now QT 5.12 has seen the light?

@shoogle
Copy link

shoogle commented Mar 12, 2019

Can this be revisited now? This has affected MuseScore and other Qt applications. It seems that Qt is doing everything right, but NVDA is purposefully ignoring accessibility events to compensate for bugs in other (non-Qt) applications.

Here are the relevant lines in nvda/_UIAHandler.py where various events are being ignored (they are commented out).

@jcsteh

These events were disabled because certain UIA providers (I think WPF 2.5 or something like that from memory) flood these events and cause NVDA to become unusable.

I appreciate that the changes allowed NVDA to work with those applications, but it also removed any incentive for developers to fix bugs in those applications. Worse, if the developers test with NVDA they might come to the conclusion that there are no bugs in their application.

In the past it was necessary for screen readers to go to great lengths to make non-accessible applications accessible. However, times are changing, and the onus is now on application developers to make their applications accessible. Let's reward applications that get it right rather than penalising them for the sake of preserving accessibility in applications that get it wrong.

@jcsteh
Copy link
Contributor

jcsteh commented Mar 12, 2019

It's not just about allowing NVDA to work with those applications. The problem is that with these events enabled, any app based on those frameworks (whether the author cares about accessibility or not) will cause a complete hang of NVDA. Prior to Windows 10, there is no way NVDA can work around this. That said, this should only affect reporting of text selection via the mouse. Text selection via the keyboard should work, as should anything else (focus, property changes, etc.).

I don't understand how this could cause the issues being experienced with MuseScore. Surely it's not using text selection events to report cursoring around the Score?

@LeonarddeR
Copy link
Collaborator

I've compared NVDA with Narrator with regard to MuseScore, and at least with NVDA, combo boxes do not behave as they should, that is, arrowing through the combo box without opening it does not announce selection changes, whereas with Narrator, this works just fine.

@LeonarddeR
Copy link
Collaborator

LeonarddeR commented Mar 12, 2019

The QComboBox comboboxes consist of a root object (the combobox itself) and its children (a list with the combobox items, and an edit control). It looks like that NVDA is not receiving any PropertyChangedEvents when changing the value of the combobox without expanding it. Note that eventHandler.shouldAcceptEvent tends to drop some events, but that's certainly not the case here, as the events aren't even entering the event handler of the UIAHandler. Therefore, I have no idea what's going on here. Cc @michaelDCurran

Having said that, the QComboBox neither implements a ExpandCollapse nor a Selection pattern, which, according to the Microsoft Docs, are required. Cc @anrocqt

@shoogle
Copy link

shoogle commented Mar 13, 2019

As @LeonarddeR said, navigation within unexpanded combo boxes is not being announced by NVDA. Interestingly, this was working under Qt 5.9 (which used MSAA) but not under Qt 5.12 (which uses UI Automation). There are other examples of this:

  • Under Qt 5.12 only control names (i.e. not descriptions) are announced by NVDA. Both names and descriptions were announced under 5.9.
  • Changes to selections made in the Score View were announced under Qt 5.9 and not under Qt 5.12.

If you want to confirm that the above is indeed true then you will need an old version of MuseScore that uses Qt 5.9. Subsequent versions use Qt 5.12, and the MuseScore developers are reluctant to switch back.

Regarding the Score View, I suspect that MuseScore was using a "trick" to get NVDA to read whatever was in the status bar, so it's possible that this is an issue at MuseScore's end. Nevertheless, the "trick" was working under Qt 5.9 and MSAA and now it is broken under Qt 5.12 and UI Automation.

If we are doing something wrong here then it would be very helpful if somebody could point us in the direction of a tutorial / example of how to implement accessibility for a custom control. Most accessibility guides just say "don't use custom controls, use built-in controls", but MuseScore's Score View is a custom control out of necessity (after all, if there was a standard way to display musical symbols in Windows then there would be no reason for MuseScore to exist).

@shoogle
Copy link

shoogle commented Mar 18, 2019

It would be great if somebody could try building it with the relevant lines in nvda/_UIAHandler.py uncommented so that all events are handled, and then testing it with MuseScore to see which of the issues highlighted so far still remain, if any:

  1. Arrowing through non-expanded combo boxes is unannounced.
  2. Control descriptions are unannounced.
  3. Navigation and selection changes within the Score View is unannounced.

I tried to build NVDA myself but I don't have enough room on my PC for all the dependencies.

@LeonarddeR
Copy link
Collaborator

LeonarddeR commented Mar 18, 2019 via email

@shoogle
Copy link

shoogle commented Mar 18, 2019

@LeonarddeR, thanks! You didn't notice any change at all?

@jcsteh

Surely [MuseScore]'s not using text selection events to report cursoring around the Score?

Cursoring changes the selection, but what event would you expect to see here?

@jcsteh
Copy link
Contributor

jcsteh commented Mar 19, 2019 via email

@shoogle
Copy link

shoogle commented Mar 19, 2019

Is there a UI Automation event we can fire that would force NVDA to read the text in the status bar, or arbitrary text of our choosing that is not necessarily displayed anywhere on the screen?

@LeonarddeR
Copy link
Collaborator

@shoogle: If I understand you correctly, all was well with older versions of MuseScore based on qt<5.11. What was the latest version that utilized this, so I can investigate the differences?

@jcsteh
Copy link
Contributor

jcsteh commented Mar 20, 2019

I haven't investigated or tested this and don't have time to do so. The following is provided in the hopes that it may help, but without any guarantee of usefulness. :)

  * We send [`QAccessibleValueChangeEvent`](https://doc.qt.io/qt-5/qaccessiblevaluechangeevent.html) with the [object being the score view](https://github.com/musescore/MuseScore/blob/dd3e6859565702bac7905ab424d3314ff82245d5/mscore/scoreaccessibility.cpp#L194) and the [value being the new status text](https://github.com/musescore/MuseScore/blob/dd3e6859565702bac7905ab424d3314ff82245d5/mscore/scoreaccessibility.cpp#L177-L179).

* According to [Qt source code](https://github.com/qt/qtbase/blob/5733dfbd90fd059e7310786faefb022b00289592/src/plugins/platforms/windows/uiautomation/qwindowsuiamainprovider.cpp#L155), `QAccessibleValueChangeEvent` raises `UIA_RangeValueValuePropertyId`.

The way MSAA and UIA handle value is quite different. In MSAA, there is only one value change event and it represents a change to the accValue property, which is generally a string. In UIA, there are two concepts of value, represented by the RangeValue pattern (numeric) and Value pattern (text). It looks like Qt only fires an event for the RangeValue change, but I'm guessing that the value in this case is text.

To make things more confusing, NVDA only has one concept of value. If the RangeValue pattern is present, it will choose RangeValue. Otherwise, it chooses Value. Qt exposes the RangeValue pattern if the accessible has the value interface (numeric). I don't know whether this control exposes that or not; I guess not, since it's not a numeric control.

So, to fix this, I think:

  1. This control must not expose the numeric value interface. (It probably doesn't already, but I'm not sure.)
  2. In Qt, if the value interface isn't present, it should fire the ValueValue property change event, not the RangeValueValue property change event.

As an ugly, hacky workaround, you could perhaps use the name instead of the value to expose the information. That would of course override any proper name you have set.

@LeonarddeR
Copy link
Collaborator

LeonarddeR commented Mar 20, 2019

Dropbox 70 switched to qt 5.12 and also suffers from the combobox issue. I'm marking this P2 to start with.

Update: Also discovered that JAWS suffers from exactly the same issue with regard to combo boxes. Selection changes are only exposed when the combobox is expanded.
Update2: Also reproduced with Narrator on Windows 10 1809. Will test with 1903 later today.

@LeonarddeR LeonarddeR added the p3 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority label Mar 20, 2019
@ghost
Copy link
Author

ghost commented Apr 9, 2019

Hi. Thanks for reporting the issues you found with the Qt accessibility framework on Windows. I apologize for the delayed response. About some of them:

  • "It looks like that NVDA is not receiving any PropertyChangedEvents when changing the value of the combo box without expanding it."
  • "QComboBox neither implements a ExpandCollapse nor a Selection pattern, which, according to the Microsoft Docs, are required."
  • "Changes to selections made in the Score View were announced under Qt 5.9 and not under Qt 5.12."

In spite of what the docs say, it seems Microsoft is not implementing the Selection pattern in their own combo boxes. I checked with the Inspect utility, examining a combo box in the folder options dialog in File Explorer, on Windows 10, and it only implements the Value and ExpandCollapse patterns. And it is working with NVDA. Only the list view inside the combo box structure implements the Selection pattern, as does Qt. Also, the ExpandCollapse pattern makes NVDA say "Expanded"/"Collapsed", but apparently only when doing it with the mouse, which does not seem to be a concern.

However, you are right about the missing UI Automation notifications when the combo box is changed with the keyboard without expanding it. Also, notifications were not being sent for updated string-type values (Value pattern), but only for range-based ones (RangeValue pattern) which probably explains the missing updates in MuseScore.

I created this bug report to track the missing notifications: https://bugreports.qt.io/browse/QTBUG-75066
And submitted the following fix for it (under review): https://codereview.qt-project.org/#/c/258622/

It should fix the combo box issues, and hopefully the Score View selection as well.

  • "Under Qt 5.12 only control names (i.e. not descriptions) are announced by NVDA. Both names and descriptions were announced under 5.9"

I have checked with Narrator, and it is saying both the names and descriptions, so it may be an issue in NVDA. Perhaps due to differences on how these fields are returned by MSAA and UI Automation.

If you can build Qt 5.12 yourselves with the proposed patch, please let me know if it actually fixes the issues you are experiencing, as well as any other issues you may have found.

Thanks,

André

@LeonarddeR
Copy link
Collaborator

As of dropbox 72 stable, they switched to QT 5.12

@shoogle
Copy link

shoogle commented Jun 25, 2019

Many thanks to @jcsteh for identifying the issue with Value pattern events, and to @anrocqt for fixing them in Qt! I just tested in Qt 5.12.4 and can confirm that the issues with combo boxes and score view selections in MuseScore have disappeared.

"Under Qt 5.12 only control names (i.e. not descriptions) are announced by NVDA. Both names and descriptions were announced under 5.9"

I have checked with Narrator, and it is saying both the names and descriptions, so it may be an issue in NVDA.

Narrator has now stopped announcing descriptions too, at least by default. However, they are available if you press Narrator+0 (i.e. CapsLock and Zero). Does NVDA have a similar shortcut to announce descriptions? (I looked but couldn't find one.)

@Adriani90
Copy link
Collaborator

cc: @feerrenrut

@LeonarddeR
Copy link
Collaborator

@anrocqt This got a bit out of my view. How is the current state of QT accessibility? I heard rumours of severe brokenness in version 5.14.

@LeonarddeR LeonarddeR changed the title Qt 5.11 accessibility requires updates on the NVDA side Qt 5.11+ accessibility requires updates on the NVDA side Mar 19, 2020
@Adriani90
Copy link
Collaborator

Now that QT is going more and more towards version 6.0, is there any status update on this issue?

@DrSooom
Copy link

DrSooom commented Sep 15, 2020

Was this issue solved by NVDA 2020.3 Beta 1 – because it wasn't closed?

@michaelDCurran
Copy link
Member

Fixed in pr #10889

@LeonarddeR
Copy link
Collaborator

@anrocqt If there are any remaining problems, I'd suggest opening new issues for them separately.

@chigkim
Copy link

chigkim commented Sep 5, 2021

Since NVDA maps FullDescription to description, it no longer announces HelpText. How do users get the message in HelpText when there are both FullDescription and HelpText? Is there a setting or shortcut to report HelpText?

@LeonarddeR
Copy link
Collaborator

@michaelDCurran wrote in #10889 (comment):

Adding a helpText property is okay, though perhaps a little too over-engineered if there are no cases of both of them returning valid info at the same time. Do you think that there are situations in UIA where helpText and fullDescription do give different (and valid) results? Rather than perhaps just using fullDescription but falling back to helpText if empty. If we do implement a helpText property, then we can map IAccessible::accHelp to that as well I guess.

May be we should yet consider splitting description and help text. @chigkim: Could you please open a new issue for this?

@cppqtdev
Copy link

Hi Team
I am facing one issue with NVDA Screen Reader when the focus goes to the Listview in Qt /QML Application
it is not reading 1 of 5 ,Let's total 5 item in the list and current item is 1 . then it should read name of the item and then read selected and then 1 of 5 or 1 of 5 item selected .

Note : When I tested this on Window Inbuilt Screen reader like narrator it read properly and also JAWS also read same behavior as Narrator screen reader .

Please support if possible .What we need to do ?

@beqabeqa473
Copy link
Contributor

beqabeqa473 commented Oct 11, 2022 via email

@LeonarddeR
Copy link
Collaborator

Could you please provide some examples to test with?

@LeonarddeR LeonarddeR reopened this Oct 11, 2022
@cppqtdev
Copy link

Hi leonardder ,
you can test with this qt qml application :
https://drive.google.com/file/d/1FQFkaLHinkJ0KIOo_RrTk7fm6GRywUQs/view?usp=sharing

@shoogle
Copy link

shoogle commented Oct 11, 2022

Could we discuss new problems in new issues please?

I know it's tempting to post here because a Qt developer is subscribed, but the risk is that they will unsubscribe if all the comments are about NVDA bugs that happen to affect Qt software, rather than things that are actually the fault of Qt.

If you create a new issue and it does turn out to be on the Qt side rather than the NVDA side, you could leave a single comment in this thread linking to it to let subscribers know that it exists, but could we avoid having the discussion here please?

@ethindp
Copy link

ethindp commented Oct 11, 2022

@shoogle I'm pretty sure that these problems are a combination of NVDA and QT. Both need to do their part. QT needs far more attention from NVDA if this problem is to be resolved.

@shoogle
Copy link

shoogle commented Oct 11, 2022

@ethindp

  1. It was said above that JAWS and Narrator are reading Qt lists properly, so it appears (at first glance at least) that this is purely an NVDA problem, so there is no reason to notify Qt developers at this stage.

  2. Even if it does turn out to require some action on the Qt side, it is still a new problem that has very little to do with the original topic of this issue. It would be better to address it in its own dedicated issue so that people searching for "NVDA doesn't say list item numbers" (or similar) would be more likely to come across it.

@ethindp
Copy link

ethindp commented Oct 11, 2022

@shoogle Your first argument is correct, or would be, if it weren't for the fact that Orca, on Linux, has similar problems regarding QT accessibility in certain contexts. A standard QPlainTextEdit, for example, cannot be navigated without modifications of some kind (I am looking into this). On Windows, edit boxes of any kind, bar QLineEdits with single-line entries, cannot be navigated at all, at least with NVDA. If it weren't for the fact that both Orca and NVDA both suffer this problem, I would agree with you that this is purely an NVDA issue. However, it is clearly a problem with both NVDA and QT, or for all we know it could purely be a QT issue. Tracking down exactly what causes this problem will be difficult but is not impossible. Your second argument would be applicable if that was the only problem that NVDA and QT have, but it isn't. I think that the reason people continue commenting here is not just because a QT developer is subscribed to this issue but because its easier just to consolidate all of them into a single issue. Perhaps it might be better to create separate issues for each QT-specific issue, then create a central issue where we can have a list of all the issues collected and their status via a task list.

@shoogle
Copy link

shoogle commented Oct 11, 2022

@ethindp, what do QLineEdits have to do with the ListView problem raised earlier? You seem to be talking about something else entirely, which further illustrates the need to use separate issues to avoid confusing people.

Qt provides an abstraction layer for the underlying accessibility frameworks, which are very different on Windows and Linux. If Orca and NVDA exhibit similar problems then that could indicate a Qt problem, but not when all the other Windows-based screen readers are working properly with Qt, which appears to be true in the case of the ListView problem.

Perhaps it might be better to create separate issues for each QT-specific issue, then create a central issue where we can have a list of all the issues collected and their status via a task list.

Yes please! I don't mind if this is used as the central issue, but I don't think the back-and-forth discussion about whether a particular problem is or is not on the Qt side should take place here.

@cppqtdev
Copy link

Could we discuss new problems in new issues please?

I know it's tempting to post here because a Qt developer is subscribed, but the risk is that they will unsubscribe if all the comments are about NVDA bugs that happen to affect Qt software, rather than things that are actually the fault of Qt.

If you create a new issue and it does turn out to be on the Qt side rather than the NVDA side, you could leave a single comment in this thread linking to it to let subscribers know that it exists, but could we avoid having the discussion here please?

Hi @shoogle ,
I don't know this problem is from qt side or NVDA side . If this problem is from qt side how other screen reader read current syntax . ??

I am stuck In between that's why I post here this issue .

I also faced one another problem with NVDA . NVDA also does not properly work on slider . Only First time it read properly whenever we move the slider using the --> and <-- (left or right button) it will not speak anything while other screen reader like narrator read slider at 5 (current value of slider).
But may be Qt follow windows screen reader (narrator ) only . May be Qt not focus on other screen reader how they behave on Qt controls ...

@Adriani90
Copy link
Collaborator

@ethindp and @cppqtdev as far as I understand two issues are still open:

  1. List view is not presented correctly in NVDA, works better with other screen readers
  2. Sliders do not work with NVDA, the value is not being reported when navigating with arrow keys.

If this is correct, could you please open an issue with these two problems and attach a test application where we can reproduce the list and the slide issue? Preferably as a binary application or something that can be run without having to install the whole development environment.

That would be very appreciated.

@Adriani90 Adriani90 added the Abandoned requested reports or updates are missing since more than 1 year, author or users are not available. label Oct 31, 2024
@Adriani90
Copy link
Collaborator

We didn't get any updates from people who could reproduce this since more than two years, even though we requested to open new issues for the remaining problems including test applications to test with.
I am closing as abandoned. If you are still having the issues mentioned in my last comment above, please open new requests, filling the templates accordingly.

@Adriani90 Adriani90 closed this as not planned Won't fix, can't repro, duplicate, stale Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Abandoned requested reports or updates are missing since more than 1 year, author or users are not available. component/qt Qt fraphical framework p3 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority
Projects
None yet
Development

No branches or pull requests