Automation and Screen Reader Support

[frastlin]

I posted
On the main Kivy issue page
But I realized this is the place I should post.
There are quite a few libraries that create accessible widgets for Linux Windows and OS X:
WXPython
PyGUI
Easy GUI

What they do is use pywin32, pyatspi and I believe coco for IOS.
The native designation is what we are after. If an app can be run through automation scripts, then only labeling needs to be done to make it completely accessible.

Kivy seems to be very much based off of pointer access and on mobile platforms this is OK (Although being able to flick from object to object is optimal), but on Windows, Linux and Mac, the mouse is almost never used. In the current Kivy setup, I can't even click a button with the enter key. So along with native widget support, Kivy will need to add keyboard handling to their desktop apps.

The two libraries above that seem to be the best are WXPython and pyGUI.
WXPython is quite an active library, but it is quite a large library and is perhaps a little over-kill for what is needed here. It is, however, the most accessible of all the options above. (There are not that many bugs and it is much more stable).

pyGUI is a lot smaller and I have not really played too much with it, but the examples I ran were kind of buggy, but they worked to some extent. Elements like labels were not the nicest I have ever seen and the buttons were bouncing my screen reader around, but I think that is more the way the key events are being processed.

It is possible to build a wrapper around the system accessibility frameworks directly, but if pywin32 is already included to Kivy and the linux accessibility is in python, there is only OS X and the two mobile platforms to deal with.
I have no experience with IOS or Android, but here are their respective accessibility modules:

Here is the OS X accessibility API for apple
iOS accessibility API
Android

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/8429833-automation-and-screen-reader-support?utm_campaign=plugin&utm_content=tracker%2F77151&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F77151&utm_medium=issues&utm_source=github).

[trivedigaurav]

@brandon: Thanks for your contributions. I think they are quite helpful.

I am not very familiar with the procedures but maybe we should try pushing
for a GSoC project on this topic!

On Mon, Feb 9, 2015 at 2:28 PM, Brandon notifications@github.com wrote:

I posted
On the main Kivy issue page kivy/kivy#2820
But I realized this is the place I should post.
There are quite a few libraries that create accessible widgets for Linux
Windows and OS X:
WXPython http://www.wxpython.org/
PyGUI http://www.cosc.canterbury.ac.nz/greg.ewing/python_gui/
Easy GUI http://easygui.sourceforge.net/

What they do is use pywin32, pyatspi and I believe coco for IOS.
The native designation is what we are after. If an app can be run through
automation scripts, then only labeling needs to be done to make it
completely accessible.

The two libraries above that seem to be the best are WXPython and pyGUI.
WXPython is quite an active library, but it is quite a large library and
is perhaps a little over-kill for what is needed here. It is, however, the
most accessible of all the options above. (There are not that many bugs and
it is much more stable).

pyGUI is a lot smaller and I have not really played too much with it, but
the examples I ran were kind of buggy, but they worked to some extent.
Elements like labels were not the nicest I have ever seen and the buttons
were bouncing my screen reader around, but I think that is more the way the
key events are being processed.

It is possible to build a wrapper around the system accessibility
frameworks directly, but if pywin32 is already included to Kivy and the
linux accessibility is in python, there is only OS X and the two mobile
platforms to deal with.
I have no experience with IOS or Android, but here are their respective
accessibility modules:

Here is the OS X accessibility API for apple
https://developer.apple.com/library/mac/documentation/Accessibility/Conceptual/AccessibilityMacOSX/AboutOSXAccessibility/AboutOSXAccessibility.html
iOS accessibility API
https://developer.apple.com/library/ios/documentation/AVFoundation/Reference/AVSpeechSynthesizer_Ref/index.html
Android
https://developer.android.com/guide/topics/ui/accessibility/index.html

—
Reply to this email directly or view it on GitHub
#113.

[frastlin]

Hello,
what is GSoC?

I have been talking to the developer of pyGUI and he said that the way pyGUI is made, it is inextricably linked with the event and native graphics.
I'm not sure how registering events or using events from something like windows UIA automation server dll or the gnome at-spi2 work.
It would be nice to get some input from someone who has used either one of these.

But if we wished to use pyGUI, it seems as if we will need to run kivy inside pyGUI.
Either that, or hack pyGUI a little so we have the mainloop inside of Kivy's mainloop.

[trivedigaurav]

@frastlin Last year, I participated in Kivy's GSoC (http://kivy.org/docs/gsoc.html) where they involved student interns to work on project ideas like these. Maybe we can include this idea as a part of the proposal for 2015.

[frastlin]

That would be awesome!
Who is the one to talk to about this?

I think that if it were a GSoC project, the goal should be to create a library that any application can call to register their widget types. There is not currently anything like this out there and that excludes screen readers and automated testing from working on any apps made with Tkinter, pygame, Ogre, pyglet, panda 3d, blender, Kivy and basically any library that uses a drawing surface like OpenGL.
I'm pretty sure that the lack of these features means that no truly massive apps (that aren't games) will be developed using any of the above libraries.

[trivedigaurav]

@tshirtman / @brousch ping!

[frastlin]

OK, I'm doing some more google searching and I found
a link that says that pywin32 uses the old MSAA and not UIA
So what we need is someone who has used UIA Automation and their equivalents on the other platforms.
Then we can get the questions answered:

1. How does one register a server widget type
2. how does one obtain the output from the widget
3. How can it fit with an external game loop

Here is one PyUIA autimation that uses IronPython
and here is another that apparently is not very extensive, but it may be a good idea to contact this person as it looks like they wrapped the DLL directly.

[ahicks92]

I've looked at this a lot more, and I'm going to ping a bunch of people I know some of whom make me look like a novice.
No GSOC cross-platform library is possible, not if the implementation is going to be good. Every platform exposes wildly different interfaces for this stuff. Most of us have considered this at one point or another, but it would be a huge effort to say the least. You could maybe do it, but you're always going to have special cases and it'd have to have a long-term maintenance plan of some sort. It's not the kind of thing you can easily declare finished, not as I see it. Here's what I know about the various platforms. This is accurate to the best of my knowledge; as a blind programmer I've looked at most of them at one time or another:
For iOS you're looking at objective-c or possibly Swift object implementations, but the language shouldn't matter so long as you're doing the subclassing properly. The Apple documentation on the subject is the simplest and most straight-forward of the lot and learning to test with Voiceover is the easiest reader to learn to test with. For most apps, you simply override some methods and they get called. Unlike Windows, you are not writing your own marshalling code; Cocoa just does it for you. The fact that kivy does not to my knowledge map to native widgets will complicate things. Specifically, it looks like you need to implement UIAccessibilityContainer for all non-leaf views and UIAccessibility for all leaf views. These do not appear to need to be on the view themselves, but don't quote me on that.
Android may not be possible. Android has a great many accessibility problems and is hated by a majority of blind people. One of these problems is that it's almost impossible to do accessibility from C/C++. I'm putting the almost in because I don't have evidence for completely, but believe this to be the case. In Android, you must override methods on your view in Java and do the entire implementation of the accessibility API as Java classes. It might be possible to have a Java accessibility API that starts by overriding the main view and then calls something down in Kivy, and it might be possible to use JNI. But in either case, it is difficult to do this platform.
I know nothing about Linux. Linux has a really small share of blind people. Really, really small. most of us try it as our desktop platform for about 15 minutes and then leave forever, and those who use it regularly typically end up living in the terminal. It also has very little documentation on the subject.
OS X is basically iOS but the classes are named differently and are a bit more complicated. The difficulty for OS X should be about the same as that for iOS, i.e. the easiest platform to make easy gains on.
Windows is more complicated. Windows has 3 major screen readers with a long history of hacking around stuff and hooking your graphics driver and doing all sorts of fun. Windows also has at least 3 accessibility APIs: MSAA, UIA, and IAccessible2. IAAccessible2 is an unofficial extension of MSAA and is used by Eclipse and Firefox. UIA is Microsoft's new "official" option. MSAA by itself is not of interest to us because, among other things, you apparently can't make text editing fields accessible with it. if I was doing Windows, I'd use UIA because that gains narrator, the built-in screen reader which is slowly becoming capable these days, and everything should be supporting it by now. Unlike other platforms, Windows accessibility is done via COM, so any language which can both implement a COM interface and get access to the windows callback for the main Kivy view should work. Should this be impossible, I have an untested theory that says you can still do it via just having access to the window handle, but this is a last resort. I believe that the accessibility for Windows can be done in Python via Comtypes; if not, I'd bet on it being possible to patch Comtypes to make it possible. The problem with Windows is testing because of the multitude of screen readers; the good news is that Windows is the single most popular platform for blind people.
The two platforms you care about immediately are iOS and Windows. These two are the favorite PC and favorite phone platform of people using screen reading technologies. Linux is probably last on the list; it's complicated and unpopular.
One way of going about this might be to make a set of kivy-specific queries and notifications that cover specifically what Kivy needs and only what Kivy needs, and then use them.
Now, let's ping people who can probably correct half of this: @jcsteh, @mwcampbell, @MarcoZehe
I'm going to hit 4 or 5 more via e-mail as well.
I can probably do Windows. I can maybe do iOS. If this becomes a GSOC or something, I'm potentially interested. All the people I know of with experience doing this kind of work already have full-time jobs doing this kind of work; I'm by no means an expert, but I do at least know where one starts. O, and can i ever test my code.

[frastlin]

Kivy has a limited number of widgets and most deal with the basic edit field, menu bar, applications menu, button and label.
What does WXPython do for accessibility in these areas? Because WX is pretty accessible on each platform WX supports...

[mwcampbell]

wxWidgets basically dodges the whole accessibility problem by using the native widgets on each platform.

[ahicks92]

Some conversation on Twitter got me looking at this again. Should this issue be filed against a different project? Looking at how things are split, it looks like this touches multiple sub-projects.

[blindguydiy]

Good day, I am a blind user and startup programmer in python. For the statement on blind people hate android I would disagree. I found android easier to use then iPhone. I want to write a app for android that can do ocr of images and more but using python kivy is about the only solution I found. So it would be great if a solution can be found. Do kivy have a option or flag that can describe the button or label when moving your finger over it like wx python have a option to include a description for widgets? I googled and could not found anything. If kivy is more accessable I could not found any documentation about it or how to do it. Thank you

[ahicks92]

@blindguydiy
There are demographic surveys that disagree with your assessment of Android. From the perspective of benefiting the maximum number of users, iOS wins. For instance here. Nonetheless Android still remains the most difficult platform by far to implement accessibility for without the library in question being in Java to start with.

This library uses direct rendering to graphics for it's controls if I remember properly, the upshot of which is that simple accessibility properties such as those you are hoping for require a full accessibility implementation, an effort on the order of 5000 lines or more.

if you are looking for a reasonably blind accessible way to do mobile app development--iOS, Android, or both--I'd suggest React Native. It's not tied to an IDE, they have accessibility built in, and your app will work on both iOS and Android. I am not aware of a Python UI library which is both accessible and supports mobile.

[mwcampbell]

@camlorn Kivy's pyjnius project has already solved the problem of implementing JNI interfaces from Python. So that specific problem is not a barrier to implementing Android accessibility in Kivy.

I saw the Bountysource link on this issue, and have pitched in $1000 out of my own pocket. That may barely make a dent in the problem, but it's what I can afford personally. I'll also be happy to offer advice to anyone who steps up and works on this problem, particularly for the Windows implementation, but I can't do the work myself, at least not now.

[denim2x]

We could use pyWinAPI (specifically IRawElementProviderSimple) for implementing automation features on Windows; on Linux, there's PyGObject with Atk.GObjectAccessible and Atk.ObjectFactory. I should have a working prototype within a week, starting with Windows

[mwcampbell]

It seems to me that the work of implementing platform accessibility APIs belongs in the main Kivy project, not Plyer, since it needs to be an integral part of the UI framework. See also kivy/kivy#5836.

[hackalog]

Tricky. There’s a huge platform specific part to it, too (i.e. plyer, or pyjnius/pyobjus)

…

On Sun, Jun 16, 2019 at 10:55 AM Matthew Campbell ***@***.***> wrote: It seems to me that the work of implementing platform accessibility APIs belongs in the main Kivy project, not Plyer, since it needs to be an integral part of the UI framework.

[mwcampbell]

I'm a newcomer to Kivy, so anything I say about project structure should be taken with a grain of salt. But there is already a precedent for using pyjnius and pyobjus directly within the main Kivy package for things that are integral to the UI -- specifically, the clipboard.

[blindguydiy]

Good day. Well from the accessibility between android and iPhone in South Africa I found a lot of blind people I know are going over to the android system for they have much more free accessibility apps. But that a side. I will have a look at React Native, but feel still there must be some way to make kivy accessibility. I am developing on windows and trying on android to make a accessibility app for free for people like me. For I know how it feels that you have to pay for what a seeing person can do for free just looking at a document or product. There is apps available out there but cost a few bucks. For some reason people think the blind community is very rich, lol. But if you take a small figure like 300000 out of those 300000 only realy 50000 blind users can really afford the accessibility apps and assistive gadgets. That is why I feel like that at least in the computer world of today that all programs should at least be some accessibility conscious. To give access to all information to all across the board and not excluding a minority because of a disability. But will look around to see if I can find some other solution. What happened to the papi pygame from 2008. Just saw they tried to do some accessibility with pygame.
Th

[MesterPerfect]

It's 2023, and there doesn't appear to be any update on Kivy being compatible with screen readers on either Windows or Android.

[DataTriny]

For anyone interested in this, the current tracking issue is kivy/kivy#8596