Screen reader accessibility

[mwcampbell]

I want to implement accessibility for screen readers in Ren'Py. This is different from self-voicing; instead of sending text directly to a speech synthesizer, this will make information about the content and structure of the UI available through platform-native accessibility APIs, so the UI can be made accessible by a screen reader, as well as other tools such as screen magnifiers and alternate input solutions for mobility-impaired people.

I want to do this using my AccessKit project, which provides a cross-platform abstraction for accessibility and implements the platform-specific accessibility APIs. So far it only has a Windows implementation, but Mac and Linux implementations are in development. AccessKit itself is written in Rust, but I plan to write Python bindings. I'm aware that on Windows, the native components of Ren'Py are built using MinGW. This shouldn't be a problem; I've already verified that AccessKit for Windows works with the GNU toolchain. This does, however, mean adding Rust as a build-time dependency.

This will require some API design work in Displayable, which is why I'm opening the issue now. With the current self-voicing implementation, a displayable only needs to provide text to be spoken. When implementing accessibility APIs, we need some more information. At a minimum, each displayable will need a role (the type, e.g. static text, button, editable text field), a name, a bounding rectangle, and flags indicating whether it is focusable and focused. BTW, static text is generally not made focusable. It doesn't need to be, because a user can use their screen reader's keyboard commands to review the static text, and once I implement live regions in AccessKit, the static text can also be read automatically if that's appropriate. There are also a variety of other properties for different types of UI elements (called nodes in AccessKit), and they form a tree, like the HTML DOM. Once a frame, Ren'Py will need to gather any new or updated nodes into a list and push that to AccessKit, which will fire the appropriate platform-specific events depending on what changed.

Screen readers and other accessibility tools can request actions on UI elements. The two most common actions are setting the keyboard focus and activating the element (doing the equivalent of a click). Ren'Py will need to implement a callback to handle these. AccessKit is free to run that callback on any thread, so Ren'Py will need to dispatch the actions to the main thread, presumably through a user-defined SDL event.

I haven't yet figured out what modifications to SDL, if any, will be needed; as far as I know, this will be the first time an SDL-based application is implementing accessibility APIs. On Windows, the window procedure (where native events are handled) needs to implement the WM_GETOBJECT window message. But I can inject a handler for that message using a Win32 technique called subclassing. I'm not sure yet what integrations will be needed on Mac, Linux, iOS, or Android. One issue I do know about, which applies to multiple platforms, is that event pumping needs to be continuous, without a fixed delay between iterations; at the same time, we don't want to busy-wait, so the main loop should actually wait for OS events when there aren't any. SDL addressed this in 2.0.16, so I already opened renpy/renpy-build#43 about that.

[renpytom]

I've taken a few minutes to look at AccessKit, and I think that there are some issues that might need to be addressed before this could be integrated with Ren'Py.

At least from your basic description, I like the design of it - a cross-platform accessibility library seems like a great idea, and I like the idea of building up a tree of nodes and then giving the screenreader access to it.

What worries me quite a bit is the size of AccessKit, and the dependency on rust and its various libraries. I'm not really a rust expert, but what I did was to use cargo -b --release to build it, and then I took a look at the various .rlib files that are generated, and get:

8	./deps/libinstant-50a6b7fe16e39b35.rlib
12	./deps/libcfg_if-0b0f8bb485e864e6.rlib
16	./deps/libfnv-be7c2eecc9c214dc.rlib
28	./deps/libdarling-ec23232bcb09e505.rlib
36	./deps/libident_case-01058272244cfd34.rlib
36	./deps/libscopeguard-96a9792e29e00096.rlib
132	./deps/librand_core-f7d8b35cb43fced9.rlib
200	./deps/libunicode_xid-03fcef8d2fa20199.rlib
268	./deps/libenumset-9f253e1d87b04cf4.rlib
300	./deps/libparking_lot_core-685f2dabf35add04.rlib
300	./deps/libsmallvec-7deaf4ab35f2cd8d.rlib
324	./deps/libarrayvec-32b5dc803418db95.rlib
408	./deps/liblock_api-c045dca1b1b4992c.rlib
424	./deps/librand_xoshiro-5921a9dc509e19b6.rlib
476	./deps/libsized_chunks-5e1add544b78b2d9.rlib
488	./deps/libparking_lot-fea9587b1655ab89.rlib
604	./deps/libquote-ff890ed16fa0cca9.rlib
620	./deps/libversion_check-5fd6b0381ad4290c.rlib
648	./deps/libstrsim-554b69673712c4b6.rlib
824	./deps/libaccesskit-49b49b807b7592a0.rlib
872	./libaccesskit_consumer.rlib
1320	./deps/libproc_macro2-36fe21c8d2537cb0.rlib
1440	./deps/libbitmaps-fc7291b3de33adc3.rlib
1540	./deps/libkurbo-b9ff762f57f80dd9.rlib
1972	./deps/libtypenum-0638aeb7fed1f3b7.rlib
2124	./deps/libim-51a780b199215474.rlib
2568	./deps/liblibc-0a537585f1e520fe.rlib
5548	./deps/libdarling_core-557e66d77fd739c4.rlib
13292	./deps/libsyn-5965add144a7b90a.rlib

The sizes are in kilobytes.

Those seem to contain LLVM IR, so I'd assume they'd shrink somewhat before being added to Ren'Py, but right now this sums up to 36M in size. That's larger than the ~27MB per platform for all the other binary dependencies of Ren'Py.

What I'd be really interested in doing would be to dynamically link to a system install of AccessKit. That's similiar to what we're doing with the Steam and Live2D libraries. I'd be very interested in using a public C API that the data about the DOM nodes could be fed into, as you described but without the Python bindings.

I don't think I could distribute a library this large and dependency-heavy directly with Ren'Py - things like syn don't really seem like they have a place in Ren'Py. AccessKit would have to be about a megabyte in size for it to make sense to distribute with Ren'Py, rather than dynamic linking it through ctypes.

[mwcampbell]

I think I can assuage your concerns about code size. A lot of what you're seeing are dependencies that are only used at build time. The actual size of a simple program that incorporates AccessKit via static linking (the Windows hello_world example) is only 358K in a release build. (You can see this for yourself by running cargo build --release --example hello_world under platforms/windows). I then modified that program to deserialize the tree structure from JSON, and the result was 693K in a release build; I figure that's more representative of the glue code that will be necessary in a complete Python binding. Granted, AccessKit is still quite incomplete, but I'll be surprised if it ever passes 2 MB, and it should stay closer to 1 MB.

Also, you'll never need to work with the .rlib files directly. For a C library crates, Cargo produces a single static or dynamic library that has all the runtime dependencies linked into it.

Still, I've promised a C API anyway, so maybe I should start there.

[renpytom]

Ah, forgive me, I misunderstood what would be involved. Those are sizes that make a lot of sense as something that could be distributed with the engine.

I do think a C api would be quite nice, regardless, and would open AccessKit up to more games.

Hearing this I think I would want to integrate AccessKit, once the SDL integration is ready.

[mwcampbell]

Given that I've addressed your concerns about code size and linking requirements, and you're willing to distribute AccessKit with the engine, would you prefer to work with a C API or a CPython extension module? Just doing a C API could save me some work, since I'd be able to use that same API to integrate AccessKit into, say, a Unity game.

I can also think of two possible approaches to a C API: either I could expose all the Rust structs and enums, with individual functions for getting and setting fields, or I could let you pass in a JSON string of the whole structure. The latter would be easier to implement, and I think it could also be better for runtime environments like Mono/.NET (used by Unity), where there's a JIT compiler and each individual native function call is fairly expensive. With CPython, on the other hand, the JSON serialization approach might be less efficient, but still probably more convenient to get up and running.

[renpytom]

Both would work for me, but I think the JSON string would probably be simpler. It basically reduces the API down to that single function, plus whatever is needed to integrate with SDL. That means I could do a small amount of binding work, and then everything else could be done in Python.

I think the important thing would be that the format of the JSON is well-documented, and that it remains stable from release to release.

[mwcampbell]

The JSON format is neither documented nor stable yet, but it will be by the time AccessKit gets to 1.0. In fact, I want to experiment with pushing the JSON all the way to the screen reader (by modifying an open-source screen reader like NVDA), instead of using the current pull-based platform accessibility APIs (which are the reason for the event loop issue I opened yesterday). In my wildest dreams, the JSON format could even be pushed across the network, for remote applications.

[FirePowi]

@mwcampbell Hi hi, we're 2/3 months after and I’d really like my Ren’Pys projects to be accessible to screen readers, is there any news?

[DataTriny]

Hello, AccessKit dev here.

We now have complete C bindings and experimental Python bindings that we hope to ship soon.

Our Windows and macOS native libraries are well under 1Mb in size, our *nix ones however weigh around 5Mb and there is currently not much we can do about it unfortunately.

@renpytom Would you consider this acceptable?

[renpytom]

That seems fine by me, especially if I can dynamically link on Linux. Do you have the information about the bindings?

[DataTriny]

• The latest release of the C bindings is here
• Progress on the Python bindings can be tracked on this PR and a test release I generated a while back can be found here.

There is an example of how to integrate with SDL2 inside bindings/c/examples/sdl as well as an example demonstrating how to integrate with pygame inside bindings/python/examples/pygame (on the PR mentioned above).

[mwcampbell]

Let me provide some clarifications on binary sizes. I'll be breaking down the sizes of the shared (dynamic) libraries on all supported platforms, for the x86-64 architecture. The sizes for other processor architectures are in the same vicinity, but of course they vary somewhat. The static libraries include full debug info on all platforms.

• Windows: the DLL is 270K, and the PDB file (with the debug info) is 3.5M.
• macOS: I just discovered that our packaged build of the macOS dynamic library is a bit broken. We're not actually shipping full debug info, but we're not stripping the library either. The current library, not stripped but also missing full debug info, is 779K. What we really ought to do is package the debug info into a dSYM bundle, then strip the actual library. If I do that manually, the stripped library is 449K, and the dSYM bundle is 4.1M.
• Linux: The shared library itself includes full debug info, because I don't know of a distro-independent way to split out the debug info in such a way that debuggers will automatically use it, as with the Windows PDB file and the macOS dSYM bundle alongside the stripped library. But downstream users can split the debug info and/or strip the library, as distro packages usually do. So, the full library with debug info is 20M, while the stripped library is 1.6M.

[mwcampbell]

The AccessKit Python package is now published on PyPI. Of course, the C API is also an option if, for some reason, adding this Rust-based Python extension module to the Ren'Py build is difficult.

[renpytom]

Is there documentation for the C library? I'd prefer not to have to deal with Rust as a direct dependency for the Ren'Py build process. I see there's a header file, and a couple of examples, but I don't really have a great idea of where integration would even begin.