On Accessibility

[aschrijver]

Hi there 👋 I watched Rik Arends presentation at RustNL. Fabulous project (I immediately posted on Fediverse in excitement). Imho together with "local-first" this shows a paradigm beyond Web 2.0 where dynamic webapps leave the browser for what it was meant to be: serving hypermedia. And break the stranglehold of Google on the web.

BUT..

In the presentation Rik mentioned that for Accessibility most likely AI would soon do the heavy lifting for us. Please, don't let a11y fall down the road-side like that. It should be a first-class citizen of any GUI framework, imho. On the Fediverse there are many visually impaired people and I notice their huge frustration and continuous advocacy to see their needs addressed in so many softwares.

In my toot there's a link to this HN submission where Ian Hickson, long-time html spec editor (and ironically at Google Flutter) defends a vision "Towards a modern web stack" and listing ARIA as a standard to support besides WebGL and WebAssembly. See his Google Doc.

PS. Heads up to this HN thread mentioning Makepad, where commenters also express similar concerns.

[makepaddev]

Hi! I am building support for 'navigation trees' into makepad which would be the backbone to add accessibility on. Currently i use it for keyboard nav of the ironfish UI. However the idea is that it should be able to spawn a shadow-dom-tree for ARIA or NSView tree on macos for traversal using accessibility tools. I do still think that AI based image decomposition/structure extract tooling is going to sweep the stage soon enough tho.

[makepaddev]

Hey,

Rustybuzz support was just merged in, so the unicode support is already taking its first steps. Albeit having some performance issues. We are working on ticking the boxes. I also do have an accesibilty backbone in the system we just have to work out the details on all the platforms.

This is a long haul project, if it doesnt fit your requirements right now then you shouldn't use it, which seems very likely at this point in time. Because it is just barely starting to fit even our own requirements to build an IDE and applications with. We are building out in the open, so you can see everything we are doing. However that does not mean that we pretend that our stack is complete yet.

And yea i think AI will sweep accessibility soon enough, why? Have you seen how fast AI is moving in understanding our world already? Image segmentation, image to text, understanding intent, etc. I'm building the infra to integrate with the system screen readers (like on apple and web) but otherwise i don't really believe in the standardisation and categorisation insanity that is ARIA on web for instance. Have you seen the category lists on that thing, nobody uses a list that long. 5-10 types tops. And above that it is even the wrong API model for a UI stack, i need a query model not a document generator.
Anyhow we're building accessibility navigation the old fashioned way, but i also have limited resources.

Btw im not desinterested, i fixed the readme link immediately. However we are not yet focussed on scaling use of the application framework. It is like launching visual basic without an IDE/editor. Once we are there we'll ramp up communication/documentation efforts to fit. But from a central onboarding application.

[makepaddev]

You seem somehow convinced i wont do accessiblity the old way. We are going to do that. I just think the model is aging and will be replaced soon atleast for people on fast compute.

[makepaddev]

What kind of AI do you think is going to do this?

• You can image-to-text things in realtime today on an M1 with apples image to text APIs and get a structured output you can screenread. And thats today.

What are the input(s) and output of this hypothetical AI?

• Image in, tree structure and text out. And it will do it more accuratrely than a div-soup website uses aria tags (which they mostly dont)

Since you are someone who talks about memory size and dependencies a lot, how many GB will it take and how much accuracy and latency do you think such an AI will have? Do you expect this to be implemented magically on all platforms "soon enough"?

• It wont be on all platforms nor go fast enough on slow compute, hence we'll do accessibilty the old way.

Accessibility occurs at a pretty low level and has very complex interactions with different components. Further it is a many-to-many problem, which is why it requires app or content authors to do the categorization and the GUI platforms to surface the categories. What indication is there in the literature that AI is going to "sweep the stage" of UI accessibility?

• I can screenread images in a structured form already today on apple. I see image segmentation algos, people are making networks that can 'visually describe' what they see, etc.

[makepaddev]

My guess is apple will be the first to offer image based accessibility built into the OS instead of looking at DOM trees' often very messy structure representation of what you see on a website. And this accessibility might even run on something like the apple AR headset to be 'generally' applicable to the entire world and not just a screen. Reading a microwave or roadsign as easily as a UI.

[makepaddev]

The primary problem of accessibilty as i understand it is having a tree structure of a UI available to 'traverse' via screenreader and then doing actions on a selected node. click/focus or enter text, roughly.

[makepaddev]

However the rendering-structure of an application vs how you would efficiently structure accesibility are also not the same.
I see a future of 'generic' accessibility through vision, and specific accessibility built into applications via a more direct model.

[makepaddev]

Anyway i think you are right btw, i too easily flipped the accessibility question to AI. We'll put more work into integrating current accessibility tech into the system.

[makepaddev]

The primary problem of accessibilty as i understand it is having a tree structure of a UI available to 'traverse' via screenreader and then doing actions on a selected node. click/focus or enter text, roughly.

Actually, accessibility covers input as well as output. You've mentioned only the output side. On the input side accessibility includes all the input methods, i.e. keyboard, mouse, touch, gestures (touch and otherwise), switch control, predictive text, customizable inputs (key repeats, macros), assistive technologies such as eye tracking, and on and on.

Modern OSes handle all of these, and lack of support for any within an app running in on an OS that otherwise supports it, is considered noncompliant and I believe the app can be found legally liable.
Afaik only if that app is used for particular domains, like government-services UI's and in that case the vendor will be required by the development contract with said government to provide accessibility.

But even on the output side, there's a reason why automatic screenreaders are considered inferior to explicit labels - there is just a lot of ambiguity. AI will certainly be able to infer the structure of an image with ever growing accuracy - but at what cost? This is where an engineer should immediately recognize the latency, memory, disk, GPU, and CPU, and battery costs and shudder. That's why your casual remarks without any qualification have seemed so off to me. I've been impressed by the effort you've taken to keep makepad lean - even for building - and so this sidebar seems really out of character and TBH comes across in a way that negates what I believe to be your norm.

Btw the latency on apple image recognition is extremely low. Anyhow you don't need to convince me on the important of accessibility tech for a UI framework. And i'm sorry if my response flipped everyone into thinking 'i dont care'. I do, just even from a humane point of view i do. And i'll try to allocate more resources to it soon. I'm just trying to juggle my priorities so we can get this insane project to some kind of value as soon as we can.

[coolbluewater]

Btw the latency on apple image recognition is extremely low.

See as an engineer my mind goes: "low" under what conditions? We're talking 10-20 of MB per image at a 60-120fps. So that's how much other memory needs to be possibly evicted from the caches. You've also not mentioned what the power draw would be. This cherry picking of traits really doesn't do much for me, especially when we're talking about a toolkit that has to be used under various architectures (even just on apple systems - take the watch for example.) Really lost me on this one my friend. Guess I'm quite influenced by the John Carmack school of analytical software engineering.

Agree on the rest of your message. Good luck. BTW is there a guide for writing an app outside of the makepad repo and getting it to compile and run for webassembly? I haven't tried yet but am curious about any possible footguns doing this.

[coolbluewater]

Rustybuzz support was just merged in, so the unicode support is already taking its first steps. Albeit having some performance issues.

Ahahaha - every user of Harfbuzz runs into this at some point. The solution is to cache the shapes at the word level. Even Behdad, the creator of Harfbuzz, has acknowledged this. It works, and how!

[makepaddev]

Yea i just merged it in because performance really doesn't matter at this point as nobody relies on the stack yet, merge in, fix the paper cuts and move on. Caching the word shapes was indeed upnext.
Also compiling Rust for wasm isnt hard, the hard part is making the Rust-in-the-wasm do something meaningful. For UI what you need to build is effectively what we did with makepad.

[coolbluewater]

I meant anything special to write an app using makepad-widgets, but keeping the code in a separate repo. I asked because the webserver looks like it may have an opinion about where the compiled code actually lives.

[coolbluewater]

Better if I open a separate issue - don't want to pollute OP's issue.

[makepaddev]

Ah yea, yea we haven't been pushing to crates.io much because we havent really moved to external-use stage yet. Right now pulling the repo into a subdirectory of your application and setting the include paths would be the way to go.

[makepaddev]

As for WASM builds we really need to complete cargo-makepad extension for wasm. Right now its a shellscript.

[makepaddev]

So to answer your question how to build wasm 'outside of our repo' its a bit janky yea for sure.

[makepaddev]

The problem with WASM builds is that it needs huge custom commandlines, uses nightly, and often needs to compile the standard library. As you can see in the shellscript

[aschrijver]

Better if I open a separate issue - don't want to pollute OP's issue.

I wouldn't worry about the OP (me), who is both fascinated and delighted by the exchange, and seeing people with much more expertise entering possible collaborations. Thank you both 🤗