Repository navigation
[Bug]: Screen readers hear nothing between "Working" and "Done" during a turn #26956
nibarAhmed
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Before submitting
Area
apps/desktop
Steps to reproduce
Expected behavior
A screen reader user can choose to hear what a sighted user sees in the timeline during a turn: the messages the agent writes while it works, its final message, and, if they want it, what it's doing right now. Whatever they turn off stays silent.
Actual behavior
Since #23752, you hear "Working" when the turn starts and "Done" (or "Failed", "Stopped", "Needs approval", "Needs your input") when it changes state. Nothing in between is announced, and neither is the final message. To find out what the agent did or said, you have to leave the composer and jump back through the message headings.
An earlier version of #23752 also announced activity labels. They were out of scope for that PR, which was narrowed to turn status only, and they were left for a follow-up with a setting (suggestion, reply). They need that setting because they spoke a new label about every 2 seconds, which is too chatty to have on for everyone. In Chromium, NVDA's only way to silence them is NVDA+5 ("Report dynamic content changes"), and that mutes every live region at once: turn status, the sidebar, and error alerts. So these announcements need their own switches in T3 Code (item 5 of #23656).
Impact
Minor bug or occasional failure
Version or commit
main @ 125b619
Environment
Windows 11, T3 Code desktop app, NVDA
Logs or stack traces
Screenshots, recordings, or supporting files
No response
Workaround
Leave the composer and use H to jump back through the message headings in the timeline.
Proposed fix (first outline)
I'm a screen reader user, and I'd be happy to work on this. Below is a proposal, not a finished design. Getting the balance between informative and noisy right will take testing with real screen readers, so the details will likely change. Other opinions are welcome, especially from other screen reader users, since preferences differ.
Add a screen reader announcements setting in Settings, with one toggle for each kind of announcement:
A message counts as mid-turn when the agent runs a tool after it, and as the final message when the turn ends after it. With both message toggles on, each message is heard exactly once.
Turn status from fix(web): screen readers hear when the open thread starts, needs you or finishes #23752 stays on all the time and is not affected by these toggles.
Labels should make sense when heard on their own. Today a web search can be read as just its query, for example "Theo Browne t3dotgg opinions TypeScript React Next.js AI coding". It should say what the step is: "Searching for Theo Browne t3dotgg opinions…".
Announcements are spoken in order of importance:
Nothing is spoken after the turn has stopped.
Use the same live region and pure-function setup as fix(web): screen readers hear when the open thread starts, needs you or finishes #23752 (
threadActivityAnnouncement.ts), so the rules can be unit tested and an announcement never re-renders the timeline.If this approach doesn't fit how you'd like it solved, I'm open to others and happy to follow your direction.
Related
#23656, #23752
All reactions