Windows Terminal: Set up UIA tree and hook it up properly

[ZoeyR]

• Your Windows build number: (Type ver at a Windows Command Prompt)
  Microsoft Windows [Version 10.0.18362.53]

• What you're doing and what's happening: (Copy & paste specific commands and their output, or include screen shots)

I am launching the Windows Terminal with the NVDA screen reader active. I then type the dir command.

• What's wrong / what should be happening instead:

NVDA does not read out the contents of the dir command window. In traditional conhost, NVDA and other screen readers are able to read out the terminal content.

[zadjii-msft]

This is correct - I don't think we have the UIA tree hooked up quite yet. I think @philnach is working on this though!

[miniksa]

Phil was working on the TSF. This is UIA. We know we need to hook this up and get it going, but we haven't got to it yet.

We're pretty sure that the XAML Islands portions of the UI should provide some level of UIA pattern implementation already.

As for the TerminalControl itself, we imagine that we can crib the implementation from conhost.exe that is currently hanging out around interactivity/win32 in the code base starting from UiaTextRange.cpp and stitch that into the rest of the UIA tree with the XAML Islands patterns to get this satisfied.

[carlos-zamora]

So, I'm changing this task to just be setting up the UIA tree on Windows Terminal and hooking it up properly. Check out the Area-Accessibility tag to see the other tasks since they all kinda feed into each other to get this exact scenario to work.

[josephsl]

Hi,

I'm one of the code contributors to NVDA screen reader project. This issue also affects Narrator as well.

If you need any assistance from NV Access and other NVDA community people, please ping us. Thanks.

CC @michaelDCurran, @jcsteh, @feerrenrut, @derekriemer, @codeofdusk

[ZoeyR]

@carlos-zamora I built the most recent branch of master and I can't seem to find any of the contents of the terminal control in the UIA tree. Should this bug be reopened?

[carlos-zamora]

@carlos-zamora I built the most recent branch of master and I can't seem to find any of the contents of the terminal control in the UIA tree. Should this bug be reopened?

I'm just going to spew information out at you, so I hope I don't come off as rude.

Just rebuilt master. The TermControl is exposed as a TextProvider. So, using Inspect.exe, I've seen the data through the TextPattern Explorer.

I've been able to verify that you can access the text through Narrator too. The navigation is definitely wonky though.

Here's a copy of the UIA tree through Inspect:

Hopefully, what you're seeing is an issue with navigation. Let me know if that's not helping though.

[ZoeyR]

Ah, if I navigate into line mode then I can get some of the text out of it, and this may be my fault, but when I execute the dir command and then try to line navigate through the output I can't seem to actually read the results.

Also, is there a plan to get live text reading working again? That was what threw me off originally.

[carlos-zamora]

Ah, the live text reading is going to get hooked up after the navigation gets fixed up a bit. At least right now, the text from the buffer is accessible. But I need to hook up an event that handles new text appearing and directly passes that to the screen reader. This is definitely worth creating a new issue, for tracking. Mind creating one for me?

[ghost]

🎉This issue was addressed in #1691, which has now been successfully released as Windows Terminal Preview v0.3.2142.0.:tada:

Handy links:

• Release Notes
• Store Download

[carlos-zamora]

Hi,

I'm one of the code contributors to NVDA screen reader project. This issue also affects Narrator as well.

If you need any assistance from NV Access and other NVDA community people, please ping us. Thanks.

CC @michaelDCurran, @jcsteh, @feerrenrut, @derekriemer, @codeofdusk

Just wanted to bring issue #2447 to your attention (and anybody on this thread). I'm working on getting Windows Terminal to fire UIA Events this month. Some of the PRs may be relevant there too. Let me know if you have any feedback or questions 😊