Single letter navigation from a braille terminal

[MichelSuch]

When in browse mode, user should be able to type a single letter command from his braille terminal.
For example, typing "h" on the terminal should move to the next title.
I have tested it on 2 braille devices (Eurobraille Esys and Baum Vario Ultra) and it does not work on both.
Maybe does it work on other devices?

[LeonarddeR]

This idea is certainly interesting. I'm planning to look into implementing braille display modifier keys (#6213), which is slightly related (e.g. for shift+letter ke combinations). It is still blocked by #2439 though. @jcsteh: I assume that, from a technical point of view, pressing dot 1235 on a braille keyboard (letter h) now sends the input to the browser itself? cc @dkager

[LeonarddeR]

I'd like to suggest a priority of p4, since although it is a nice to have, there is a workaround since people can assign desired keys in the input gestures dialog. If not, that's most certainly something we should fix as well, since people should be able to override key combinations reserved for braille input at some point. I also belief that #6213 should be addressed first.

[LeonarddeR]

@MichelSuch: Would you like to test this using next? I"m using next-14122,9b0534fc and with the pc keyboard braille input add-on, I'm able to use single letter quick navigation perfectly fine. Backwards doesn't work though, but that, as I mentioned, can be implemented in #6213

Also note that on (Eurobraille Esys, NVDA's braille input layer is not being used, so we can't do anything about that.

[jcsteh]

@leonardder commented on 20 Jun 2017, 23:44 GMT+10:

@jcsteh: I assume that, from a technical point of view, pressing dot
1235 on a braille keyboard (letter h) now sends the input to the browser
itself?

I assume you mean 125. :) It sends the Unicode character (not the key) to the system.

@leonardder commented on 21 Jun 2017, 01:14 GMT+10:

I"m using next-14122,9b0534fc and with the pc keyboard braille input add-on, I'm able to use single letter quick navigation perfectly fine.

This is interesting; I didn't expect this, so I did a bit of investigation. The reason this works is that if you have NVDA's "Handle keys from other applications" option enabled, NVDA intercepts the keys generated by the system when we send Unicode characters. This didn't work properly before, but it got fixed by 243d39f (which was actually intended to address an entirely different symptom). That's nice, but it won't play so well with modifiers, since it isn't an emulated gesture, and it also doesn't work if "Handle keys from other applications" is disabled (which doesn't make much sense).

I think the correct solution here is to detect non-editable cases in brailleInput and emulate keystrokes instead of Unicode characters.

Backwards doesn't work though, but that, as I mentioned, can be implemented in #6213

That'll help for braille displays which have dedicated modifier keys, but many don't. I think we still need a way to enter modifiers using the braille keyboard (space+dots combinations). Interestingly, that's probably significantly cleaner to implement than #6213.

[LeonarddeR]

Replying to @jcsteh:

I assume you mean 125. :)

Oops, yes

I think the correct solution here is to detect non-editable cases in brailleInput and emulate keystrokes instead of Unicode characters.

I'm not sure what you mean with non-editable cases here

That'll help for braille displays which have dedicated modifier keys, but many don't.

My idea was to allow people to set whatever key they like to a modifier, e.g. setting a scroll key to shift, or even control + shift wil automatically consider the key to be a modifier as it emulates a modifier or a group of modifiers. But that's something that needs to be discussed in #6213.

I think we still need a way to enter modifiers using the braille keyboard (space+dots combinations). Interestingly, that's probably significantly cleaner to implement than #6213.

Why the space+dots combinations specifically? That would only allow the space to be a modifier if I understand you correctly?

[jcsteh]

@leonardder commented on 21 Jun 2017, 20:50 GMT+12:

I think the correct solution here is to detect non-editable cases in brailleInput and emulate keystrokes instead of Unicode characters.

I'm not sure what you mean with non-editable cases here

I mean a control that doesn't accept characters as text input. You can use the new private NVDAObject._hasNavigableText property for part of this, but you must also treat browse mode as non-editable.

I think we still need a way to enter modifiers using the braille keyboard (space+dots combinations).

Why the space+dots combinations specifically? That would only allow the space to be a modifier if I understand you correctly?

I mean space+dots, then character. For example, to do shift+h with Narrator in Windows 10 Insiders, you might do space+dot8+dot4, release everything, then braille h.

[LeonarddeR]

@jcsteh commented on 20 jul. 2017 01:55 CEST:

I mean space+dots, then character. For example, to do shift+h with Narrator in Windows 10 Insiders, you might do space+dot8+dot4, release everything, then braille h.

This is interesting. However, what would nvda+Control+Shift+H look like, for example? Initially, the idea of #6213 popped into my mind due to the amount of function keys on Hims displays which can be used as such. In that case, ctrl+H would be just one key stroke, f3+dot1+dot2+dot5. Your proposal is slightly different. I assume I can build upon the new brailleInput system which allows to delay translation to occur after multiple key presses?

[jcsteh]

@leonardder commented on 1 Aug 2017, 05:06 GMT+10:

This is interesting. However, what would nvda+Control+Shift+H look like, for example?

With Narrator, you'd do the following (changing NVDA to caps lock because that's what Narrator uses): space+dot8+dot6 (hold caps lock), space+dot8+dot3 (hold control), space+dot8+dot4 (hold shift), then finally braille h. See this article about the Insider build which introduced this for further details. Note that with JAWS braille, you can (and I think must) combine modifiers; in this case, space+dot8+dot5+dot3+dot7, then braille h. The JAWS way of doing it requires less keystrokes.

Initially, the idea of #6213 popped into my mind due to the amount of function keys on Hims displays which can be used as such.

For sure. It certainly makes sense to support those where available, but not all displays have such keys.

Your proposal is slightly different.

My proposal is intended to allow modifiers to be entered purely using the braille keyboard and I'd argue it's more broadly applicable. However, the two proposals aren't at all mututally exclusive, though they'll require somewhat separate implementations.

I assume I can build upon the new brailleInput system which allows to delay translation to occur after multiple key presses?

I don't think this modifier stuff can really benefit from existing code, since that code is all about maintaining text state in a buffer. However, you could track held modifiers in brailleInput for sure and then make sure they get pressed before the braille characters are sent and released afterwards (in BrailleInputHandler.sendChars).

One other thing to consider is that ideally, if the user is using a contracted table, we'd want to switch to an uncontracted table when there are held modifiers. We don't support the concept of a configured uncontracted table yet, though.

[LeonarddeR]

Requests for this are increasing, so I will adopt this, unless someone else wants to do it. Some rough ideas. @jcsteh, as you know the code much better than I do, your ideas are welcome.

@jcsteh commented on 21 jun. 2017 07:51 CEST:

I think the correct solution here is to detect non-editable cases in brailleInput and emulate keystrokes instead of Unicode characters.

I belief this is actually not that difficult. I propose:

1. In BrailleInputHandler._translate, instead of throwing everything into sendChars:

   • Check for editable text cases as well as the length of newText
   • If not editable text and newText is exactly one character, emulate a gesture instead of sending unicode cahracters

2. The case I'd also like to cover is contracted braille table input. I belief that quick nav should be possible with contracted tables as well. This will mostly require a user to type one letter combination and force translation with dots7-8. I belief this is actually as simple as checking the length of newText and, if more than one character, warn the user that the given braille input consists of more than one character and is therefore not supported in the currently focused control, and return false.

[LeonarddeR]

There is a prototype for this in the @BabbageCom i7306 branch, which makes use of liblouis partial trans for contracted input. This means that, even if we use an contracted table, we can treat its input as uncontracted outside text controls.