Use braille display keys as modifier keys

[LeonarddeR]

The feature I propose is best described with the following example.

There are several braille displays with function keys assigned to specific keyboard modifier keys, one of which is the Hims Braille Edge. This display has 8 function keys, from left to right meant to be assigned to Esc, Tab, Ctrl, Alt, Shift, Insert, Windows, and the application key. This display also has two circular groups of four buttons, meant to be arrow keys.

The mentioned braille display function keys are especially handy if you combine them with other function keys or arrow keys. So when you press advance4 (ctrl) + advance1 (tab), you should be able to simulate the ctrl+tab gesture on the keyboard.

IN the current situation, every single combination has to be manually defined in NVDA. Thus, if you assign br(hims):advance3 to kb:control and br(hims):leftSideLeftArrow to kb:leftArrow, the combination advance3+leftSideLeftArrow isn't automatically assigned to kb:control+leftArrow, unless you specifically do this, either manually or in the driver itself. Especially the in-driver assignments consume a lot of code and are error-prone, since both the keyboard key and braille key assignments reside in strings and there is currently no meganism to test them other than with the braille display in front of you.

This leads to the idea of adding a way to use braille display keys assigned to keyboard modifier keys as modifier keys themselves. In the case of Hims, if advance3 is assigned to ctrl and advance4 to alt, br(hims):advance3+advance4+leftSideLeftArrow should sum up to kb:ctrl+alt+leftArrow without the need of manually assigning this combination. Using these modifier keys in other composite key strokes (containing arrow key emulators or even braille input combinations) should be possible without manual assignments. This would open great possibilities, pressing key strokes like ctrl+c, ctrl+v, ctrl+space without keyboard intervention and manual assignments.

I understand that this proposal can conflict with current key assignments in certain display drivers, which is a point to discuss. I see the following options:

1. Always give assigned combinations precedense. When advance3=ctrl and leftSideLeftArrow=leftArrow, it should be possible to assign advance3+leftSideLeftArrow to something entirely different, like moving the review cursor left
2. Ignore those assigned combinations. If advance3+leftSideLeftArrow is assigned for moving the review cursor left, that assignment should be ignored and the combination should consistently simulate ctrl+leftArrow

In other words, point 1 stands for freedom, point 2 stands for concistency. Since I see reasons why both options could have preference, I would vote for making this an option in the braille settings dialog

[LeonarddeR]

Some research revealed the following:

1. Implementing combinations like advance3+dot1+dot2+dot3 = ctrl+l will be quite hard at first sight, as it will require the concatenation of two types of gestures, namely a brailleInput.BrailleInputGesture and a braille.BrailleDisplayGesture
2. The other type of composite commands wouldn't be that difficult. We can:
   • Implement a system in inputCore GlobalGestureMap to dynamically create new mappings for possible combinations. So, if advance3=control and leftSiteDownArrow=downArrow, dynamically create advance3+leftSideDownArrow=control+downArrow. This would have the advantage that all these dynamically created keys can be seen and removed in the input gestures dialog.

• Patch scriptHandler._getObjScript to see whether a composisite keyboard emulation script can be returned if scriptName is None. However, inputCore._AllGestureMappingsRetriever won't know them this way so it is not possible to review them in input gestures dialog, for example. This one is probably more ugly.

[LeonarddeR]

@jcsteh: Could you give your opinion about a possible implementation of this?

[jcsteh]

@leonardder commented on 27 Jul 2016, 20:41 GMT+10:

I understand that this proposal can conflict with current key assignments in certain display drivers, which is a point to discuss. ...
Since I see reasons why both options could have preference, I would vote for making this an option in the braille settings dialog

I'd be more inclined to make braille display drivers choose, rather than confusing users. I think in most cases, it'd make sense for drivers to rebind commands which conflict. Any ideas what other screen readers do here?

@leonardder commented on 28 Jul 2016, 05:16 GMT+10:

1. Implementing combinations like advance3+dot1+dot2+dot3 = ctrl+l will be quite hard at first sight, as it will require the concatenation of two types of gestures, namely a brailleInput.BrailleInputGesture and a braille.BrailleDisplayGesture

In most (maybe all?) drivers, these are already combined; BrailleDisplayGesture has special support for BrailleInputGesture when they're both inherited. (If either self.dots or self.space is set, BrailleInputGesture's identifiers and display name get used.) We could add attributes for modifiers in BrailleInputGesture and teach it how to use them when sending input.

2. The other type of composite commands wouldn't be that difficult. We can:
   • Implement a system in inputCore GlobalGestureMap to dynamically create new mappings for possible combinations. So, if advance3=control and leftSiteDownArrow=downArrow, dynamically create advance3+leftSideDownArrow=control+downArrow. This would have the advantage that all these dynamically created keys can be seen and removed in the input gestures dialog.

I'm not a massive fan of this, since it means processing the entire gesture map with very specific code for keyboard modifiers. inputCore is really meant to be as input agnostic as it can. I'm also not sure it makes sense to show all these dynamically created keys. They're either modifiers or they're not. If they're modifiers, selectively rebinding them seems inconsistent/confusing.

Again, any idea what other screen readers do here?

[LeonarddeR]

it'd make sense for drivers to rebind commands which conflict.

I agree

Any ideas what other screen readers do here?

I belief that this functionality would be unique to NVDA. As far as I know, JAWS and SuperNova have no such thing as a modifier key on a braille display, so they require display vendors to bind all combinations manually. I'm not 100% sure, though.

We could add attributes for modifiers in BrailleInputGesture and teach it how to use them when sending input.

That's a good idea, I'll need to keep that one in mind whenever I'd start working on this.

I'm not a massive fan of this, since it means processing the entire gesture map with very specific code for keyboard modifiers. inputCore is really meant to be as input agnostic as it can. I'm also not sure it makes sense to show all these dynamically created keys. They're either modifiers or they're not. If they're modifiers, selectively rebinding them seems inconsistent/confusing.

Does this mean you prefer the scriptHandler._getObjScript approach, or do you have a third alternative in mind?

[jcsteh]

@leonardder commented on 21 Jun 2017, 20:56 GMT+12:

Does this mean you prefer the scriptHandler._getObjScript approach, or do you have a third alternative in mind?

I guess the _getObjScript approach will work. I agree it's a bit ugly, but I don't have an alternative.

[LeonarddeR]

I started working on this, and it seems there is a nicer approach than _getObjScript, namely overriding _get_script on BrailleDisplayGesture. This way, most of the work can be done in the braille module itself.

@jcsteh: There are some braille display drivers which set the keyNames property on their input gesture class. However, some use sets, some use lists, and it seems that, especially for the last drivers you wrote or improved, you used a list. Do you have a particular reason why you went for a list and not for a set here? I'd say using a list can result in ids like routing+routing+routing for the baum driver, and that makes no sense as there is only one routing index.

The reason why I'm asking, I'd like to have the braille display specific gestures to have a keyNames property, which defaults to set(self.id.split("+")) in my current code base but can also be set by the gesture code itself, like id. However, that requires me to change some of the current driver's gestures to use a set for keyNames. I'd like to know whether you have problems with that approach.

[jcsteh]

We previously used sets because we were using set order to normalize identifiers. However, since that order isn't reliable, we no longer do this and instead sort. So, building a set is pointless because we're just going to go and sort it anyway. One other advantage of not using a set is that the id is also used as the display name If you use a set instead, the order reported to the user is arbitrary. In contrast, if you use a list, the display has control over this. This might just be based on the order of the key identifiers, but this normally still makes more sense than set order. Is there any particular reason keyNames needs to be a set? Note that the cost of building a small set can actually outweigh the cost of linear searching a list in some cases. Also, if you really do need a set, you could consider having a cached property which converts to a set for use in specific cases.

[LeonarddeR]

@jcsteh commented on 16 aug. 2017 08:57 CEST:

Is there any particular reason keyNames needs to be a set? Note that the cost of building a small set can actually outweigh the cost of linear searching a list in some cases. Also, if you really do need a set, you could consider having a cached property which converts to a set for use in specific cases.

Sets are great for checking whether a group of keys is part of another group of keys. I assume with lists, this consumes more resources as well as more code, right? I think I will go for a list for keyNames, as I understand and support your arguments for this.

[jcsteh]

Yes, if you're testing for multiple possibilities, set intersection is probably going to be more efficient... in theory.