New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use braille display keys as modifier keys #6213

Closed
leonardder opened this Issue Jul 27, 2016 · 9 comments

Comments

Projects
None yet
4 participants
@leonardder
Collaborator

leonardder commented Jul 27, 2016

The feature I propose is best described with the following example.

There are several braille displays with function keys assigned to specific keyboard modifier keys, one of which is the Hims Braille Edge. This display has 8 function keys, from left to right meant to be assigned to Esc, Tab, Ctrl, Alt, Shift, Insert, Windows, and the application key. This display also has two circular groups of four buttons, meant to be arrow keys.

The mentioned braille display function keys are especially handy if you combine them with other function keys or arrow keys. So when you press advance4 (ctrl) + advance1 (tab), you should be able to simulate the ctrl+tab gesture on the keyboard.

IN the current situation, every single combination has to be manually defined in NVDA. Thus, if you assign br(hims):advance3 to kb:control and br(hims):leftSideLeftArrow to kb:leftArrow, the combination advance3+leftSideLeftArrow isn't automatically assigned to kb:control+leftArrow, unless you specifically do this, either manually or in the driver itself. Especially the in-driver assignments consume a lot of code and are error-prone, since both the keyboard key and braille key assignments reside in strings and there is currently no meganism to test them other than with the braille display in front of you.

This leads to the idea of adding a way to use braille display keys assigned to keyboard modifier keys as modifier keys themselves. In the case of Hims, if advance3 is assigned to ctrl and advance4 to alt, br(hims):advance3+advance4+leftSideLeftArrow should sum up to kb:ctrl+alt+leftArrow without the need of manually assigning this combination. Using these modifier keys in other composite key strokes (containing arrow key emulators or even braille input combinations) should be possible without manual assignments. This would open great possibilities, pressing key strokes like ctrl+c, ctrl+v, ctrl+space without keyboard intervention and manual assignments.

I understand that this proposal can conflict with current key assignments in certain display drivers, which is a point to discuss. I see the following options:

  1. Always give assigned combinations precedense. When advance3=ctrl and leftSideLeftArrow=leftArrow, it should be possible to assign advance3+leftSideLeftArrow to something entirely different, like moving the review cursor left
  2. Ignore those assigned combinations. If advance3+leftSideLeftArrow is assigned for moving the review cursor left, that assignment should be ignored and the combination should consistently simulate ctrl+leftArrow

In other words, point 1 stands for freedom, point 2 stands for concistency. Since I see reasons why both options could have preference, I would vote for making this an option in the braille settings dialog

@leonardder

This comment has been minimized.

Show comment
Hide comment
@leonardder

leonardder Jul 27, 2016

Collaborator

Some research revealed the following:

  1. Implementing combinations like advance3+dot1+dot2+dot3 = ctrl+l will be quite hard at first sight, as it will require the concatenation of two types of gestures, namely a brailleInput.BrailleInputGesture and a braille.BrailleDisplayGesture
  2. The other type of composite commands wouldn't be that difficult. We can:
    • Implement a system in inputCore GlobalGestureMap to dynamically create new mappings for possible combinations. So, if advance3=control and leftSiteDownArrow=downArrow, dynamically create advance3+leftSideDownArrow=control+downArrow. This would have the advantage that all these dynamically created keys can be seen and removed in the input gestures dialog.
  • Patch scriptHandler._getObjScript to see whether a composisite keyboard emulation script can be returned if scriptName is None. However, inputCore._AllGestureMappingsRetriever won't know them this way so it is not possible to review them in input gestures dialog, for example. This one is probably more ugly.
Collaborator

leonardder commented Jul 27, 2016

Some research revealed the following:

  1. Implementing combinations like advance3+dot1+dot2+dot3 = ctrl+l will be quite hard at first sight, as it will require the concatenation of two types of gestures, namely a brailleInput.BrailleInputGesture and a braille.BrailleDisplayGesture
  2. The other type of composite commands wouldn't be that difficult. We can:
    • Implement a system in inputCore GlobalGestureMap to dynamically create new mappings for possible combinations. So, if advance3=control and leftSiteDownArrow=downArrow, dynamically create advance3+leftSideDownArrow=control+downArrow. This would have the advantage that all these dynamically created keys can be seen and removed in the input gestures dialog.
  • Patch scriptHandler._getObjScript to see whether a composisite keyboard emulation script can be returned if scriptName is None. However, inputCore._AllGestureMappingsRetriever won't know them this way so it is not possible to review them in input gestures dialog, for example. This one is probably more ugly.

leonardder added a commit to BabbageCom/nvda that referenced this issue Jul 28, 2016

@leonardder

This comment has been minimized.

Show comment
Hide comment
@leonardder

leonardder May 27, 2017

Collaborator

@jcsteh: Could you give your opinion about a possible implementation of this?

Collaborator

leonardder commented May 27, 2017

@jcsteh: Could you give your opinion about a possible implementation of this?

@jcsteh

This comment has been minimized.

Show comment
Hide comment
@jcsteh

jcsteh Jun 21, 2017

Contributor

@leonardder commented on 27 Jul 2016, 20:41 GMT+10:

I understand that this proposal can conflict with current key assignments in certain display drivers, which is a point to discuss. ...
Since I see reasons why both options could have preference, I would vote for making this an option in the braille settings dialog

I'd be more inclined to make braille display drivers choose, rather than confusing users. I think in most cases, it'd make sense for drivers to rebind commands which conflict. Any ideas what other screen readers do here?

@leonardder commented on 28 Jul 2016, 05:16 GMT+10:

  1. Implementing combinations like advance3+dot1+dot2+dot3 = ctrl+l will be quite hard at first sight, as it will require the concatenation of two types of gestures, namely a brailleInput.BrailleInputGesture and a braille.BrailleDisplayGesture

In most (maybe all?) drivers, these are already combined; BrailleDisplayGesture has special support for BrailleInputGesture when they're both inherited. (If either self.dots or self.space is set, BrailleInputGesture's identifiers and display name get used.) We could add attributes for modifiers in BrailleInputGesture and teach it how to use them when sending input.

  1. The other type of composite commands wouldn't be that difficult. We can:
    • Implement a system in inputCore GlobalGestureMap to dynamically create new mappings for possible combinations. So, if advance3=control and leftSiteDownArrow=downArrow, dynamically create advance3+leftSideDownArrow=control+downArrow. This would have the advantage that all these dynamically created keys can be seen and removed in the input gestures dialog.

I'm not a massive fan of this, since it means processing the entire gesture map with very specific code for keyboard modifiers. inputCore is really meant to be as input agnostic as it can. I'm also not sure it makes sense to show all these dynamically created keys. They're either modifiers or they're not. If they're modifiers, selectively rebinding them seems inconsistent/confusing.

Again, any idea what other screen readers do here?

Contributor

jcsteh commented Jun 21, 2017

@leonardder commented on 27 Jul 2016, 20:41 GMT+10:

I understand that this proposal can conflict with current key assignments in certain display drivers, which is a point to discuss. ...
Since I see reasons why both options could have preference, I would vote for making this an option in the braille settings dialog

I'd be more inclined to make braille display drivers choose, rather than confusing users. I think in most cases, it'd make sense for drivers to rebind commands which conflict. Any ideas what other screen readers do here?

@leonardder commented on 28 Jul 2016, 05:16 GMT+10:

  1. Implementing combinations like advance3+dot1+dot2+dot3 = ctrl+l will be quite hard at first sight, as it will require the concatenation of two types of gestures, namely a brailleInput.BrailleInputGesture and a braille.BrailleDisplayGesture

In most (maybe all?) drivers, these are already combined; BrailleDisplayGesture has special support for BrailleInputGesture when they're both inherited. (If either self.dots or self.space is set, BrailleInputGesture's identifiers and display name get used.) We could add attributes for modifiers in BrailleInputGesture and teach it how to use them when sending input.

  1. The other type of composite commands wouldn't be that difficult. We can:
    • Implement a system in inputCore GlobalGestureMap to dynamically create new mappings for possible combinations. So, if advance3=control and leftSiteDownArrow=downArrow, dynamically create advance3+leftSideDownArrow=control+downArrow. This would have the advantage that all these dynamically created keys can be seen and removed in the input gestures dialog.

I'm not a massive fan of this, since it means processing the entire gesture map with very specific code for keyboard modifiers. inputCore is really meant to be as input agnostic as it can. I'm also not sure it makes sense to show all these dynamically created keys. They're either modifiers or they're not. If they're modifiers, selectively rebinding them seems inconsistent/confusing.

Again, any idea what other screen readers do here?

@leonardder

This comment has been minimized.

Show comment
Hide comment
@leonardder

leonardder Jun 21, 2017

Collaborator

it'd make sense for drivers to rebind commands which conflict.

I agree

Any ideas what other screen readers do here?

I belief that this functionality would be unique to NVDA. As far as I know, JAWS and SuperNova have no such thing as a modifier key on a braille display, so they require display vendors to bind all combinations manually. I'm not 100% sure, though.

We could add attributes for modifiers in BrailleInputGesture and teach it how to use them when sending input.

That's a good idea, I'll need to keep that one in mind whenever I'd start working on this.

I'm not a massive fan of this, since it means processing the entire gesture map with very specific code for keyboard modifiers. inputCore is really meant to be as input agnostic as it can. I'm also not sure it makes sense to show all these dynamically created keys. They're either modifiers or they're not. If they're modifiers, selectively rebinding them seems inconsistent/confusing.

Does this mean you prefer the scriptHandler._getObjScript approach, or do you have a third alternative in mind?

Collaborator

leonardder commented Jun 21, 2017

it'd make sense for drivers to rebind commands which conflict.

I agree

Any ideas what other screen readers do here?

I belief that this functionality would be unique to NVDA. As far as I know, JAWS and SuperNova have no such thing as a modifier key on a braille display, so they require display vendors to bind all combinations manually. I'm not 100% sure, though.

We could add attributes for modifiers in BrailleInputGesture and teach it how to use them when sending input.

That's a good idea, I'll need to keep that one in mind whenever I'd start working on this.

I'm not a massive fan of this, since it means processing the entire gesture map with very specific code for keyboard modifiers. inputCore is really meant to be as input agnostic as it can. I'm also not sure it makes sense to show all these dynamically created keys. They're either modifiers or they're not. If they're modifiers, selectively rebinding them seems inconsistent/confusing.

Does this mean you prefer the scriptHandler._getObjScript approach, or do you have a third alternative in mind?

@jcsteh

This comment has been minimized.

Show comment
Hide comment
@jcsteh

jcsteh Jul 19, 2017

Contributor

@leonardder commented on 21 Jun 2017, 20:56 GMT+12:

Does this mean you prefer the scriptHandler._getObjScript approach, or do you have a third alternative in mind?

I guess the _getObjScript approach will work. I agree it's a bit ugly, but I don't have an alternative.

Contributor

jcsteh commented Jul 19, 2017

@leonardder commented on 21 Jun 2017, 20:56 GMT+12:

Does this mean you prefer the scriptHandler._getObjScript approach, or do you have a third alternative in mind?

I guess the _getObjScript approach will work. I agree it's a bit ugly, but I don't have an alternative.

@leonardder

This comment has been minimized.

Show comment
Hide comment
@leonardder

leonardder Aug 16, 2017

Collaborator

I started working on this, and it seems there is a nicer approach than _getObjScript, namely overriding _get_script on BrailleDisplayGesture. This way, most of the work can be done in the braille module itself.

@jcsteh: There are some braille display drivers which set the keyNames property on their input gesture class. However, some use sets, some use lists, and it seems that, especially for the last drivers you wrote or improved, you used a list. Do you have a particular reason why you went for a list and not for a set here? I'd say using a list can result in ids like routing+routing+routing for the baum driver, and that makes no sense as there is only one routing index.

The reason why I'm asking, I'd like to have the braille display specific gestures to have a keyNames property, which defaults to set(self.id.split("+")) in my current code base but can also be set by the gesture code itself, like id. However, that requires me to change some of the current driver's gestures to use a set for keyNames. I'd like to know whether you have problems with that approach.

Collaborator

leonardder commented Aug 16, 2017

I started working on this, and it seems there is a nicer approach than _getObjScript, namely overriding _get_script on BrailleDisplayGesture. This way, most of the work can be done in the braille module itself.

@jcsteh: There are some braille display drivers which set the keyNames property on their input gesture class. However, some use sets, some use lists, and it seems that, especially for the last drivers you wrote or improved, you used a list. Do you have a particular reason why you went for a list and not for a set here? I'd say using a list can result in ids like routing+routing+routing for the baum driver, and that makes no sense as there is only one routing index.

The reason why I'm asking, I'd like to have the braille display specific gestures to have a keyNames property, which defaults to set(self.id.split("+")) in my current code base but can also be set by the gesture code itself, like id. However, that requires me to change some of the current driver's gestures to use a set for keyNames. I'd like to know whether you have problems with that approach.

@jcsteh

This comment has been minimized.

Show comment
Hide comment
@jcsteh

jcsteh Aug 16, 2017

Contributor
Contributor

jcsteh commented Aug 16, 2017

@leonardder

This comment has been minimized.

Show comment
Hide comment
@leonardder

leonardder Aug 16, 2017

Collaborator

@jcsteh commented on 16 aug. 2017 08:57 CEST:

Is there any particular reason keyNames needs to be a set? Note that the cost of building a small set can actually outweigh the cost of linear searching a list in some cases. Also, if you really do need a set, you could consider having a cached property which converts to a set for use in specific cases.

Sets are great for checking whether a group of keys is part of another group of keys. I assume with lists, this consumes more resources as well as more code, right? I think I will go for a list for keyNames, as I understand and support your arguments for this.

Collaborator

leonardder commented Aug 16, 2017

@jcsteh commented on 16 aug. 2017 08:57 CEST:

Is there any particular reason keyNames needs to be a set? Note that the cost of building a small set can actually outweigh the cost of linear searching a list in some cases. Also, if you really do need a set, you could consider having a cached property which converts to a set for use in specific cases.

Sets are great for checking whether a group of keys is part of another group of keys. I assume with lists, this consumes more resources as well as more code, right? I think I will go for a list for keyNames, as I understand and support your arguments for this.

@jcsteh

This comment has been minimized.

Show comment
Hide comment
@jcsteh

jcsteh Aug 16, 2017

Contributor
Contributor

jcsteh commented Aug 16, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment