Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emulate mouse on touch screens #3886

Closed
nvaccessAuto opened this issue Feb 14, 2014 · 28 comments · Fixed by #10065
Closed

Emulate mouse on touch screens #3886

nvaccessAuto opened this issue Feb 14, 2014 · 28 comments · Fixed by #10065

Comments

@nvaccessAuto
Copy link

@nvaccessAuto nvaccessAuto commented Feb 14, 2014

Reported by k_kolev1985 on 2014-02-14 13:53
At present, on a touch screen, NVDA can interact with things on the screen only via object navigation. If an object is well designed, it is OK. But if not - problems arise. For example, if an object has no default action assigned, there is no way (at least as far as I know) to interact with it, only via the touch screen - NVDA will say "no action" and nothing will happen. For other cases in witch object exploring fails when working only with a touch screen, see tickets #3779 and #3780. To read text on a touch screen with gestures, there is the so called "text mode", To switch between those 2 modes (text and objects), the user must tap with 3 fingers on the touch screen.

So, to improve the touch screen experience with NVDA, I suggest that a third mode of screen exploration is added to NVDA. The mode will emulate the mouse cursor. When this new mode is activated and the user explores the screen with his finger, NVDA will behave as if the mouse cursor is following the user's finger and NVDA will read things on the screen as if the user is moving the mouse cursor on the screen with "mouse tracking" enabled. Gestures for performing various actions with the mouse buttons must be added, but I think that at least some of them (eg. left single click, right single click, mouse buttons lock) are already available and only gestures for those actions must be assigned. Also, gestures for left double click, middle button single click and wheel scrolling must also be added.

I don't know however if my suggestion of mouse emulation is even possible to implement, but I wont know until I ask - don't I (?) :).

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented Feb 21, 2014

Comment 1 by jteh on 2014-02-21 06:08
I think touch has its own unique possibilities and I'd be very reluctant to tie it to the concept of mouse movement. That said, NVDA's touch support certainly does need some further work.

Regarding actions, the best thing is probably to make the activate command click if there is no action. We already do this in some cases, but not all.

I'm not sure I follow the issue with text review. The concept of review modes is subject to change, but even as it is now, I don't follow why mouse emulation would be a particularly good improvement. In most cases, you should already be able to read by line by simply moving your finger. Do you have mouse reading set to word? That's the only reason I can think of that mouse emulation would be an improvement for you.

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented Feb 21, 2014

Comment 2 by k_kolev1985 on 2014-02-21 08:09
Yes, if the NVDA touch support gets improved, the reasons to tie it to the mouse movement are less important.

Yes, the idea to simulate a click where there is no action seams good, specially if it will help for easier invoking of context menus and other now hard to do stuff on the touch screen only. BTW: I've noticed that in some cases, Narrator knows what to do with some controls (specially toggle-buttons/check-boxes), where NVDA does not - it says either that there is no action, or nothing at all, even when working with the keyboard. A good example for this is the option (check-box/toggle-button) for turning on/off the automatic updates of apps in the Windows 8 Store. With NVDA, it cannot be activated neither via the space-bar, nor via object navigation or the touch screen - when executing the command, NVDA reports that the command was executed, even though it actually it was not. It can only be activated via a mouse click. If NVDA is not running, or if we pass-through the space-bar command, the state of the control will be changed with no problems at all. But I guess this issues with missing or misbehaving actions for controls should be reported in another ticket (?).

Actually, reading by word via touch exploration is not what I'm concerned about. The problem is that in some cases, reading of whole text blocks in certain places in Windows itself fails with touch explorations, while reading them with mouse tracking works as it should.

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented May 25, 2014

Comment 3 by nvdakor on 2014-05-25 08:42
Hi,
I'd rather vote for new command set that can be invoked regardless of touch modes. The idea is as follows:

  • Once an action or mouse click is about to be performed on a touchscreen, try fetching the screen coordinates under the touch point (perhaps using screen/touch explorer). Then NVDA will tell Windows to perform the desired action such as tap, tap and hold (right mouse click) and so on.
  • As suggested by someone, perform clicks if no action can be done on an object. This requires that globalCommands.reviewActivate be modified based on the following implementation strategies: if this will be used on the keyboard, simply doing a click when there is no action for the object would be suffice; however, if this is exclusive to touch, then we need to detect the source of the gesture and either do a click or say that there is no action (for the keyboard). I vote for the second approach.
    Possible gesture assignments:
  • Tripple tap: left-mouse click (equivalent: double tap/click).
  • Tap and hold: right mouse click (and tap and hold actually works, just tested by creating a global plugin that plays a tone when a user performs tap and hold gesture).
    @mick: thoughts? I know you used to have (or have) a touch device, so thought you would know more about technical details.
    Thanks.

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented May 25, 2014

Comment 4 by nvdakor on 2014-05-25 08:44
Hi,
As a follow-up, a possible flaw with my previous idea is what if we cannot locate the point where the finger touched the screen. Then the workaround (for now) would be to route the mouse to the navigator object then do mouse clicks. This isn't ideal for both blind and sighted users.
Thanks.

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented May 25, 2014

Comment 5 by Joseph Lee <joseph.lee22590@... on 2014-05-25 13:58
In [9472caa]:

Global commands: added a gesture to perform right click on touch screens. re #3886.

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented May 25, 2014

Comment 6 by nvdakor on 2014-05-25 14:00
Hi,
The new script is named touch_rightClick in global commands and is assigned to tap and hold on touchscreens. The function body is a mixture of scripts to move the mouse and to perform right clicks. If this works, then left mouse click might be similar, or we might adopt Jamie's suggestion. Thanks.

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented Aug 29, 2014

Comment 8 by jteh on 2014-08-29 07:47
Thanks Joseph. Sorry for taking so long to get to this. This implementation for right click looks fine to me for the most part.

We should probably check that the object doesn't have the offscreen or invisible states and the coordinates are positive before clicking.

I think the description should be "Clicks the right mouse button at the current touch position. This is generally used to activate a context menu."

I'd be happy for review_activate to be modified to click the object if there is no action, though we should probably log a debugWarning when this occurs. I think this is just as useful for keyboard users, particularly in ribbons. See above re checking for offscreen.

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented Aug 29, 2014

Comment 9 by nvdakor on 2014-08-29 08:12
Hi,
I'll modify the script doc then (others, do you have any suggestions?).
Can you give me an example object (ribbons, web element, etc.) where invisible state and offscreen is logged via dev info? I'll take a look at it and modify the routine accordingly.
For the user guide, I'll add the touchscreen command to right mouse click entry for key commands and will modify the command's description (if that's needed; I'm sure it is needed).
Thanks.

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented Aug 29, 2014

Comment 10 by jteh (in reply to comment 9) on 2014-08-29 09:35
Replying to nvdakor:

Can you give me an example object (ribbons, web element, etc.) where invisible state and offscreen is logged via dev info?

Skip links on websites, but usually, you have to make sure they aren't focused. Otherwise, they become visible. Try http://www.webaim.org/

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented Aug 29, 2014

Comment 11 by Joseph Lee <joseph.lee22590@... on 2014-08-29 10:26
In [59ddbae]:

Global Commands: touch right click - ignore invisible or offscreen objects, as well as if any of the object coordinates is negative. re #3886.
In some cases, objects may appear invisible until focus moves to it. This is noticeable on www.webaim.org's skip to content anchor.
Also don't bother clicking when the minimum of the coordinates is negative.
Also modified script doc for touch_rightClick (all suggested by jamie@nvaccess.org).

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented Sep 8, 2014

Comment 13 by jteh on 2014-09-08 04:29
Looks good. Please update the User Guide. Do you plan to modify review_activate as per comment:8 or are you waiting on something?

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented Sep 8, 2014

Comment 14 by nvdakor (in reply to comment 13) on 2014-09-08 11:35
Replying to jteh:

Looks good. Please update the User Guide. Do you plan to modify review_activate as per comment:8 or are you waiting on something?

I'll update the user guide at least (add touch gesture column like some of the review commands). As for review_activate, I'll leave it up to you since you know more about how it works than I do. Thanks for your reviews.

@nvaccessAuto
Copy link
Author

@nvaccessAuto nvaccessAuto commented Sep 8, 2014

Comment 15 by nvdakor on 2014-09-08 11:53
Hi,
Modified the user guide in commit 04a9b24:

  • Changed the wording to introductory text to mouse commands table, as a touch gesture has been added.
  • Added touch column in mouse commands table to include tap and hold, as well as changed the description of right mouse click command.
    Thanks.

@surfer0627
Copy link
Contributor

@surfer0627 surfer0627 commented Jun 6, 2016

Hi,
If there is something, caused this issue does not add to the official release?

@josephsl josephsl self-assigned this Jun 6, 2016
@josephsl
Copy link
Collaborator

@josephsl josephsl commented Jun 6, 2016

Hi,
Not yet - for the most part, it is complete - will need to test if it can be incubated safely, as it's been almost two years since commiting to that branch).
Thanks.

@bhavyashah
Copy link

@bhavyashah bhavyashah commented Aug 19, 2018

@josephsl It has been two more years since #3886 (comment). :) Could you please provide a status update for your contribution?

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Aug 19, 2018

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Aug 9, 2019

Hi,

Found it - knew there was an issue about mouse support.

At least for right mouse click, I do have a solution, which will be done as a pull request this weekend (basically transferring parts of Enhanced Touch Gestures add-on to Core).

Thanks.

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Aug 9, 2019

Hi,

Removed t3886 branch from NV Access repo so I can rebase this branch from my side on Python 3 master (correcting linting and Python 3 issues as we go).

Thanks.

josephsl added a commit to josephsl/nvda that referenced this issue Aug 9, 2019
josephsl added a commit to josephsl/nvda that referenced this issue Aug 9, 2019
…jects, as well as if any of the object coordinates is negative. re nvaccess#3886.

In some cases, objects may appear invisible until focus moves to it. This is noticeable on www.webaim.org's skip to content anchor.
Also don't bother clicking when the minimum of the coordinates is negative.
Also modified script doc for touch_rightClick (all suggested by jamie@nvaccess.org).
josephsl added a commit to josephsl/nvda that referenced this issue Aug 9, 2019
…ces, reword introductory text for mouse commands table. Re nvaccess#3886
josephsl added a commit to josephsl/nvda that referenced this issue Aug 9, 2019
josephsl added a commit to josephsl/nvda that referenced this issue Aug 9, 2019
Lint fixes include spaces around operators, dictionary key/value assignment, and three noqa comments.
JulienCochuyt pushed a commit to accessolutions/nvda that referenced this issue Aug 12, 2019
JulienCochuyt pushed a commit to accessolutions/nvda that referenced this issue Aug 12, 2019
…jects, as well as if any of the object coordinates is negative. re nvaccess#3886.

In some cases, objects may appear invisible until focus moves to it. This is noticeable on www.webaim.org's skip to content anchor.
Also don't bother clicking when the minimum of the coordinates is negative.
Also modified script doc for touch_rightClick (all suggested by jamie@nvaccess.org).
josephsl added a commit to josephsl/nvda that referenced this issue Sep 8, 2019
josephsl added a commit to josephsl/nvda that referenced this issue Sep 8, 2019
…jects, as well as if any of the object coordinates is negative. re nvaccess#3886.

In some cases, objects may appear invisible until focus moves to it. This is noticeable on www.webaim.org's skip to content anchor.
Also don't bother clicking when the minimum of the coordinates is negative.
Also modified script doc for touch_rightClick (all suggested by jamie@nvaccess.org).
josephsl added a commit to josephsl/nvda that referenced this issue Sep 8, 2019
…ces, reword introductory text for mouse commands table. Re nvaccess#3886
josephsl added a commit to josephsl/nvda that referenced this issue Sep 8, 2019
josephsl added a commit to josephsl/nvda that referenced this issue Sep 8, 2019
Lint fixes include spaces around operators, dictionary key/value assignment, and three noqa comments.
michaelDCurran pushed a commit that referenced this issue Feb 16, 2020
* Global commands: added a gesture to perform right click on touch screens. re #3886.

* Oops - fix indentation.

* Global Commands: touch right click - ignore invisible or offscreen objects, as well as if any of the object coordinates is negative. re #3886.
In some cases, objects may appear invisible until focus moves to it. This is noticeable on www.webaim.org's skip to content anchor.
Also don't bother clicking when the minimum of the coordinates is negative.
Also modified script doc for touch_rightClick (all suggested by jamie@nvaccess.org).

* Remove blank line.

* User guide: document tap and hold for right mouse click on touch devices, reword introductory text for mouse commands table. Re #3886

* Touch right mouse click/Python 3: coordinates should be integers, not float. re #3886.

* Touch mouse command: lint fixes. Re #3886.

Lint fixes include spaces around operators, dictionary key/value assignment, and three noqa comments.
@nvaccessAuto nvaccessAuto added this to the 2020.1 milestone Feb 16, 2020
@Adriani90
Copy link
Collaborator

@Adriani90 Adriani90 commented Feb 17, 2020

While #10065 solves the mouse right click issue, there are also other mouse actions which still need to be considered and addressed. I am reopening this discussion.

@Adriani90 Adriani90 reopened this Feb 17, 2020
@josephsl
Copy link
Collaborator

@josephsl josephsl commented Feb 17, 2020

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Feb 17, 2020

Hi,

Let's discuss this on separate issues for now.

@ChrisMedley
Copy link

@ChrisMedley ChrisMedley commented Mar 23, 2020

We now have no right click touch gesture what so ever now in 2020.1 beta 1
Worked fine in 2019.3
Why has this been removed?

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Mar 23, 2020

@ChrisMedley
Copy link

@ChrisMedley ChrisMedley commented Mar 23, 2020

Yes I am using enhanced touch gestures 20.03 and even with all addons disabled the right click gesture still no longer working

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Mar 23, 2020

@ChrisMedley
Copy link

@ChrisMedley ChrisMedley commented Mar 23, 2020

OK, this is odd, I did as you asked and turned on NVDA help (nvda+1)
I tried the the single finger double tap and hold and got nothing, I then turned off nvda help and the single finger double tap and hold gesture worked!

Again turned on nvda help and it also reported right click this time also
And its still working after turning off nvda help

Dont ask me what happened but it's working now for whatever reason

apologies and thank you for your help

@josephsl
Copy link
Collaborator

@josephsl josephsl commented Mar 23, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants