Emulate mouse on touch screens

[nvaccessAuto]

Reported by k_kolev1985 on 2014-02-14 13:53
At present, on a touch screen, NVDA can interact with things on the screen only via object navigation. If an object is well designed, it is OK. But if not - problems arise. For example, if an object has no default action assigned, there is no way (at least as far as I know) to interact with it, only via the touch screen - NVDA will say "no action" and nothing will happen. For other cases in witch object exploring fails when working only with a touch screen, see tickets #3779 and #3780. To read text on a touch screen with gestures, there is the so called "text mode", To switch between those 2 modes (text and objects), the user must tap with 3 fingers on the touch screen.

So, to improve the touch screen experience with NVDA, I suggest that a third mode of screen exploration is added to NVDA. The mode will emulate the mouse cursor. When this new mode is activated and the user explores the screen with his finger, NVDA will behave as if the mouse cursor is following the user's finger and NVDA will read things on the screen as if the user is moving the mouse cursor on the screen with "mouse tracking" enabled. Gestures for performing various actions with the mouse buttons must be added, but I think that at least some of them (eg. left single click, right single click, mouse buttons lock) are already available and only gestures for those actions must be assigned. Also, gestures for left double click, middle button single click and wheel scrolling must also be added.

I don't know however if my suggestion of mouse emulation is even possible to implement, but I wont know until I ask - don't I (?) :).

[nvaccessAuto]

Comment 1 by jteh on 2014-02-21 06:08
I think touch has its own unique possibilities and I'd be very reluctant to tie it to the concept of mouse movement. That said, NVDA's touch support certainly does need some further work.

Regarding actions, the best thing is probably to make the activate command click if there is no action. We already do this in some cases, but not all.

I'm not sure I follow the issue with text review. The concept of review modes is subject to change, but even as it is now, I don't follow why mouse emulation would be a particularly good improvement. In most cases, you should already be able to read by line by simply moving your finger. Do you have mouse reading set to word? That's the only reason I can think of that mouse emulation would be an improvement for you.

[nvaccessAuto]

Comment 2 by k_kolev1985 on 2014-02-21 08:09
Yes, if the NVDA touch support gets improved, the reasons to tie it to the mouse movement are less important.

Yes, the idea to simulate a click where there is no action seams good, specially if it will help for easier invoking of context menus and other now hard to do stuff on the touch screen only. BTW: I've noticed that in some cases, Narrator knows what to do with some controls (specially toggle-buttons/check-boxes), where NVDA does not - it says either that there is no action, or nothing at all, even when working with the keyboard. A good example for this is the option (check-box/toggle-button) for turning on/off the automatic updates of apps in the Windows 8 Store. With NVDA, it cannot be activated neither via the space-bar, nor via object navigation or the touch screen - when executing the command, NVDA reports that the command was executed, even though it actually it was not. It can only be activated via a mouse click. If NVDA is not running, or if we pass-through the space-bar command, the state of the control will be changed with no problems at all. But I guess this issues with missing or misbehaving actions for controls should be reported in another ticket (?).

Actually, reading by word via touch exploration is not what I'm concerned about. The problem is that in some cases, reading of whole text blocks in certain places in Windows itself fails with touch explorations, while reading them with mouse tracking works as it should.

[nvaccessAuto]

Comment 3 by nvdakor on 2014-05-25 08:42
Hi,
I'd rather vote for new command set that can be invoked regardless of touch modes. The idea is as follows:

• Once an action or mouse click is about to be performed on a touchscreen, try fetching the screen coordinates under the touch point (perhaps using screen/touch explorer). Then NVDA will tell Windows to perform the desired action such as tap, tap and hold (right mouse click) and so on.
• As suggested by someone, perform clicks if no action can be done on an object. This requires that globalCommands.reviewActivate be modified based on the following implementation strategies: if this will be used on the keyboard, simply doing a click when there is no action for the object would be suffice; however, if this is exclusive to touch, then we need to detect the source of the gesture and either do a click or say that there is no action (for the keyboard). I vote for the second approach.
  Possible gesture assignments:
• Tripple tap: left-mouse click (equivalent: double tap/click).
• Tap and hold: right mouse click (and tap and hold actually works, just tested by creating a global plugin that plays a tone when a user performs tap and hold gesture).
  @mick: thoughts? I know you used to have (or have) a touch device, so thought you would know more about technical details.
  Thanks.

[nvaccessAuto]

Comment 4 by nvdakor on 2014-05-25 08:44
Hi,
As a follow-up, a possible flaw with my previous idea is what if we cannot locate the point where the finger touched the screen. Then the workaround (for now) would be to route the mouse to the navigator object then do mouse clicks. This isn't ideal for both blind and sighted users.
Thanks.

[nvaccessAuto]

Comment 5 by Joseph Lee <joseph.lee22590@... on 2014-05-25 13:58
In [9472caa]:

Global commands: added a gesture to perform right click on touch screens. re #3886.

[nvaccessAuto]

Comment 6 by nvdakor on 2014-05-25 14:00
Hi,
The new script is named touch_rightClick in global commands and is assigned to tap and hold on touchscreens. The function body is a mixture of scripts to move the mouse and to perform right clicks. If this works, then left mouse click might be similar, or we might adopt Jamie's suggestion. Thanks.

[nvaccessAuto]

Comment 8 by jteh on 2014-08-29 07:47
Thanks Joseph. Sorry for taking so long to get to this. This implementation for right click looks fine to me for the most part.

We should probably check that the object doesn't have the offscreen or invisible states and the coordinates are positive before clicking.

I think the description should be "Clicks the right mouse button at the current touch position. This is generally used to activate a context menu."

I'd be happy for review_activate to be modified to click the object if there is no action, though we should probably log a debugWarning when this occurs. I think this is just as useful for keyboard users, particularly in ribbons. See above re checking for offscreen.

[nvaccessAuto]

Comment 9 by nvdakor on 2014-08-29 08:12
Hi,
I'll modify the script doc then (others, do you have any suggestions?).
Can you give me an example object (ribbons, web element, etc.) where invisible state and offscreen is logged via dev info? I'll take a look at it and modify the routine accordingly.
For the user guide, I'll add the touchscreen command to right mouse click entry for key commands and will modify the command's description (if that's needed; I'm sure it is needed).
Thanks.

[nvaccessAuto]

Comment 10 by jteh (in reply to comment 9) on 2014-08-29 09:35
Replying to nvdakor:

Can you give me an example object (ribbons, web element, etc.) where invisible state and offscreen is logged via dev info?

Skip links on websites, but usually, you have to make sure they aren't focused. Otherwise, they become visible. Try http://www.webaim.org/

[nvaccessAuto]

Comment 11 by Joseph Lee <joseph.lee22590@... on 2014-08-29 10:26
In [59ddbae]:

Global Commands: touch right click - ignore invisible or offscreen objects, as well as if any of the object coordinates is negative. re #3886.
In some cases, objects may appear invisible until focus moves to it. This is noticeable on www.webaim.org's skip to content anchor.
Also don't bother clicking when the minimum of the coordinates is negative.
Also modified script doc for touch_rightClick (all suggested by jamie@nvaccess.org).

[nvaccessAuto]

Comment 13 by jteh on 2014-09-08 04:29
Looks good. Please update the User Guide. Do you plan to modify review_activate as per comment:8 or are you waiting on something?

[nvaccessAuto]

Comment 14 by nvdakor (in reply to comment 13) on 2014-09-08 11:35
Replying to jteh:

Looks good. Please update the User Guide. Do you plan to modify review_activate as per comment:8 or are you waiting on something?

I'll update the user guide at least (add touch gesture column like some of the review commands). As for review_activate, I'll leave it up to you since you know more about how it works than I do. Thanks for your reviews.

[nvaccessAuto]

Comment 15 by nvdakor on 2014-09-08 11:53
Hi,
Modified the user guide in commit 04a9b24:

• Changed the wording to introductory text to mouse commands table, as a touch gesture has been added.
• Added touch column in mouse commands table to include tap and hold, as well as changed the description of right mouse click command.
  Thanks.

[surfer0627]

Hi,
If there is something, caused this issue does not add to the official release?

[josephsl]

Hi,
Not yet - for the most part, it is complete - will need to test if it can be incubated safely, as it's been almost two years since commiting to that branch).
Thanks.

[bhavyashah]

@josephsl It has been two more years since #3886 (comment). :) Could you please provide a status update for your contribution?

[josephsl]

Hi, I've sort of forgotten about this, as there are more important work being done at the moment. Thanks for reminding me about this. Thanks.

…

----- Original Message ----- From: bhavyashah <notifications@github.com To: nvaccess/nvda <nvda@noreply.github.com Date sent: Sat, 18 Aug 2018 19:18:56 -0700 Subject: Re: [nvaccess/nvda] Emulate mouse on touch screens (#3886) @josephsl It has been two more years since #3886 (comment) 65. :) Could you please provide a status update for your contribution?

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #3886 (comment) 73

[josephsl]

Hi,

Found it - knew there was an issue about mouse support.

At least for right mouse click, I do have a solution, which will be done as a pull request this weekend (basically transferring parts of Enhanced Touch Gestures add-on to Core).

Thanks.

[josephsl]

Hi,

Removed t3886 branch from NV Access repo so I can rebase this branch from my side on Python 3 master (correcting linting and Python 3 issues as we go).

Thanks.

[Adriani90]

While #10065 solves the mouse right click issue, there are also other mouse actions which still need to be considered and addressed. I am reopening this discussion.

[josephsl]

Hi, I advise filing separate issues. Closing (again). From: Adriani90 <notifications@github.com> Sent: Sunday, February 16, 2020 11:16 PM To: nvaccess/nvda <nvda@noreply.github.com> Cc: Joseph Lee <joseph.lee22590@gmail.com>; Mention <mention@noreply.github.com> Subject: Re: [nvaccess/nvda] Emulate mouse on touch screens (#3886) While #10065 <#10065> solves the mouse right click issue, there are also other mouse actions which still need to be considered and addressed. I am reopening this discussion. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3886?email_source=notifications&email_token=AB4AXEC64VNSCCZNPKMJYCDRDI2SRA5CNFSM4CF2W6UKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5JYSY#issuecomment-586849355> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4AXEFSQURODZAZNHXGBUTRDI2SRANCNFSM4CF2W6UA> .

[josephsl]

Hi,

Let's discuss this on separate issues for now.

[ChrisMedley]

We now have no right click touch gesture what so ever now in 2020.1 beta 1
Worked fine in 2019.3
Why has this been removed?

[josephsl]

Hi, are you using Enhanced Touch Gestures add-on by any chance? If so, try restarting NVDA with all add-ons disabled and see if that makes a difference. If it does make a difference, I’ll take a look at it. Thanks.

[ChrisMedley]

Yes I am using enhanced touch gestures 20.03 and even with all addons disabled the right click gesture still no longer working

[josephsl]

Hi, that’s odd, as it is working on my computer. Can you try enabling input help mode (NVDA+1, or if using Enhanced Touch Gestures, four finger double tap) and see what “tap and hold with one finger” says? Or better yet, can you attach a debug log where you try right mouse click with touchscreens (restart NVDA with debug logging enabled, try a right-mouse click while NVDA is running, press NvDA+F1 to open log viewer, and copy and paste everything into a file and attach it)? Thanks.

[ChrisMedley]

OK, this is odd, I did as you asked and turned on NVDA help (nvda+1)
I tried the the single finger double tap and hold and got nothing, I then turned off nvda help and the single finger double tap and hold gesture worked!

Again turned on nvda help and it also reported right click this time also
And its still working after turning off nvda help

Dont ask me what happened but it's working now for whatever reason

apologies and thank you for your help

[josephsl]

Hi, I was about to suggest restarting your computer, but at least it is working. Let us know if it doesn’t work again. Thanks.