implement nativeWebTap for Android

[0x1mason]

Related to #4005.

Workaround is deriving x.y coordinates from WV element and then passing them to the native context.

[jlipps]

we already do this under the hood in the iOS code (triggered via the nativeWebTap capability; so it's possible we can refactor some of that out).

[0x1mason]

I'd like to fix this bug. Been hacking on it a bit already.

[0x1mason]

@jlipps Instead of worrying about state (ie isWebContext) how about we only do taps via X,Y coordinates? Ie we always do Element ==> X,Y coord conversion on the server side. That'd really simplify things.

In fact, we could do all the gestures that way (obviously all is out of scope for this ticket--just a thought 😉).

[0x1mason]

Every time we get an action, we check for the element field and if it exists convert it to xy. Now all the devices are getting normalized data and don't have to worry about it.

[0x1mason]

If we do need to worry about state, then we only have one place to do it.

[jlipps]

Instead of worrying about state (ie isWebContext) how about we only do taps via X,Y coordinates?

When we have a native element, I want to use the native click method that Android provides; using X,Y coordinates instead is going to be a lot less stable. We'll probably run into lots of edge cases with coordinate maps being off due to different versions of android including/not including a header bar in coordinate reporting, different resolutions, etc.

I think this is a case where we really do need to worry about state. Especially given that we're using Chromedriver, we have no guarantee, since we're not in charge of Chromedriver element naming conventions, that there's no overlap between Chromedriver's element namespace and ours. It could be that there's element with id "2" in Chromedriver-land and also in Appium-land. We'll end up needing to use isWebContext to disambiguate anyway, so we should just use it exclusively from the beginning.

[0x1mason]

What you're saying is that 10,10 in a webview may not map to 10,10 in
the native context, right?

On 11/14/14, Jonathan Lipps notifications@github.com wrote:

Instead of worrying about state (ie isWebContext) how about we only do
taps via X,Y coordinates?

When we have a native element, I want to use the native click method that
Android provides; using X,Y coordinates instead is going to be a lot less
stable. We'll probably run into lots of edge cases with coordinate maps
being off due to different versions of android including/not including a
header bar in coordinate reporting, different resolutions, etc.

I think this is a case where we really do need to worry about state.
Especially given that we're using Chromedriver, we have no guarantee, since
we're not in charge of Chromedriver element naming conventions, that there's
no overlap between Chromedriver's element namespace and ours. It could be
that there's element with id "2" in Chromedriver-land and also in
Appium-land. We'll end up needing to use isWebContext to disambiguate
anyway, so we should just use it exclusively from the beginning.

---

Reply to this email directly or view it on GitHub:
#4075 (comment)

[0x1mason]

@jlipps I'm also not clear on why overlapping namespaces matter. We
definitely would have to pay enough attention to state to get the
absolute position of an element; we'd have to know what to query. But
after the absolute position is figured, then as long 10,10 in a
webview is 10,10 in a native context, then presumably we'd be in good
shape.

On 11/14/14, Jonathan Lipps notifications@github.com wrote:

Instead of worrying about state (ie isWebContext) how about we only do
taps via X,Y coordinates?

When we have a native element, I want to use the native click method that
Android provides; using X,Y coordinates instead is going to be a lot less
stable. We'll probably run into lots of edge cases with coordinate maps
being off due to different versions of android including/not including a
header bar in coordinate reporting, different resolutions, etc.

I think this is a case where we really do need to worry about state.
Especially given that we're using Chromedriver, we have no guarantee, since
we're not in charge of Chromedriver element naming conventions, that there's
no overlap between Chromedriver's element namespace and ours. It could be
that there's element with id "2" in Chromedriver-land and also in
Appium-land. We'll end up needing to use isWebContext to disambiguate
anyway, so we should just use it exclusively from the beginning.

---

Reply to this email directly or view it on GitHub:
#4075 (comment)

[0x1mason]

To clarify 10,10 being the same, I mean the coords correspond to the
same pixel on the screen in both contexts.

On 11/14/14, Eric Millin eric@saucelabs.com wrote:

@jlipps I'm also not clear on why overlapping namespaces matter. We
definitely would have to pay enough attention to state to get the
absolute position of an element; we'd have to know what to query. But
after the absolute position is figured, then as long 10,10 in a
webview is 10,10 in a native context, then presumably we'd be in good
shape.

On 11/14/14, Jonathan Lipps notifications@github.com wrote:

Instead of worrying about state (ie isWebContext) how about we only do
taps via X,Y coordinates?

When we have a native element, I want to use the native click method that
Android provides; using X,Y coordinates instead is going to be a lot less
stable. We'll probably run into lots of edge cases with coordinate maps
being off due to different versions of android including/not including a
header bar in coordinate reporting, different resolutions, etc.

I think this is a case where we really do need to worry about state.
Especially given that we're using Chromedriver, we have no guarantee,
since
we're not in charge of Chromedriver element naming conventions, that
there's
no overlap between Chromedriver's element namespace and ours. It could be
that there's element with id "2" in Chromedriver-land and also in
Appium-land. We'll end up needing to use isWebContext to disambiguate
anyway, so we should just use it exclusively from the beginning.

---

Reply to this email directly or view it on GitHub:
#4075 (comment)

[0x1mason]

In any case, it sounds like you've experienced instability with coordinates, so I'll stay with the current approach for now.

[jlipps]

Can you clarify what you think I mean by "element namespace"? Or I could say, "element ID-space"

[ychab]

Hi,

Sorry but just to be sure, this issue aims at executing touch actions in webview context againsts WB elements for Android (already implemented for iOS)?

Because for the moment, this is not yet implemented. And you cannot find WB elements in a native context for performance reasons (in order to use touch action in native context with WB elements). The only alternative you have for now is to convert yourself X,Y coordinates of the WB elements in the X,Y coordinates of the native context and apply touch actions in native context with these converted X,Y coordinates.

So to avoid this, once this issue would be solved, the end-user in mobile web test case on Android would be able to find WB element in webview context (with Chromedriver), send request to Appium for touch actions on them, still in WB context. Appium won't proxy these requests to the Chromedriver (avoidProxy regex). Instead, it would try to convert X,Y coordinates of the WB elements in the X,Y coordinates of the native context (including browser's header bar, orientation screen, etc) and then execute touch actions with these converted X,Y in native context, so using UIautomator (or something like that, sorry).

It is correct?

If true, it would be awesome ! Because indeed, chromedriver didn't implements well all touch commands and it could be a great work-around !

[0x1mason]

@chaby1 Yes, it sounds like you've got it.

[sonton]

It has been one year since this bug open. Is there any update on it yet ? It would help me big time.
Thanks

[monleylu]

when can solve this problem ?there are so much many mobile webelement use tap instead of click event to respone user action on android chrome browse

[fabytta]

This need to be resolved somehow, and switching between WB and native contexts to get the coordinates and then tap on the position seems to be too much time consuming (due to several switches in the same page)
Any other solution or workaround?

[TheCodeSharman]

I'm also in the situation where the application under test uses a WebView and a need to simulate tap events instead of click events. Are there any plans to implement this? Would be really helpful for me too.

[jlipps]

Yes, this is in our prioritized backlog so there are plans to implement it. It will still take some time to get to, however.

[anjanabendre]

Will you please update us on the fix of this issue.
We have hybrid app, developed using ionic cordova platform. Most of the pages are in webview and tap/touch and drag and drop is required for all elements. Now its not working.

So our automation for this application is completely on hold.

Appreciate quick fix on these issues.

Thanks,
Anjana Bendre

[JackGarbiec]

I would also ask if it's in the plans to be completed any time soon? It takes a lot of 'sticking a plaster' solutions to get touch actions to work in an android web app.

[mrakibmiah]

its been awhile. any update regarding this issue?

[shibupanda]

@jlipps
It may not be a right thread but I just like to know whether is there any function available where we can pass web element from webview and expect to return X and Y co-ordinate as Native. So with this I can bypass nativeWebTap capability as I don't want to tap every element with co-ordinate.

I am using Java for the testcases.

[jlipps]

@shibupanda no, there is no such functionality, but you could accomplish this on the client side by using the same algorithm that nativeWebTap uses, basically:

1. find the rect of the element in the webpage
2. find the rect for the webview containing the element
3. convert the web position of the element to a native position by translating the scale and position of the webview to that of the entire native screen.