A Short Explanation: Why there are problems with landscape orientation

[mykola-mokhnach]

Facebook decided to close Wiki from editing by third-parties, so I just leave it here.

Bug Description

There are many complains about interaction issues in non-default screen orientation mode. All these problems are coming from a single source. XCTest has a known bug when it fails to transform element coordinates properly in non-portrait orientation. It means that if https://github.com/facebook/WebDriverAgent/blob/52512a7fb39a697fc727d372c8765a43c74f6d37/WebDriverAgentLib/Commands/FBOrientationCommands.m#L101 was called with orientation value different from UIDeviceOrientationPortrait then interface elements, which belong to some windows from the current hierarchy, will change their frames, while others won't.

The Problem

Usually, it is the case, that if the particular window belongs to a system service then its descendants will have "correct" frame coordinates and if it a custom application then the4 coordinates will be "corrupted". Under "corrupted" I assume that click or other actions will be executed not in the place, where we expect it to be executed (or simply offscreen) if sent to such element. This also includes the case when coordinates are calculated based on the global screen size, since the "real" screen dimensions constructed by XCTest are not the same as returned to you.

The described problem also affects, for example, screenshots. The are incorrectly rotated in non-portrait mode (the bottom side of the picture is not the bottom of the screen).

Also, it is known, that the problem shows itself differently in iOS9 and iOS10+.

The Solution

The WDA implementation from Facebook tries to workaround the problem by adjusting coordinates based on the current interface orientation value. It assumes the coordinates should be always fixed if the orientation is different from the default one. See

https://github.com/facebook/WebDriverAgent/blob/52512a7fb39a697fc727d372c8765a43c74f6d37/WebDriverAgentLib/Commands/FBElementCommands.m#L457

and

https://github.com/facebook/WebDriverAgent/blob/52512a7fb39a697fc727d372c8765a43c74f6d37/WebDriverAgentLib/Categories/XCUIElement%2BFBTap.m#L37

Here you have the methods, which "fix" the coordinates:

https://github.com/facebook/WebDriverAgent/blob/52512a7fb39a697fc727d372c8765a43c74f6d37/WebDriverAgentLib/Utilities/FBMathUtils.m#L41
https://github.com/facebook/WebDriverAgent/blob/52512a7fb39a697fc727d372c8765a43c74f6d37/WebDriverAgentLib/Utilities/FBMathUtils.m#L56

So Why It Still Fails?

This works in most of the cases, but sometimes, due to the fact described in The Problem section, element coordinates might not be inverted. And since we try to "fix" them anyway, this leads to "missclicks". Also, Apple still changes this behaviour in major XCTest releases (perhaps, it's even going to be fixed sometimes ;-) ).

Is There Any Solution?

In the Appium fork we've applied more "advanced" logic to detect the fact of coordinates corruption in non-default screen orientations. This logic relies on the fact that if an element has "corrupted" frame then it's window is inverted relatively to the application's window frame. You can check

https://github.com/appium/WebDriverAgent/blob/02887c171699821314c1928642303c36f037ae8a/WebDriverAgentLib/Utilities/FBBaseActionsSynthesizer.m#L57

https://github.com/appium/WebDriverAgent/blob/02887c171699821314c1928642303c36f037ae8a/WebDriverAgentLib/Utilities/FBMathUtils.m#L86

for more details. We still don't know whether the particular element coordinates are OK if global coordinates are used. Also, this trick does not work with upside-down orientation. That is why the solution would be to avoid using global coordinates, but rather rely on some particular parent container UI element (or the window itself, which is always a child of Application element), while calculating offsets.

How Can I Help?

We are welcome to contributions, that can help us to solve the described problems or find better/faster ways to workaround them.

[shynkevich-alex]

any progress?

[GWesley]

All tried, not work on iPhone7 plus 11.3

[vikramvi]

Facebook should show a bit more Sympathy & Empathy to thousands of struggling automation engineers across globe.

I'm sure they have enough money to allocate dedicated resources for this critical project.

@mykola-mokhnach Thanks for your endless patience with FB :)