Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Android visual test screenshots more deterministic #16591

Merged
merged 4 commits into from
Aug 14, 2023

Conversation

BretJohnson
Copy link
Member

@BretJohnson BretJohnson commented Aug 7, 2023

Make two changes so that Android visual test screenshots are more deterministic,
returning the same thing whether running locally or on CI, part of a single test run
or a batch test run:

  1. Use the same nav bar style for the gallery pages and the issues pages. This looks better aesthetically and avoids an issue where issue page screenshots aren't consistent, depending on whether you've ever visited a gallery page or not.

  2. Crop the 3 button nav from the bottom of the screen for Android. That UI isn't part of the app
    and the color of the buttons can vary (they temporarily change to a different color when pressed).
    Not including that UI in the screenshots avoids any spurious differences here.

See more details in the Teams chat:
https://teams.microsoft.com/l/message/19:46c9eef68efe400393f538bd778e4718@thread.v2/1691428231798?context=%7B%22contextType%22%3A%22chat%22%7D

Use the same nav bar style for the gallery pages and the issues pages.
This looks better aesthetically and avoids an issue where issue
page screenshots aren't consistent, depending on whether you've
ever visited a gallery page or not.

See more details in the Teams chat here:
https://teams.microsoft.com/l/message/19:46c9eef68efe400393f538bd778e4718@thread.v2/1691428231798?context=%7B%22contextType%22%3A%22chat%22%7D
Eilon
Eilon previously approved these changes Aug 7, 2023
Copy link
Member

@Eilon Eilon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Teams chat is not accessible to me, but this sounds reasonable.

@Eilon Eilon added the area-infrastructure CI, Maestro / Coherency, upstream dependencies/versions label Aug 7, 2023
@BretJohnson BretJohnson changed the title Use consistent nav bar styling Make Android visual test screenshots more deterministic Aug 7, 2023
@Eilon
Copy link
Member

Eilon commented Aug 10, 2023

Looks like the test is failing on iOS and Windows now: https://dev.azure.com/xamarin/public/_build/results?buildId=92639&view=ms.vss-test-web.build-test-results-tab&runId=2055450&resultId=100012&paneView=debug

Error message
VisualTestUtils.VisualTestFailedException :
Snapshot different than baseline: Issue15330Test.png (12.01% difference)
If the correct baseline has changed (this isn't a a bug), then update the baseline image.
See test attachment or download the build artifacts to get the new snapshot file.


More info: https://aka.ms/visual-test-workflow



Stack trace
   at VisualTestUtils.VisualRegressionTester.Fail(String message)
   at VisualTestUtils.VisualRegressionTester.VerifyMatchesSnapshot(String name, ImageSnapshot actualImage, String environmentName, ITestContext testContext)
   at Microsoft.Maui.AppiumTests.UITestBase.VerifyScreenshot(String name) in /Users/builder/azdo/_work/3/s/src/Controls/tests/UITests/Tests/UITestBase.cs:line 183
   at Microsoft.Maui.AppiumTests.Issues.Issue15330.Issue15330Test() in /Users/builder/azdo/_work/3/s/src/Controls/tests/UITests/Tests/Issues/Issue15330.cs:line 18
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Copy link
Member

@Eilon Eilon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to fix failing test(s).

@BretJohnson BretJohnson requested a review from a team as a code owner August 11, 2023 03:24
@Eilon
Copy link
Member

Eilon commented Aug 11, 2023

@BretJohnson - iOS and Windows still seem sad 😢

VisualTestUtils.VisualTestFailedException :
Snapshot different than baseline: Issue15330Test.png (6.93% difference)
If the correct baseline has changed (this isn't a a bug), then update the baseline image.
See test attachment or download the build artifacts to get the new snapshot file.


More info: https://aka.ms/visual-test-workflow



Stack trace
   at VisualTestUtils.VisualRegressionTester.Fail(String message)
   at VisualTestUtils.VisualRegressionTester.VerifyMatchesSnapshot(String name, ImageSnapshot actualImage, String environmentName, ITestContext testContext)
   at Microsoft.Maui.AppiumTests.UITestBase.VerifyScreenshot(String name) in /Users/builder/azdo/_work/3/s/src/Controls/tests/UITests/Tests/UITestBase.cs:line 183
   at Microsoft.Maui.AppiumTests.Issues.Issue15330.Issue15330Test() in /Users/builder/azdo/_work/3/s/src/Controls/tests/UITests/Tests/Issues/Issue15330.cs:line 18
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

@Eilon
Copy link
Member

Eilon commented Aug 11, 2023

Though, the % difference is lower now, so it's getting better 😉

@PureWeen PureWeen requested a review from Eilon August 12, 2023 17:24
@PureWeen PureWeen merged commit e36d222 into main Aug 14, 2023
39 checks passed
@PureWeen PureWeen deleted the update-tests-navbar-background branch August 14, 2023 15:14
rmarinho pushed a commit that referenced this pull request Aug 19, 2023
* Use consistent nav bar styling

Use the same nav bar style for the gallery pages and the issues pages.
This looks better aesthetically and avoids an issue where issue
page screenshots aren't consistent, depending on whether you've
ever visited a gallery page or not.

See more details in the Teams chat here:
https://teams.microsoft.com/l/message/19:46c9eef68efe400393f538bd778e4718@thread.v2/1691428231798?context=%7B%22contextType%22%3A%22chat%22%7D

* Crop off the 3 button nav at the bottom of Android screens.

* Only call iOS SetPrefersLargeTitles on gallery pages

* Update snapshots, with updated tabbar color
@github-actions github-actions bot locked and limited conversation to collaborators Dec 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-infrastructure CI, Maestro / Coherency, upstream dependencies/versions platform/android 🤖
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants