New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update BlazorWebView Device Tests to have longer timesouts and more retries #21318
Conversation
public static partial class WebViewHelpers | ||
{ | ||
const int MaxWaitTimes = 30; | ||
const int WaitTimeInMS = 1000; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The retries are now consistently 30 retries * 1000ms.
|
||
if (testHtmlLoadedAttributeValue != "true") | ||
await Retry(async () => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This code here never had a retry (despite the exception message saying it did!). And this is where the test failure that motivated this change occured - so hopefully adding retries mitigates that particular flakiness.
@@ -19,7 +19,7 @@ | |||
|
|||
namespace Microsoft.Maui.DeviceTests | |||
{ | |||
public class HandlerTestBasement : TestBase, IDisposable | |||
public class HandlerTestBasement : TestBase, IAsyncDisposable |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
MAUI team reviewers: This is the part of the change that can affect other tests: I changed this to be IAsyncDisposable (which the BlazorWebView tests always did). In theory this is a good change anyway because MauiApp is IAsyncDisposable so should really always be disposed that way.
@@ -4,7 +4,7 @@ | |||
|
|||
namespace Microsoft.Maui.DeviceTests | |||
{ | |||
public abstract class CoreHandlerTestBase : HandlerTestBase, IDisposable |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(This was never needed because the base class already declared its disposableness.)
@@ -0,0 +1,20 @@ | |||
using Microsoft.Maui.Controls; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file and the next few files (AppStub, ControlHandlerTestBase, etc.) are copied from the MAUI Controls DeviceTests, but I then removed anything that wasn't needed by the BlazorWebView tests (it was too hard to get it to compile, and too hard to directly reuse the types).
@ruiminhu / @PureWeen - can one of you do a tiny review for the small changes to the DeviceTests base handler types? (I put comments in the PR to explain everything). @dotnet/dotnet-maui-blazor-eng - anyone available to review my BlazorWebView test changes? There's a lot of code shuffling but the core change is simply to add more retries to some test segments that have failed on CI (but always pass on rerun 😠 ). |
This change was motivated by some flakiness in BlazorWebView tests. The tests fail sometimes, but pass on rerun. And there are no customer reports of these particular errors either.
Example failure: https://dev.azure.com/xamarin/public/_build/results?buildId=111379&view=ms.vss-test-web.build-test-results-tab&runId=2634804&resultId=100000&paneView=debug
With this test output:
There are a few parts to this change:
HandleTestBase
classes used in other control tests. For some reason the tests would run fine on CI, but fail on my machine due to some service not being registered. No idea how that's possible. But the base test classes used in these tests got very out-of-date compared to other tests, so this brings them inline with other tests.