Flaky Various Blazor Template timeouts/failures

[jkotalik]

It looks like almost any blazor template test can fail due to a shared fixture being used between them and it timing out.

Builds: https://dev.azure.com/dnceng/internal/_build/results?buildId=1033436&view=ms.vss-test-web.build-test-results-tab&runId=32069600&resultId=125477&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

https://dev.azure.com/dnceng/public/_build/results?buildId=1035875&view=ms.vss-test-web.build-test-results-tab&runId=32132094&resultId=125303&paneView=debug

System.TimeoutException : Timeout 30000ms exceeded.
=========================== logs ===========================
<launching> C:\h\w\AC6D092A\w\A51C0945\e\ms-playwright\firefox-1225\firefox\firefox.exe -no-remote -headless -profile C:\h\w\AC6D092A\t\playwright_firefoxdev_profile-0AKzH5 -juggler-pipe -silent
<launched> pid=4752
[err] *** You are running in headless mode.
[out]
[out] Juggler listening to the pipe
[out] console.warn: SearchSettings: "get: No settings file exists, new profile?" (new Error("", "(unknown module)"))
[err] JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
[err]
[err] ###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[err]
Note: use DEBUG=pw:api environment variable and rerun to capture Playwright logs.

[ghost]

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[HaoK]

Also will track related quarantined blazor template tests

• Flaky BlazorWasmHostedTemplate_IndividualAuth_Works_WithOutLocalDB test #30820
• Test Failure: BlazorWasmHostedTemplate_IndividualAuth_Works_WithLocalDB #30700
• Flaky BlazorServerTemplateWorks_IndividualAuth test #30807

[Tratcher]

BlazorWasmHostedPwaTemplate_Works Status:
30 days, 89 runs, 41% pass rate.
It looks like it was passing from 4/14-4/26 before hitting the current failure 100% of the time:

System.TimeoutException : Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for selector "ul" to be visible
============================================================
Note: use DEBUG=pw:api environment variable to capture Playwright logs.

[HaoK]

Finally got a chance to look at this, looks like things are passing again after todays latest SDK update by @captainsafia , not sure what this is indicating, but looking at some of these failures, the error is due to a missing method exception from string values in Http header dictionary.

@Tratcher @dougbu @javiercn any ideas what might have been out of sync here? Could this be due to IdentityServer4 being in the call stack? But looks like something was busted for 2 weeks until we got an sdk update, might be good to understand the cause in still

[658.589s] [Templates.Test.BlazorWasmTemplateTest] [Information]       System.MissingMethodException: Method not found: 'Microsoft.Extensions.Primitives.StringValues Microsoft.AspNetCore.Http.IHeaderDictionary.get_Authorization()'.
[658.589s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
[658.589s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in System.Private.CoreLib.dll:token 0x60047e1+0x28
[658.589s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync() in Microsoft.AspNetCore.Authentication.JwtBearer.dll:token 0x6000032+0x1c
[658.589s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.AuthenticateAsync() in Microsoft.AspNetCore.Authentication.dll:token 0x6000039+0x9c
[658.589s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme) in Microsoft.AspNetCore.Authentication.Core.dll:token 0x6000021+0x1ee
[658.589s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.AuthenticateAsync() in Microsoft.AspNetCore.Authentication.dll:token 0x6000039+0x8f
[658.589s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme) in Microsoft.AspNetCore.Authentication.Core.dll:token 0x6000021+0x1ee
[658.589s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) in Microsoft.AspNetCore.Authentication.dll:token 0x6000044+0x2fa
[658.590s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context) in IdentityServer4.dll:token 0x60001df+0x9a
[658.590s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context) in Microsoft.AspNetCore.Http.Abstractions.dll:token 0x6000084+0x10c
[658.590s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context) in Microsoft.AspNetCore.Http.Abstractions.dll:token 0x600007b+0x1ec
[658.590s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore.MigrationsEndPointMiddleware.Invoke(HttpContext context) in Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore.dll:token 0x600008e+0x2a4
[658.590s] [Templates.Test.BlazorWasmTemplateTest] [Information]          at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context) in Microsoft.AspNetCore.Diagnostics.dll:token 0x60000aa+0x82

[Tratcher]

IHeaderDictionary.get_Authorization are new properties added in 6.0. I'll ping you.

[HaoK]

@Tratcher and I chatted a bit, and looks like @dougbu you might have fixed this issue already with 07c6a62#diff-5ae897b6b974eb8d878b7a7635d5d59d478115e73dba09f3832adcbafb7d137a Where we weren't testing templates against the latest aspnet-shared-fx bits. This should be fixed by your PR over the weekend right?

[dougbu]

We had a window in which we weren't testing templates using the latest ASP.NET runtime and my PR this weekend fixed that issue. If it explains the remaining flakiness, we're good.

[HaoK]

Thanks @dougbu , I think that explains the drop to 0%, we still have a ways to go on the rest

Status as of 5/11 for the various blazor template tests:

Seems pretty reliable (failed 2 times out of 1698 - last 14 days - 99.88%)
Templates.Test.BlazorWasmTemplateTest.BlazorWasmHostedTemplate_AzureActiveDirectoryTemplate_Works

(Unknown because these were 0% for last 2 weeks)
Templates.Test.BlazorWasmTemplateTest.BlazorWasmHostedTemplate_IndividualAuth_Works_WithOutLocalDB
Templates.Test.BlazorWasmTemplateTest.BlazorWasmHostedPwaTemplate_Works
Templates.Test.BlazorWasmTemplateTest.BlazorWasmHostedTemplate_Works

Broken due to missing method exception from 4/26 (sdk update?), seems healthy-ish after 5/11 sdk update

(75-87% last 30 days)
Templates.Test.BlazorWasmTemplateTest.BlazorWasmStandaloneTemplate_Works
Templates.Test.BlazorWasmTemplateTest.BlazorWasmStandalonePwaTemplate_Works
Templates.Test.BlazorServerTemplateTest.BlazorServerTemplateWorks_IndividualAuth
Templates.Test.BlazorServerTemplateTest.BlazorServerTemplateWorks_NoAuth

Mostly 30 second timeout failures, waiting for selector "ul" to be visible
sporadic PlaywrightSharp.PlaywrightSharpException : net::ERR_CERT_DATABASE_CHANGED at https://localhost:65432 as well
Sporadic total work item fail with no obvious errors (something in dotnet publish/build returned error code)

[HaoK]

Moving this to preview 7, the retry PR once merged should give us more data to see how the tests behave with 3 retries, hopefully that will let us unquarantine some of these, but likely there will be more fixes/tweaks needed to get to an acceptable pass rate.

[TanayParikh]

Tests currently skipped via #40129, will be unskipped via #38818 along with being re-enabled in helix.

[ghost]

Thanks for contacting us.

We're moving this issue to the .NET 7 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[mkArtakMSFT]

@TanayParikh do we still need this issue given the linked issue above?

[TanayParikh]

Closing as duplicate of #38818.

[TanayParikh]

Re-opening as this fails in the (non-helix) build agents too.