Playwright based BlazorTemplates.Tests skipped on helix

[HaoK]

Seeing intermittent hangs in these tests:

Added <BuildHelixPayload>false</BuildHelixPayload> to take this out of the aspnetcore-ci checks since all tests were quarantined anyways

Failing build: https://dev.azure.com/dnceng/public/_build/results?buildId=1495934&view=ms.vss-test-web.build-test-results-tab&runId=42705898&resultId=121679&paneView=debug

Console logs:

Console log: 'BlazorTemplates.Tests--net7.0' from job 869db767-1c21-489c-8aca-77d1593dcd0e workitem ecc85496-07ff-46eb-9808-b21e67bdfa4c (windows.11.amd64.clientpre.open) executed on machine a002VD2

C:\h\w\A89D0986\w\B2F209E8\e>set PATH=C:\h\w\A89D0986\p\dotnet-cli;C:\python3.9.1\Scripts\;C:\python3.9.1\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\runner\AppData\Local\Microsoft\WindowsApps 

C:\h\w\A89D0986\w\B2F209E8\e>set DOTNET_ROOT=C:\h\w\A89D0986\p\dotnet-cli 

C:\h\w\A89D0986\w\B2F209E8\e>set DOTNET_CLI_TELEMETRY_OPTOUT=1 

C:\h\w\A89D0986\w\B2F209E8\e>set DOTNET_CLI_HOME=C:\h\w\A89D0986\w\B2F209E8\e\.dotnet 

C:\h\w\A89D0986\w\B2F209E8\e>set NUGET_PACKAGES=C:\h\w\A89D0986\w\B2F209E8\e\.nuget 

C:\h\w\A89D0986\w\B2F209E8\e>call RunPowershell.cmd mssql\InstallSqlServerLocalDB.ps1   || exit /b 1 
"PS: Running 'C:\h\w\A89D0986\w\B2F209E8\e\mssql\InstallSqlServerLocalDB.ps1' "
Installing SQL Server 2016 Express LocalDB

Downloading 'SqlLocalDB.msi' to 'C:\h\w\A89D0986\w\B2F209E8\e\mssql\obj\SqlLocalDB.msi'.
GET https://download.microsoft.com/download/9/0/7/907AD35F-9F9C-43A5-9789-52470555DB90/ENU/SqlLocalDB.msi
Download of 'https://download.microsoft.com/download/9/0/7/907AD35F-9F9C-43A5-9789-52470555DB90/ENU/SqlLocalDB.msi' complete, saved to C:\h\w\A89D0986\w\B2F209E8\e\mssql\obj\SqlLocalDB.msi...

Running 'msiexec /package "C:\h\w\A89D0986\w\B2F209E8\e\mssql\obj\SqlLocalDB.msi" /NoRestart /Passive IACCEPTSQLLOCALDBLICENSETERMS=YES HIDEPROGRESSBAR=YES'.

[ghost]

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[TanayParikh]

We have a some overlap in our selenium and playwright tests.

Selenium:

Playwright:

In the interest of improving maintainability, should we consider removing the playwright test suite entirely. The playwright tests have already been removed for the components e2e tests. This means, if we were to remove playwright from the template tests, we would no longer have playwright within this repo and eliminate that dependency entirely. As it stands, the selenium template tests have been reliable.

Tests we'd have to port from Playwright to Selenium in this case:

• BlazorTemplates.Tests.BlazorServerTemplateTest.BlazorServerTemplate_IdentityWeb_BuildAndPublish
• BlazorTemplates.Tests.BlazorWasmTemplateTest.BlazorWasmStandaloneTemplate_Works
• BlazorTemplates.Tests.BlazorWasmTemplateTest.BlazorWasmHostedTemplate_Works
• BlazorTemplates.Tests.BlazorWasmTemplateTest.BlazorWasmHostedPwaTemplate_Works
• BlazorTemplates.Tests.BlazorServerTemplateTest.BlazorServerTemplateWorks_IndividualAuth
• BlazorTemplates.Tests.BlazorServerTemplateTest.BlazorServerTemplateWorks_NoAuth

Test that we'd need to expand (currently build and publish, would need to add validation):

• Templates.Test.BlazorWasmTemplateTest.BlazorWasmStandalonePwaTemplateCanCreateBuildPublish -> BlazorTemplates.Tests.BlazorWasmTemplateTest.BlazorWasmStandalonePwaTemplate_Works

[TanayParikh]

As part of this task, remember to unskip the test via the test attribute: #40051 / #40129

[TanayParikh]

Note we aren't currently sure whether we want to run the Blazor Template Playwright tests on helix due to platform specific issues with Playwright. Focus is on the build agents.

If we do decide to enable helix, we'll have to subsequently complete: #30825

Also somewhat related: #34554

• if we do end up going the helix route, then we may want to see if the ARM machines can be re-enabled for those tests

[TanayParikh]

Note we aren't currently sure whether we want to run the Blazor Template Playwright tests on helix due to platform specific issues with Playwright. Focus is on the build agents.

We're running the tests specifically on the build agents for now due to issues with Helix compatibility. Backlogging this until we decide we'd like these tests running on helix.

[ghost]

Thanks for contacting us.

We're moving this issue to the .NET 9 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[mkArtakMSFT]

Closing this as a dupe of #46430