Description
I have a number of applications that were targeting .NET 8 preview 4 that have been upgraded to .NET preview 5 this week. In four different applications (only 3 are public), I've observed issues with their UI tests that use Microsoft.Playwright where a NullReferenceException is thrown from within the Playwright code during test execution.
These failures occur on Windows, macOS and Linux, but exactly what fails seems to vary slightly between the different applications and operating systems.
After doing various investigation, including debugging one of the applications consuming Microsoft.Playwright from source, I wonder if there's an issue somewhere related to AsyncLocal<T> and/or TaskCreationOptions.RunContinuationsAsynchronously in .NET 8 preview 5? This is partly just a hunch as I've seen PRs into the repo recently around Task-related refactoring, I might be way off in my assessment.
The code for the relevant parts of Playwright in v1.35.0 using these are here:
When referencing the Playwright code from source I added the following lines before the AsyncLocal<T> is accessed:
if (ApiZone is null)
{
throw new InvalidOperationException("ApiZone is null.");
}
if (ApiZone.Value is null)
{
throw new InvalidOperationException("ApiZone.Value is null.");
}
if (ApiZone.Value[0] is null)
{
throw new InvalidOperationException("ApiZone.Value[0] is null.");
}Re-running the tests with this edit changes the test failures to the following:
Error Message:
System.InvalidOperationException : ApiZone.Value[0] is null.
Stack Trace:
at Microsoft.Playwright.Transport.Connection.InnerSendMessageToServerAsync[T](String guid, String method, Dictionary`2 dictionary, Boolean keepNulls) in C:\Coding\microsoft\playwright-dotnet\src\Playwright\Transport\Connection.cs:line 161
This seems to suggest to me that the try-catch which is editing ApiZone.Value[0] is not running in the execution order that is expected by the async calls. This is just my inference though at trying to explain away the exception - I'm not familiar with the inner workings of Playwright's implementation.
Observing the UI tests with headless mode disabled (as well as reviewing screenshots and recordings) don't seem to suggest any issues with the applications themselves in terms of their expected behaviour (i.e. genuine test failures), it seems that something is just causing Playwright to fail internally.
An additional factor to me thinking this is something related to AsyncLocal<T> is that if I run an affected test which is data-driven and remove all of the browser combinations and just run a single browser (whatever that browser is), then the test passed on each of my attempts. Similarly, the same test may fail on different lines during the assertions, rather than always the exact same failure.
Reproduction Steps
- Clone martincostello/dependabot-helper@6006968
- Run
build.ps1 -TestFilter Can_Configure_Repositories
Expected behavior
The tests pass.
Actual behavior
One or more tests fail with an exception similar to the below:
Failed MartinCostello.DependabotHelper.UITests.Can_Configure_Repositories(browserType: "firefox", browserChannel: null) [6 s]
Error Message:
System.NullReferenceException : Object reference not set to an instance of an object.
Stack Trace:
at Microsoft.Playwright.Transport.Connection.InnerSendMessageToServerAsync[T](String guid, String method, Dictionary`2 dictionary, Boolean keepNulls) in /_/src/Playwright/Transport/Connection.cs:line 150
at Microsoft.Playwright.Transport.Connection.WrapApiCallAsync[T](Func`1 action, Boolean isInternal)
at Microsoft.Playwright.Core.ElementHandle.QuerySelectorAsync(String selector) in /_/src/Playwright/Core/ElementHandle.cs:line 181
at MartinCostello.DependabotHelper.Pages.ConfigurePage.RepositoryItem.IsSelectedAsync() in /_/tests/DependabotHelper.Tests/Pages/ConfigurePage.cs:line 141
at Shouldly.ShouldlyTaskExtensions.ShouldBeFalse(Task`1 task) in /_/tests/DependabotHelper.Tests/Infrastructure/ShouldlyTaskExtensions.cs:line 22
at MartinCostello.DependabotHelper.UITests.<>c__DisplayClass5_0.<<Can_Configure_Repositories>b__0>d.MoveNext() in /_/tests/DependabotHelper.Tests/UITests.cs:line 166
--- End of stack trace from previous location ---
at MartinCostello.DependabotHelper.Infrastructure.BrowserFixture.WithPageAsync(Func`2 action, String testName) in /_/tests/DependabotHelper.Tests/Infrastructure/BrowserFixture.cs:line 59
at MartinCostello.DependabotHelper.Infrastructure.BrowserFixture.WithPageAsync(Func`2 action, String testName) in /_/tests/DependabotHelper.Tests/Infrastructure/BrowserFixture.cs:line 64
at MartinCostello.DependabotHelper.Infrastructure.BrowserFixture.WithPageAsync(Func`2 action, String testName) in /_/tests/DependabotHelper.Tests/Infrastructure/BrowserFixture.cs:line 76
at MartinCostello.DependabotHelper.Infrastructure.BrowserFixture.WithPageAsync(Func`2 action, String testName) in /_/tests/DependabotHelper.Tests/Infrastructure/BrowserFixture.cs:line 76
at MartinCostello.DependabotHelper.Infrastructure.BrowserFixture.WithPageAsync(Func`2 action, String testName) in /_/tests/DependabotHelper.Tests/Infrastructure/BrowserFixture.cs:line 76
at MartinCostello.DependabotHelper.UITests.Can_Configure_Repositories(String browserType, String browserChannel) in /_/tests/DependabotHelper.Tests/UITests.cs:line 126
Regression?
Yes. The test pass with .NET 7.0.7 and with .NET 8 preview 4.
Known Workarounds
None.
Configuration
- .NET SDK
8.0.100-preview.5.23303.2
- Observed on Windows, Linux and macOS GitHub Actions runners as well as locally with Windows 11.
Other information
No response
Description
I have a number of applications that were targeting .NET 8 preview 4 that have been upgraded to .NET preview 5 this week. In four different applications (only 3 are public), I've observed issues with their UI tests that use Microsoft.Playwright where a
NullReferenceExceptionis thrown from within the Playwright code during test execution.These failures occur on Windows, macOS and Linux, but exactly what fails seems to vary slightly between the different applications and operating systems.
After doing various investigation, including debugging one of the applications consuming Microsoft.Playwright from source, I wonder if there's an issue somewhere related to
AsyncLocal<T>and/orTaskCreationOptions.RunContinuationsAsynchronouslyin .NET 8 preview 5? This is partly just a hunch as I've seen PRs into the repo recently around Task-related refactoring, I might be way off in my assessment.The code for the relevant parts of Playwright in v1.35.0 using these are here:
TaskCreationOptions.RunContinuationsAsynchronouslybeing usedNullReferenceExceptioncomes from, which is accessing anAsyncLocal<T>try-finallyblock awaiting a delegate that is invoking theInnerSendMessageToServerAsync()method containing the above codeWhen referencing the Playwright code from source I added the following lines before the
AsyncLocal<T>is accessed:if (ApiZone is null) { throw new InvalidOperationException("ApiZone is null."); } if (ApiZone.Value is null) { throw new InvalidOperationException("ApiZone.Value is null."); } if (ApiZone.Value[0] is null) { throw new InvalidOperationException("ApiZone.Value[0] is null."); }Re-running the tests with this edit changes the test failures to the following:
This seems to suggest to me that the
try-catchwhich is editingApiZone.Value[0]is not running in the execution order that is expected by the async calls. This is just my inference though at trying to explain away the exception - I'm not familiar with the inner workings of Playwright's implementation.Observing the UI tests with headless mode disabled (as well as reviewing screenshots and recordings) don't seem to suggest any issues with the applications themselves in terms of their expected behaviour (i.e. genuine test failures), it seems that something is just causing Playwright to fail internally.
An additional factor to me thinking this is something related to
AsyncLocal<T>is that if I run an affected test which is data-driven and remove all of the browser combinations and just run a single browser (whatever that browser is), then the test passed on each of my attempts. Similarly, the same test may fail on different lines during the assertions, rather than always the exact same failure.Reproduction Steps
build.ps1 -TestFilter Can_Configure_RepositoriesExpected behavior
The tests pass.
Actual behavior
One or more tests fail with an exception similar to the below:
Regression?
Yes. The test pass with .NET 7.0.7 and with .NET 8 preview 4.
Known Workarounds
None.
Configuration
8.0.100-preview.5.23303.2Other information
No response