[wasm] WBT: Blazor tests on windows failing with Failed to bind to address for random ports

[radical]

Reproduction Steps

• Queue: browser-wasm linux Release WasmBuildTests
• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=413445&view=results
• Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-heads-release-80-rc1-e5e75e14b32149239e/NoWorkload-Wasm.Build.Tests/1/console.cabf329c.log?helixlogtype=result
• Output:

    Wasm.Build.Tests.Blazor.WorkloadRequiredTests.WorkloadNotRequiredForInvariantGlobalization(config: "Release", invariant: False, publish: False) [FAIL]
       Expected 0 exit code but got 134: /root/helix/work/workitem/e/dotnet-none/dotnet run -c Release --no-build
      Standard Output:
      [] fail: Microsoft.Extensions.Hosting.Internal.Host[11]
      []       Hosting failed to start
      []       System.IO.IOException: Failed to bind to address http://127.0.0.1:5145: address already in use.
      []        ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address already in use
      []        ---> System.Net.Sockets.SocketException (98): Address already in use
      []          at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, Boolean disconnectOnFailure, String callerName)
      []          at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
      []          at System.Net.Sockets.Socket.Bind(EndPoint localEP)
      []          at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
      []          at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
      []          --- End of inner exception stack trace ---
      []          at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
      []          at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass28_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
      []       --- End of stack trace from previous location ---
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
      []          --- End of inner exception stack trace ---
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(ListenOptions[] listenOptions, AddressBindContext context, Func`2 useHttps, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
      []          at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
      []          at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
      []          at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
      [] Unhandled exception. System.IO.IOException: Failed to bind to address http://127.0.0.1:5145: address already in use.
      []  ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address already in use
      []  ---> System.Net.Sockets.SocketException (98): Address already in use
      []    at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, Boolean disconnectOnFailure, String callerName)
      []    at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
      []    at System.Net.Sockets.Socket.Bind(EndPoint localEP)
      []    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
      []    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
      []    --- End of inner exception stack trace ---
      []    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
      []    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass28_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
      [] --- End of stack trace from previous location ---
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
      []    --- End of inner exception stack trace ---
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.LocalhostListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(ListenOptions[] listenOptions, AddressBindContext context, Func`2 useHttps, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
      []    at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
      []    at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
      []    at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
      []    at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
      []    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
      []    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
      []    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
      []    at Microsoft.AspNetCore.Components.WebAssembly.DevServer.Program.Main(String[] args)
      

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "BuildRetry": false,
  "ErrorPattern": "Microsoft.AspNetCore.Connections.AddressInUseException:",
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎
Result validation: ⚠️ Build internal information not found. This may happen if your build is too old. Please use a build that is no older than two weeks. If the problem persists, contact .NET Engineering Services Team and share this issue.
Validation performed at: 10/17/2023 1:46:13 AM UTC

Report

Build	Definition	Test	Pull Request
703469	dotnet/runtime	Wasm.Build.Tests.Blazor.BuildPublishTests.DefaultTemplate_WithoutWorkload	#103233

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	1

[ghost]

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Error Blob

Tagging this for KnownBuildError in case it's happening consistently and we need to fix this in 8.0.

{
  "ErrorMessage": "",
  "BuildRetry": false,
  "ErrorPattern": "Wasm\.Build\.Tests.*System\.IO\.IOException: Failed to bind to address http://127\.0\.0\.1:(?!5000)",
  "ExcludeConsoleLog": false
}

Reproduction Steps

• Queue: browser-wasm linux Release WasmBuildTests
• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=413445&view=results
• Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-heads-release-80-rc1-e5e75e14b32149239e/NoWorkload-Wasm.Build.Tests/1/console.cabf329c.log?helixlogtype=result

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Author:	radical
Assignees:	-
Labels:	arch-wasm, blocking-clean-ci, Known Build Error
Milestone:	-

[radical]

@missymessa Any idea why this is not getting picked up for known-build-error analysis?

[radical]

@maraf Looking at the logs, the same port number is used in one of the earlier tests, but even if that process is still running the devserver shouldn't have picked a port still in use. Seems like a devserver issue?

[AlitzelMendez]

@radical it wasn't picked up because it had escaping problems, you can notice this because the json message was in red, like this:

{
 "ErrorPattern": "Wasm\.Build\.Tests.*System\.IO\.IOException: Failed to bind to address http://127\.0\.0\.1:(?!5000)"
}

I fixed the escaping and now the validation was able to run, but I noticed that your regex was trying to look into multiple lines, this feature is not available for known issues, the error needs to be a single line, also the validation returned a backtracking error, all this needs to be fixed in order than known issues can work

let me know if you need help with any of this

[radical]

I fixed the escaping and now the validation was able to run, but I noticed that your regex was trying to look into multiple lines, this feature is not available for known issues,

Oh, I was basing this on the doc:

https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssues.md#regex-matching

We recommend you test your regular expression, to do it you can use [regex101 tester](https://regex101.com/) (choose .NET (C#) flavor) with the following regex options:

Single line
Insentitive
No backtracking

The single line here would mean that . would match newlines also, even according to that website.

If that isn't supported then what is the recommended way to identify failure messages that are not unique. For example System.IO.IOException: Failed to bind to address can match lots of test failures but the issue is specifically about this failure in Wasm.Build.Tests. Without the multi-line regex, how do I narrow down the match?

[AlitzelMendez]

If that isn't supported then what is the recommended way to identify failure messages that are not unique. For example System.IO.IOException: Failed to bind to address can match lots of test failures but the issue is specifically about this failure in Wasm.Build.Tests. Without the multi-line regex, how do I narrow down the match?

Unfortunately, there is not a lot that we can suggest on these cases, if this is narrow to a specific test name you can use the test name, otherwise the only option is try to identify an error line that is unique to your issue.

we provide a little bit more information about why the multiline is not supported of this issue: dotnet/arcade#13253

[radical]

If that isn't supported then what is the recommended way to identify failure messages that are not unique. For example System.IO.IOException: Failed to bind to address can match lots of test failures but the issue is specifically about this failure in Wasm.Build.Tests. Without the multi-line regex, how do I narrow down the match?

Unfortunately, there is not a lot that we can suggest on these cases, if this is narrow to a specific test name you can use the test name, otherwise the only option is try to identify an error line that is unique to your issue.

we provide a little bit more information about why the multiline is not supported of this issue: dotnet/arcade#13253

Maybe allowing matching against the helix item name, or maybe two single line patterns to match? That should be cheaper than a multiline regex search over a big text file.

[pavelsavara]

does it not happen anymore ? Or the filter doesn't work ?