Container tests fail with EADDRINUSE

[jbagga]

ContainerFallbackWebTests.KestrelX64CoreCLR failed on the Ubuntu test run
http://aspnetci/viewLog.html?buildId=190297&tab=buildResultsDiv&buildTypeId=XPlat_Linux_UbuntuUniverseCoreCLR

[jbagga]

cc @halter73

[halter73]

I have no idea what's going on with this one. The client simply failed to establish a TCP connection. Maybe Kestrel just took too long to start this one time? That's why there's a retry.

Even though TC marked it as flaky, if you look at the test history it looks like this is the only time this test has ever failed: http://aspnetci/project.html?tab=testDetails&projectId=XPlat&testNameId=8180339492066270673&page=1

[jbagga]

ConfigWalkingValuesWebTests.KestrelX64CoreCLR
Failure details: http://aspnetci/viewLog.html?buildId=191998&tab=buildResultsDiv&buildTypeId=XPlat_Linux_UbuntuUniverseCoreCLR.

@halter73 This one has the same error. Fails to connect and retry is canceled.

@muratg Are these rare failures acceptable?

[Eilon]

@halter73 we want to abolish all flaky tests. We have too many flaky tests across the board, so it's a death-by-a-thousand-paper-cuts situation.

[muratg]

@natemcmaster could you take a look? (@halter73 is out this week.)

[natemcmaster]

ContainerFallbackWebTests failed with

Unhandled Exception: System.IO.IOException: Failed to bind to address http://localhost:6000 on the IPv4 loopback interface: port already in use. ---> System.AggregateException: One or more errors occurred. (Error -98 EADDRINUSE address already in use) ---> Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -98 EADDRINUSE address already in use

ConfigWalkingValuesWebTests failed with the same error, but ports 5900. I couldn't find any other tests that explicitly use ports 5900 or 6000. Could this be an issue in Kestrel?

[Eilon]

Didn't we have some automatic port selector thingy that we used somewhere for Kestrel tests? Or some other tests?

@mikeharder @Tratcher maybe you guys remember?

[natemcmaster]

If we do that would help. To help debug situations like this, I'm make a pull request to make SelfHostDeployer log the status of requested port before it attempts to start the web server. See aspnet/Hosting#993

[mikeharder]

Most Kestrel tests specify port 0 and let the OS dynamically assign a free port. A few Kestrel tests do bind to specific ports, and we use a tiny helper method to choose a free port:

private static int GetNextPort()
{
    using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
    {
        // Let the OS assign the next available port. Unless we cycle through all ports
        // on a test run, the OS will always increment the port number when making these calls.
        // This prevents races in parallel test runs where a test is already bound to
        // a given port, and a new test is able to bind to the same port due to port
        // reuse being enabled by default by the OS.
        socket.Bind(new IPEndPoint(IPAddress.Loopback, 0));
        return ((IPEndPoint)socket.LocalEndPoint).Port;
    }
}

https://github.com/aspnet/KestrelHttpServer/blob/dev/test/Microsoft.AspNetCore.Server.Kestrel.FunctionalTests/AddressRegistrationTests.cs#L508

This code could be copied where needed since it's so small and simple, or we could move it into a (source) shared class.

[natemcmaster]

Thanks @mikeharder. I've used this approach in #216 and removed all hard-coded port numbers from tests.

[natemcmaster]

#216