Disposing Socket then rebinding fails with SocketError.AddressAlreadyInUse on Unix

[halter73]

.NET Core 2.0.0 runtime
macOS 10.12.6 Seirra and Ubuntu 16.04.2 LTS (Xenial Xerus)

using System;
using System.Net;
using System.Net.Sockets;

namespace SocketRebind
{
    class Program
    {
        static void Main(string[] args)
        {
            for (var i = 0;; i++)
            {
                Console.WriteLine("Iteration #{0}", i);

                var serverSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
                serverSocket.Bind(new IPEndPoint(IPAddress.Loopback, 5000));
                serverSocket.Listen(512);

                var clientSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
                clientSocket.Connect(IPAddress.Loopback, 5000);

                var acceptSocket = serverSocket.Accept();

                acceptSocket.Dispose();
                clientSocket.Dispose();
                serverSocket.Dispose();
            }
        }
    }
}

Expected behavior:

Program should iterate indefinitely. This is what happens on Windows.

Actual behavior:

After the first iteration, Socket.Bind throws "SocketException: Address already in use"

Output (identical on macOS and Ubuntu):

$ dotnet run
Iteration #0
Iteration dotnet/corefx#1

Unhandled Exception: System.Net.Sockets.SocketException: Address already in use
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at SocketRebind.Program.Main(String[] args) in /Users/shalter/source/halter73/SocketRebind/Program.cs:line 16
$
$
$ dotnet run
Iteration #0

Unhandled Exception: System.Net.Sockets.SocketException: Address already in use
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at SocketRebind.Program.Main(String[] args) in /Users/shalter/source/halter73/SocketRebind/Program.cs:line 16

[stephentoub]

I assume if you add:

serverSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, 1);

before the Bind call, it then works fine?

Related:
https://github.com/dotnet/corefx/issues/11292
dotnet/corefx#11509

[tmds]

serverSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, 1);

👍

For me, It's strange Windows allows to bind during TIME_WAIT.

[halter73]

Thanks for the suggestion. The ReuseAddress option successfully allowed rebinding on macOS and Linux. I think we should consider making the option the default for more consistency between Windows and Unix.

@tmds What problems might arise binding to an endpoint with a socket in the TIME_WAIT state?

[stephentoub]

I think we should consider making the option the default for more consistency between Windows and Unix.

Unfortunately that doesn't purely bring consistency as it leads to other issues, e.g. you probably don't want this code succeeding, but it does:

using System;
using System.Net;
using System.Net.Sockets;

namespace SocketRebind
{
    class Program
    {
        static void Main(string[] args)
        {
            for (var i = 0;; i++)
            {
                Console.WriteLine("Iteration #{0}", i);

                var serverSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
                serverSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, 1);
                serverSocket.Bind(new IPEndPoint(IPAddress.Loopback, 5000));
                serverSocket.Listen(512);

                var serverSocket2 = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
                serverSocket2.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, 1);
                serverSocket2.Bind(new IPEndPoint(IPAddress.Loopback, 5000));
                serverSocket2.Listen(512);

                serverSocket2.Dispose();
                serverSocket.Dispose();
            }
        }
    }
}

As the thorough write-up at https://stackoverflow.com/a/14388707 states, "Welcome to the wonderful world of portability... or rather the lack of it."

[halter73]

Thanks @stephentoub. It's true that I don't want to allow two sockets to bind to the same endpoint simultaneously.

I'm open to suggestions on how to actually make the Unix behavior more similar to Windows. With libuv, we are able to immediate rebind to a port without resorting to using SO_REUSEADDR. I'm still reading through the write-up. Maybe that will give me some ideas.

[tmds]

@tmds What problems might arise binding to an endpoint with a socket in the TIME_WAIT state?

@halter73 If you are using the port for a TCP listen socket, it's appropriate to set the option (per https://stackoverflow.com/questions/6960219/why-not-using-so-reuseaddr-on-unix-tcp-ip-servers).
The state is to allow some lost/duplicate packets to be delivered to the 'right' socket and not to a 'new'/'no' socket.
I am used to setting this on TCP listen sockets. I have never set this on non-listen sockets or considered the implications of setting it on all sockets.

Unfortunately that doesn't bring consistency as it leads to other issues, e.g. you probably don't want this code succeeding, but it does:

@stephentoub I believe this is because corefx is setting both SO_REUSEADDR and SO_REUSEPORT. I think this would not be possible if SO_REUSEPORT was not set (not 100% sure). Perhaps it would be better to not set SO_REUSEPORT.
@halter73 the example applies even to multiple processes. If you spin up Kestrel a couple of times when setting ReuseAddress. On Linux, the kernel will load-balances across those processes.

[tmds]

With libuv, we are able to immediate rebind to a port without resorting to using SO_REUSEADDR.

I guess libuv is setting SO_REUSEADDR and not SO_REUSEPORT.

[stephentoub]

With libuv, we are able to immediate rebind to a port without resorting to using SO_REUSEADDR

That's likely because libuv sets SO_REUSEADDR:
https://github.com/libuv/libuv/blob/1a96fe33343f82721ba8bc93adb5a67ddcf70ec4/src/unix/tcp.c#L107

[halter73]

Interesting. Is there no way to set SO_REUSEADDR and not SO_REUSEPORT with the managed Socket API?

[stephentoub]

Perhaps it would be better to not set SO_REUSEPORT.

The reasoning behind this is outlined at https://github.com/dotnet/corefx/issues/11292#issuecomment-243911958 and dotnet/corefx#11509. If this is going to be changed, someone would need to do a thorough investigation to validate it's actually making things better, not worse.

[stephentoub]

I believe this is because corefx is setting both SO_REUSEADDR and SO_REUSEPORT

@tmds, the example I sent behaves the same on Windows, where there is no PAL in the middle.

[halter73]

If it is too breaking to stop setting SO_REUSEPORT when setting SocketOptionName.ReuseAddress, I would appreciate having some way to set SO_REUSEADDR by itself so the Kestrel Sockets transport can behave more similarly to the Libuv transport on Unix.

[stephentoub]

I would appreciate having some way to set SO_REUSEADDR by itself so the Kestrel Sockets transport can behave more similarly to the Libuv transport on Unix.

You could always fall back to a P/Invoke to setsockopt, but that's far from ideal.

I don't think we want to change how SocketOptionName.ReuseAddress is implemented, as it's explicitly implemented as it is to maintain parity with Windows when it's set to true.

Someone could look into what the impact would be of setting SO_REUSEADDR (not SO_REUSEPORT) in the unix PAL when a socket is created, but I don't know what kind fall out that would have, and it's possible there could be variations between unix platforms such that we'd need/want to only do so in specific cases.

[tmds]

@tmds, the example I sent behaves the same on Windows, where there is no PAL in the middle.

The behavior of the sockets is different. When ExclusiveAddressUse is false, one server becomes unavailable (vs load-balance to both on Linux). When ExclusiveAddressUse is true, setting ReuseAddress will throw.

I guess @halter73 may want to set ExclusiveAddressUse on Windows.

Someone could look into what the impact would be of setting SO_REUSEADDR (not SO_REUSEPORT) in the unix PAL when a socket is created, but I don't know what kind fall out that would have, and it's possible there could be variations between unix platforms such that we'd need/want to only do so in specific cases.

I think, as a minimum, we should not set SO_REUSEPORT on Linux. The load balancing is not expected behavior.

[tmds]

Someone could look into what the impact would be of setting SO_REUSEADDR (not SO_REUSEPORT) in the unix PAL when a socket is created, but I don't know what kind fall out that would have, and it's possible there could be variations between unix platforms such that we'd need/want to only do so in specific cases.

@stephentoub I've been exploring the wonderful world of portability...

SO_REUSEADDR en SO_REUSEPORT control 4 features on Windows and Unix platforms.

On Windows, default behavior of a new Socket is to (1):

• TCP/UDP: allow different addresses to bind to the same port (e.g. 0.0.0.0:5000 and 127.0.0.1:5000)
• TCP: allow binding during TCP_WAIT

To have the same behavior on Unix, you need to set SO_REUSEADDR.

However, when setting this on Unix. It also means multiple sockets can receive the same multicast traffic (2).
On Windows, to allow multiple sockets to receive the same multicast traffic, one needs to set SocketOptionName.ReuseAddress.

Setting this, also means on Windows, you can now bind to exactly the same address on TCP and UDP sockets (stealing the port (3)).

Linux supports enabling/disabling multicast sharing via SO_REUSEADDR. I believe this is also the case for BSD Unix(*).

Port stealing is not supported on Linux (SO_REUSEPORT does loadbalancing (4)), it may be a feature of BSD Unix (using SO_REUSEPORT*).
To be allowed to steal the port on Windows, the socket you are stealing from must have set SO_EXCLUSIVEADDRUSE to false. In .NET this is Socket.ExclusiveAddressUse.

So:

• We can set SO_REUSEADDR for all UDP&TCP sockets or only for TCP sockets. If we set it for UDP too, UDP sockets allow (as on Windows) to bind to same port on different addresses. But they also enable multicast sharing by default (which is uncommon on all platforms).
• (as implemented) SocketOptionName.ReuseAddress needs to be wired to SO_REUSEADDR to enable/disable multicast sharing.
• SocketOptionName.ReuseAddress must not be wired to SO_REUSEPORT on Linux, to avoid unexpected kernel load balancing.
• If port stealing isn't an .NET Core feature, we may get rid of SO_REUSEPORT on all Unix. If it is .NET Core feature, it makes sense to tie it to Socket.ExclusiveAddressUse=false somehow.

My preference would be to:

• Set SO_REUSEADDR to true for TCP sockets on creation
• Control SO_REUSEADDR using ReuseAddress.
• Not consider port stealing a .NET Core feature

Off course, it needs to be validated if this all holds up on different platforms.

*: I don't have a BSD Unix derivative, the most interesting doc I found on this is: http://www.unixguide.net/network/socketfaq/4.11.shtml.

[stephentoub]

cc: @wfurt, @geoffkizer, @Priya91

[tmds]

I missed something when reading http://www.unixguide.net/network/socketfaq/4.11.shtml.

But when binding a multicast address (its main use), SO_REUSEADDR is
considered identical to SO_REUSEPORT (p. 731, "TCP/IP Illustrated,
Volume 2"). So for portability of multicasting applications I always
use SO_REUSEADDR.

There is also the non-portable way which is to bind to the any address (or an interface address). So to enable sharing SO_REUSEPORT is needed. We can limit enabling undesired features (in particular Linux kernel TCP load balancing) by only setting this for UDP/Dgram sockets.
I'm not sure Windows supports what is called here the portable way. The bind documentation explicitly states binding to the any/an interface address for multicast. So most existing code may be using the non-portable way

[tmds]

Control SO_REUSEADDR using ReuseAddress.

Actually we are better off not controlling SO_REUSEADDR for TCP as it may still introduce unwanted xplat differences.

So, updated proposal:

On TCP Bind set SO_REUSEADDR.
For TCP SetSocketOption SocketAddress is a fake toggle. It would be appropriate to throw PNSE when setting to true, but that would definitely be breaking existing code.

This matches with libuv's behavior for tcp and udp.

[tmds]

Codifying the problem:

var serverSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
serverSocket.Bind(new IPEndPoint(IPAddress.Loopback, 5000));
serverSocket.Listen(512);

Now, we see what @halter73 observed, on unix this throws AddressAlreadyInUse when restarting our server.
Attempt to fix:

serverSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, 1);

Now, it works on Linux. But on Windows, ReuseAddress means we'd like to steal the address.
Stealing is not polite. Let's ensure no one can steal from us:

serverSocket.ExclusiveAddressUse = true;

This is the default on Linux, on Windows it makes ReuseAddress throw.
Don't treat others in ways you would not like to be treated...
Only setting ReuseAddress on Unix then:

if (OSPlatform.CurrentPlatform.IsUnix)
{
   serverSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, 1);
}

Are we there yet?

No, on Unix we are now actually stealing/allowing others to steal our port (BSD-Unix) or loadbalancing the port with others (Linux).
This is due to SocketOptionName.ReuseAddress not only setting SO_REUSEADDR, but SO_REUSEPORT are well .

Suggested change:

For TCP Sockets:

• don't set SO_REUSEADDR and SO_REUSEPORT with SocketOptionName.ReuseAddress
• on the Bind call, set SO_REUSEADDR to 1.

Then this code works on all platforms:

var serverSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
// optional: serverSocket.ExclusiveAddressUse = true;
serverSocket.Bind(new IPEndPoint(IPAddress.Loopback, 5000));
serverSocket.Listen(512);

Open question:

What should SocketOptionName.ReuseAddress do for a TCP socket?

• throw PNSE when set to true, return false when retrieved
• ignore the value and pretend everything went well when set, return false when retrieved
• store the value but don't use it, so a retrieved value matches a set value

@wfurt @geoffkizer @Priya91 @halter73 @stephentoub your thoughts?

[halter73]

@tmds If SO_REUSEADDR is set for TCP sockets when binding on Unix already, could SocketOptionName.ReuseAddress be mapped exclusively to SO_REUSEPORT on Unix?

This seems to align the behavior of Windows and Linux a little more. I guess the behavior is never going to align exactly considering the port stealing vs load balancing behavior on BSD and Linux respectively.

Another problem I see with this idea is that there would no longer be a managed API on Unix to set SO_REUSEADDR, but I don't know how common/important that option is for non-TCP and/or non-listen sockets.

[tmds]

could SocketOptionName.ReuseAddress be mapped exclusively to SO_REUSEPORT on Unix?

To look at how xplat .NET core needs to behave, Windows is the reference (most existing code).
Linux does load balancing.
To expose load balancing (if requested), it would be best to add an additional socket option e.g. SocketOptionName.LoadBalancing.
For BSD Unix, we could still set SO_REUSEPORT, if the feature is requested and the semantics really match with those of Windows.

Another problem I see with this idea is that there would no longer be a managed API on Unix to set SO_REUSEADDR, but I don't know how common/important that option is for non-TCP and/or non-listen sockets.

For non-TCP I'd keep the current behavior. This is needed to make UDP multicast work xplat.

[tmds]

@wfurt @geoffkizer @Priya91 @halter73 @stephentoub I will look at implementing the proposed changes next week and do a PR.

[karelz]

@tmds any progress on this one?

[vgolynets]

I agree with @tmds, it would be nice to have this fix in some next 2.0.x update. Now our code looks like:

       public static void Main(string[] args)
        {
            var serverUrl = "http://*:31000/";
            Console.WriteLine("Server host {0}", serverUrl);

            var closing = new AutoResetEvent(false);
            Console.CancelKeyPress += (sender, ev) =>
            {
                closing.Set();
                ev.Cancel = true;
            };

            try
            {
                using (WebApp.Start(serverUrl, OsaApplication.Initialize))
                {
                    Console.WriteLine("Server started at {0}", DateTime.UtcNow);
                    closing.WaitOne();
                }
            }
            catch (HttpListenerException)
            {
                // This catch section is just a workaround for .Net Core issue
                Console.WriteLine("Oops, we catched https://github.com/dotnet/corefx/issues/25016 bug");
            }
            Console.WriteLine("Server stopped at {0}", DateTime.UtcNow);
        }

Do you believe it's acceptable solution? IMO no, looks ugly.

[EDIT] Add C# syntax highlighing by @karelz

[karelz]

Do you believe it's acceptable solution? IMO no, looks ugly.

"Ugly", but working workaround & affecting a few developers is not a good reason for porting a bug fix into servicing branch. Otherwise we would make servicing branch the new master :)
The key questions for releasing a "hot fix" in servicing branch are:

• Is there a workaround?
• How many customers are affected?

If you tell me that you cannot work around the problem at all, or that the number of developers affected is more than just few, then I am interested in more details.

[tmds]

Is there a workaround?

In most cases there isn't because the Socket is not accessible to the developer.
e.g. an external dev cannot workaround this in HttpListener/Kestrel, because he does not have access to the Socket.

How many customers are affected?

This affects every TCP server. Unless the Socket code gets some 'extra' that isn't needed on Windows, it will fail to restart in the 2 minutes after being stopped.

[karelz]

e.g. an external dev cannot workaround this in HttpListener/Kestrel, because he does not have access to the Socket.

Kestrel should not hit this issue as it does not use Sockets in 2.0 - see https://github.com/dotnet/corefx/issues/24562#issuecomment-350358401.
HttpListener may be a good point (I guess it uses Sockets under the hood). Given it is obsoleted API, I'd like to see couple of examples of directly affected customers before we service it -- so far I saw hints of one in the replies above: https://github.com/dotnet/corefx/issues/24562#issuecomment-350667917.

[tmds]

HttpListener may be a good point (I guess it uses Sockets under the hood).

As does every other .NET TCP server (except Kestrel).

[karelz]

As does every other .NET TCP server (except Kestrel).

What are they? How popular they are? How many customers have you seen using them? (I haven't seen any so far)

[MGAndreasen]

Port stealing might be desired by someone, so atleast it should be an option to be enabled.

I use ansi c on linux for a few things where saving connections info from socket to a file, then dropping socket in one process, then exec a new version of the process, rebinding the socket in the new process loading the connection info from the old process, then closing the old process.

[MGAndreasen]

like keeping already connections linger, and continue processing on the new process, without anyone knowing ;)

[tmds]

@Mga-Denmark port stealing is still possible by setting ReuseAddress.

[joshgarnett]

@karelz we just ported over from Mono 4.8 to .NET Core 2 and are hitting this issue. In some cases we've been able to specify the Reuse Address socket option, but we are also using HttpListener. All it takes to hit is restarting the server. The new process will fail to start for about a minute.

[halter73]

@joshgarnett This should be fixed with .NET Core 2.1. If you need a workaround for now, it's pretty ugly, but here it is. Just call EnableRebinding(socket) prior to socket.Bind(...).

[joshgarnett]

@halter73 how's that work with something like HttpListener where the underlying socket isn't exposed?

[halter73]

@joshgarnett I'm not sure if there's a workaround for HttpListener. For the times you have direct access to the managed socket, you can call EnableRebinding on said socket and the subsequent bind will succeed even if you just disposed an HttpListener on the same port.

HttpListener might still fail to bind to a port used by a recently-disposed managed socket even if the recently-disposed socket used the EnableRebinding workaround.

In other words:

There is one final thing you should know about SO_REUSEADDR. Everything written above will work as long as the socket you want to bind to has address reuse enabled. It is not necessary that the other socket, the one which is already bound or is in a TIME_WAIT state, also had this flag set when it was bound. The code that decides if the bind will succeed or fail only inspects the SO_REUSEADDR flag of the socket fed into the bind() call, for all other sockets inspected, this flag is not even looked at.

https://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t/14388707#14388707

The entire SO answer (which was already linked to in a comment above) is a good read if you're interested.

[keith-miller]

Is there a fix for using HttpListener on Linux for this yet? We are using Net Core 2.1.3 currently. Thanks!

[stephentoub]

We are using Net Core 2.1.3

That would be the sdk version. What version of the runtime are you using? What's the target framework in your csproj?

[keith-miller]

Sorry about that, we are using <TargetFramework>netcoreapp2.0</TargetFramework> and I'm guessing that should be <TargetFramework>netcoreapp2.1</TargetFramework>?

[karelz]

Correct

[stephentoub]

Yup. Please give it a try and let us know whether the problem is addressed. Thanks.

[keith-miller]

I don't see a dotnet-sdk-2.1.302 package in the Ubuntu Microsoft repository, which is latest on Windows. There is a dotnet-sdk-2.1.4 package, but it doesn't support netcoreapp2.1:

/usr/share/dotnet/sdk/2.1.4/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.TargetFrameworkInference.targets(135,5): error : The current .NET SDK does not support targeting .NET Core 2.1. Either target .NET Core 2.0 or lower, or use a version of the .NET SDK that supports .NET Core 2.1.

Looks like those packages are still using the Net Core 2.0.5 runtimes and not 2.1.2.

[joshgarnett]

@doyouevensunbro I noticed that the debian/ubuntu repository is missing the latest SDK version yesterday. I adapted the debian dockerfile to work on Ubuntu Xenial.

# Install .NET CLI dependencies
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        libc6 \
        libgcc1 \
        libgssapi-krb5-2 \
        libicu55 \
        liblttng-ust0 \
        libssl1.0.2 \
        libstdc++6 \
        zlib1g \
    && rm -rf /var/lib/apt/lists/*

# Install .NET Core SDK
ENV DOTNET_SDK_VERSION 2.1.302

RUN curl -SL --output dotnet.tar.gz https://dotnetcli.blob.core.windows.net/dotnet/Sdk/$DOTNET_SDK_VERSION/dotnet-sdk-$DOTNET_SDK_VERSION-linux-x64.tar.gz \
    && dotnet_sha512='2166986e360f1c3456a33723edb80349e6ede115be04a6331bfbfd0f412494684d174a0cfb21d2feb00d509ce342030160a4b5b445e393ad83bedb613a64bc66' \
    && sha512sum dotnet.tar.gz \
    && echo "$dotnet_sha512 dotnet.tar.gz" | sha512sum -c - \
    && mkdir -p /usr/share/dotnet \
    && tar -zxf dotnet.tar.gz -C /usr/share/dotnet \
    && rm dotnet.tar.gz \
    && ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet

# Configure Kestrel web server to bind to port 80 when present
ENV ASPNETCORE_URLS=http://+:80 \
    # Enable detection of running in a container
    DOTNET_RUNNING_IN_CONTAINER=true \
    # Enable correct mode for dotnet watch (only mode supported in a container)
    DOTNET_USE_POLLING_FILE_WATCHER=true \
    # Skip extraction of XML docs - generally not useful within an image/container - helps perfomance
    NUGET_XMLDOC_MODE=skip

# Trigger first run experience by running arbitrary cmd to populate local package cache
RUN dotnet help

[keith-miller]

@joshgarnett Thanks! Will give that a try today.

[keith-miller]

Ended up just grabbing the 2.1.302 tarball and manually installing it, and I'm happy to say it fixed the HttpListener socket issue. Thanks!

[stephentoub]

Thanks for confirming.