Spike: SocketException: Address already in use when running on Ubuntu

[gaviriar]

Hi There,

I was testing the Spike sample application on Linux, the code builds but when I try to run it I get the following exception:

Multicast DNS spike
IP address 192.168.1.108
IP address fe80::913d:56ef:385b:60a3%3
8/15/18 6:26:18 PM [DEBUG] Makaretu.Dns.MulticastService - Found nic 'wlp1s0'.

Unhandled Exception: discovered NIC 'wlp1s0'
System.Net.Sockets.SocketException: Address already in use
  at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
  at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
  at System.Net.Sockets.Socket.Bind(EndPoint localEP)
  at Makaretu.Dns.MulticastService.Listener() in /home/unitelabs/Desktop/unitelabs/local/net-mdns/src/MulticastService.cs:line 524
  at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
  at System.Threading.ThreadPoolWorkQueue.Dispatch()

Built On

OS: Ubuntu 18.04
cli version: .NET Command Line Tools (2.1.302)
dotnet version: netcoreapp2.0

Just some additional information, it seems that the exception is thrown here when trying to bind to the MultiCast port 5353. When checking the list of processes that own port 5353 sudo lsof -i :5353 I get the following output:

COMMAND    PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
avahi-dae 1138     avahi   12u  IPv4   1695      0t0  UDP *:mdns 
avahi-dae 1138     avahi   13u  IPv6   1696      0t0  UDP *:mdns 
chrome    2988 unitelabs  120u  IPv4  84340      0t0  UDP *:mdns 
chrome    2988 unitelabs  157u  IPv6  84341      0t0  UDP *:mdns 

Any clues or suggestions why this might be the case? Secondly, do you plan to provide support on linux for this library?

Regards,
Ricardo

[richardschneider]

Thanks for the detailed bug report.

Yes, *nix is supported by the package. CircleCI and Travis are used for CI *nix tests.

I'm writing a test to show the reported failure.

[richardschneider]

It looks like avahi grabs the port in exclusive mode and doesn't let other applications snoop mDNS traffic. See the bug report at https://bugs.launchpad.net/ubuntu/+source/avahi/+bug/1321663

Could you stop the daemon and try again?

[gaviriar]

Hi @richardschneider thanks for your swift reply. I figured it would be an issue related to either avahi or chrome. After some testing can safely conclude that the issue is not just with avahi but also happens when only chrome is bound to the multicast port.

It looks like avahi grabs the port in exclusive mode and doesn't let other applications snoop mDNS traffic

If the comment above was true, then chrome would not be able to bound to the port either, but it is isnt it?

Could it be possible that the issue has to do with the .NET socket not actually setting the SO_REUSEADDR flag when setting the SocketOptionName.ReuseAddress option?

[richardschneider]

You could be right. I seem to remember some issues with .Net sockets and socket options.

[gaviriar]

I see, do you happen to have any clues on how we could resolve it?

[richardschneider]

@gaviriar Take a look at this code.

It appears that SO_REUSEPORT is set but NOT SO_REUSEADDR when setting ReuseAddress.

I have no idea how to fix this, except to report a bug.

For some background info see https://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t

[QTimort]

@richardschneider you can have a look into this, might be usefull: https://github.com/QTimort/bind-reuse-port

[richardschneider]

Thanks @QTimort it does look interesting.

Its been a few years (may a decade or so) since I worked with C and *nix, so please be forgiving with these newbie/oldie questions.

• Can you provide a single libbindreuseport.so that is usable on all Linux hosts?
• Or do we need a separate .so for each architecture (intel 32, 64, arm, ....)
• If number 2, how does this play with Nuget?

[QTimort]

@richardschneider

• Or do we need a separate .so for each architecture (intel 32, 64, arm, ....)
  Yep
• If number 2, how does this play with Nuget?
  You should provide the .so files and then use the good one depending of the architecture

[gaviriar]

Hi @richardschneider, just a quick update on some testing that has been done (mainly by @QTimort ).

It seems to be that this issue does not occur if you happen to have the .NET core 2.0 installation on your target Ubuntu machine.

Whereas when using a .NET core 2.1 installation this issue persists. Therefore there must be some difference between the two versions which causes this on 2.1 :(

Do you happen to know if this will be taken care of in any later releases .NET core?

[QTimort]

Hey @richardschneider this should be fixed by dotnet/corefx#32046

[richardschneider]

@QTimort thanks for the information. I guess we must wait for .NET Core 2.2 to verify everything is working.

[cerna]

Testing this issue further I discovered that the error 'Address already in use':

Unhandled Exception: System.Net.Sockets.SocketException: Address already in use
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Makaretu.Dns.MulticastClient..ctor(IPEndPoint multicastEndpoint, IEnumerable`1 nics)
   at Makaretu.Dns.MulticastService.FindNetworkInterfaces()
   at ZeroconfConsole3.Program.Main(String[] args) in /home/mechan/Stažené/ZeroconfConsole3/Program.cs:line 16

is still present in .NET Core 2.2 (hovewer this problem is prevalent in all project of similar nature), but the code is running on .Net Core 3.0 Preview. (Not sure the functionality is 100%.) Can someone confirm?

Tested on Debian Stretch (up to date) with rt-preempt and avahi running.

BTW:
Looking at the latest pull #41 "CI work", there was deletion of call to Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ExclusiveAddressUse, false); whis is unsupported in .Net Core 2.0, so nuget should work without hacking on v. 2.0.

Edit:
And well, running on 3.0 Preview is problematic (it's preview), because after some time there is exception:

Unhandled Exception: System.Threading.Tasks.TaskSchedulerException: An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Thread.StartInternal()
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
   at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Delegate action, Object state, CancellationToken cancellationToken, TaskScheduler scheduler, TaskCreationOptions options, InternalTaskOptions internalOptions)
   at System.Net.NetworkInformation.NetworkChange.CreateSocket()
   at System.Net.NetworkInformation.NetworkChange.add_NetworkAddressChanged(NetworkAddressChangedEventHandler value)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.NetworkInformation.NetworkChange.OnAddressChanged()
   at System.Net.NetworkInformation.NetworkChange.OnSocketEvent(NetworkChangeKind kind)
   at System.Net.NetworkInformation.NetworkChange.ProcessEvent(Int32 socket, NetworkChangeKind kind)
   at Interop.Sys.ReadEvents(Int32 socket, NetworkChangeEvent onNetworkChange)
   at System.Net.NetworkInformation.NetworkChange.LoopReadSocket(Int32 socket)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)

[richardschneider]

@cerna wow, thanks for the detailed analysis!

I was hoping that the next release of .Net Core would fix this.

Looking at the latest pull #41 "CI work", there was deletion of call to Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ExclusiveAddressUse, false); whis is unsupported in .Net Core 2.0, so nuget should work without hacking on v. 2.0.

Yes, PR #40 introduced a bug that make the Circile CI tests fail for .Net Core 1.1. These tests are run on a Debian GNU platform. In the previous code the setting of the socket was only done when running on Windows.

[richardschneider]

A new library allows setting raw socket values. Maybe it will help. https://github.com/tmds/Tmds.LibC

[richardschneider]

Some Linux .Net Core implementations do not correctly set the SO_REUSEADDR socket option. This is needed so that other apps, such as avahi or chrome, can also do multicast DNS.

Tmds.LibC is used to set the bit.