UdpClient inconsistent behaviour between Windows and Linux.

[philiphenriksen]

Description

When binding a UdpClient to some specific IPEndPoint, it acts differently on Windows and Linux;
On Windows, the client receives both broadcast and unicast messages.
On Linux, the client only receives unicast messages.

Reproduction Steps

Create two dlls; one for a Windows machine and one for a Linux machine.
Replace the values of thisDeviceIp, otherDeviceIp and broadcastIp with ones appropriate for the devices.
Both devices should be able to communicate with one another and share broadcast IP.

static void Main(string[] args)
{
   string thisDeviceIp = "10.0.0.1"; // 10.0.0.1 is a Windows host.
   string otherDeviceIp = "10.0.0.2"; // 10.0.0.2 is a Linux host.
   string broadcastIp = "10.0.0.255";
   string interfaceName = "eno1" // Name of network interface with IP "thisDeviceIp" on the Linux host

    UdpClient udpClient = new()
    {
        EnableBroadcast = true
    };
    if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
    {
        // Attempt to set the socket option "SO_BINDTODEVICE" on Linux.
        udpClient.Client.SetRawSocketOption(1, 25, Encoding.UTF8.GetBytes(interfaceName));
    }
    client.Client.Bind(new IPEndPoint(IPAddress.Parse(thisDeviceIp), 0xA000));

    IPEndPoint unicastAdddress = new(IPAddress.Parse(otherDeviceIp), 0xA000);
    IPEndPoint broadcastAddress = new(IPAddress.Parse(broadcastIp), 0xA000);
    _ = Task.Factory.StartNew(() => Listener(udpClient));

   // Sender logic
    while (true)
    {
        ConsoleKeyInfo f = Console.ReadKey(true);
        if (f.Modifiers.HasFlag(ConsoleModifiers.Control) && f.Key == ConsoleKey.C)
        {
            return;
        }

        if (f.Key == ConsoleKey.A)
        {
            udpClient.Send(Encoding.UTF8.GetBytes("data"), unicastAdddress);
        }
        else if (f.Key == ConsoleKey.B)
        {
            udpClient.Send(Encoding.UTF8.GetBytes("data"), broadcastAddress);
        }
    }
}

static void Listener(UdpClient udpClient)
{
    int messages = 0;
    IPEndPoint? remoteHost = null;

    // Receiver logic.
    while (true)
    {
        udpClient.Receive(ref remoteHost);
        if (remoteHost != null)
        {
            Console.WriteLine($"Received message from {remoteHost} ({messages++})");
        }
    }
}

Expected behavior

Pressing A on one device should show up as Received message from {ip} on the other host (unicast).
Pressing B on one device should show up as Received message from {ip} on both hosts (broadcast).

Actual behavior

Pressing A on one device correctly displays a message in the other.
Pressing B, however, only shows messages on the Windows host.

Regression?

No response

Known Workarounds

Bind to IPAddress.Any instead of a specific IP.
Not preferable, since both computers may have multiple network adapters with different IP addresses and using IPAdress.Any seems to pick semi-randomly.

Configuration

.NET version: Microsoft.NETCore.App 6.0.13
OS:

• Windows host: Windows 10 x64
• Linux host: Ubuntu 22.04.1 LTS (GNU/Linux 5.19.0-32-generic x86_64)

This issue does not seem to be specific to the specified Linux version, as this has also been tested with ubuntu 18.04 and 20.04.

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

When binding a UdpClient to some specific IPEndPoint, it acts differently on Windows and Linux;
On Windows, the client receives both broadcast and unicast messages.
On Linux, the client only receives unicast messages.

Reproduction Steps

Create two dlls; one for each machine.
Replace the values of thisDeviceIp, otherDeviceIp and broadcastIp with ones appropriate for the devices.
Both devices should be able to communicate with one another and share broadcast IP.

static void Main(string[] args)
{
   string thisDeviceIp = "10.0.0.1"; // 10.0.0.1 is a Windows host.
   string otherDeviceIp = "10.0.0.2"; // 10.0.0.2 is a Linux host.
   string broadcastIp = "10.0.0.255";

    UdpClient udpClient = new(new IPEndPoint(IPAddress.Parse(thisDeviceIp), 0xA000))
    {
        EnableBroadcast = true
    };

    IPEndPoint unicastAdddress = new(IPAddress.Parse(otherDeviceIp), 0xA000);
    IPEndPoint broadcastAddress = new(IPAddress.Parse(broadcastIp), 0xA000);
    _ = Task.Factory.StartNew(() => Listener(udpClient));

   // Sender logic
    while (true)
    {
        ConsoleKeyInfo f = Console.ReadKey(true);
        if (f.Modifiers.HasFlag(ConsoleModifiers.Control) && f.Key == ConsoleKey.C)
        {
            return;
        }

        if (f.Key == ConsoleKey.A)
        {
            udpClient.Send(Encoding.UTF8.GetBytes("data"), unicastAdddress);
        }
        else if (f.Key == ConsoleKey.B)
        {
            udpClient.Send(Encoding.UTF8.GetBytes("data"), broadcastAddress);
        }
    }
}

static void Listener(UdpClient udpClient)
{
    int messages = 0;
    IPEndPoint? remoteHost = null;

    // Receiver logic.
    while (true)
    {
        udpClient.Receive(ref remoteHost);
        if (remoteHost != null)
        {
            Console.WriteLine($"Received message from {remoteHost} ({messages++})");
        }
    }
}

Expected behavior

Pressing A on one device should show up as Received message from {ip} on the other host (unicast).
Pressing B on one device should show up as Received message from {ip} on both hosts (broadcast).

Actual behavior

Pressing A on one device correctly displays a message in the other.
Pressing B, however, only shows messages on the Windows host.

Regression?

No response

Known Workarounds

Bind to IPAddress.Any instead of a specific IP.
Not preferable, since both computers may have multiple network adapters with different IP addresses and using IPAdress.Any seems to pick semi-randomly.

Configuration

.NET version: Microsoft.NETCore.App 6.0.13
OS:

• Windows host: Windows 10 x64
• Linux host: Ubuntu 22.04.1 LTS (GNU/Linux 5.19.0-32-generic x86_64)

This issue does not seem to be specific to the specified Linux version, as this has also been tested with ubuntu 18.04 and 20.04.

Other information

No response

Author:	philiphenriksen
Assignees:	-
Labels:	area-System.Net
Milestone:	-

[antonfirsov]

This is a kernel behavior, I don't think we can do anything about it. There is a similar issue: #25269 where disabling firewalld helped for OP, however it didn't do the trick in my case.

Binding the Socket/UdpClient to the broadcast address makes it receive the broadcast packets, but then it doesn't receive the unicast ones. I have spent some time searching the internet to understand this kernel behavior, but found no usable results. /cc @tmds

Binding the socket to IPAddress.Any will make it receive both broadcast and unicast packets:

UdpClient udpClient = new(new IPEndPoint(IPAddress.Any, 0xA000))
{
    EnableBroadcast = true
};

@philiphenriksen is this problematic in your case?

[philiphenriksen]

It has been a bit problematic as IPAddress.Any limits us to one instance per port per device.

Not a problem on the Linux machine, as it's mostly used to host a single release-build of the service for benchmarking, but it does cause some issues on Windows hosts where we develop the service.
We have to run multiple instances in this environment; one instance for debugging and another on a different network interface for sending messages to the first one.

This has led to us introducing the following code to separate between Linux and Windows hosts so that the Windows-hosted service only consumes a specific IP address, which works as a workaround:

IPEndpoint localEndpoint = DetermineIPEndpoint();
UdpClient client = new()
{
    EnableBroadcast = true
};
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
    client.Client.Bind(localEndpoint);
}
else
{
    client.Client.Bind(new IPEndPoint(IPAddress.Any, localEndpoint.Port));
}

If nothing else, it would be nice for this behaviour to be documented somewhere.
The problem is mostly this: the fact that UdpClient instances only receive unicast messages on Linux hosts unless you bind to IPAddress.Any, even with EnableBroadcast enabled, is hard to discover.

[tmds]

Here are some things you can try:

since both computers may have multiple network adapters with different IP addresses

Using the SO_BINDTODEVICE option should allow you to set the interface where you want to receive packets.
This option isn't exposed as a SocketOptionName, so you need to use SetRawSocketOption and figure out the proper arguments.

I'm not sure if this also affects out-going packets. If it doesn't SocketOptionName.MulticastInterface may help.

as IPAddress.Any limits us to one instance per port per device.

Try setting SocketOptionName.ReuseAddress.
If you try this and it seems to do what you need, verify that when run two sockets simultaneously that both are still receiving all packets.

I will try some things when I have some time for it.
If you've written some code for these suggestions, please include it with the issue.

[ghost]

This issue has been marked needs-author-action and may be missing some important information.

[philiphenriksen]

@tmds
Okay, so I think I found the integer values for SO_BINDTODEVICE here: https://codebrowser.dev
#define SOL_SOCKET 1
#define SO_BINDTODEVICE 25

Using that, I've tried adding the following code to the Linux host:

UdpClient udpClient = new()
{
    EnableBroadcast = true
};
udpClient.Client.SetRawSocketOption(1, 25, Encoding.UTF8.GetBytes("eno1")); // eno1 is the interface with "thisDeviceIp"
udpClient.Client.Bind(new IPEndPoint(IPAddress.Parse(thisDeviceIp), 0xA000));

Even with this change sending and receiving works just like before, with broadcast messages still being ignored.
The socket option itself seems to run correctly, as specifying an interface that doesn't exist throws an exception:

System.Net.Sockets.SocketException (0xFFFDFFFE): Unknown socket error

Adding ReuseAddress seems to allow multiple instances to start simultaneously, but only one of the instances receives unicast messages until the other is shut down on both Linux and Windows.
Both Windows instances still receive broadcast messages though.

udpClient.Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
udpClient.Client.Bind(new IPEndPoint(IPAddress.Parse(thisDeviceIp), 0xA000));

[tmds]

I should be able to try some things myself next week.

Even with this change sending and receiving works just like before

IPAddress.Parse(thisDeviceIp) should be IPAddress.Any or you can't receive any broadcasts on Linux.

but only one of the instances receives unicast messages until the other is shut down on both Linux and Windows.

Instead of using ReuseAddress, try setting SO_REUSEADDR, and try setting SO_REUSEPORT, but not both at the same time and see if that makes a difference.

[philiphenriksen]

Setting SO_REUSEADDR(2) orSO_REUSEPORT(15) just gives me the following error:

System.Net.Sockets.SocketException (22): Invalid argument

With the following code:

// SO_REUSEADDR
//udpClient.Client.SetRawSocketOption(1, 2, new byte[] { 1 });

// SO_REUSEPORT
udpClient.Client.SetRawSocketOption(1, 15, new byte[] { 1 });

Only one of the lines were active at a time.

[tmds]

System.Net.Sockets.SocketException (22): Invalid argument

The man page says: "Argument is an integer boolean flag.", so you need to use an int.

[philiphenriksen]

Tested with the following instead for both flags, which avoids the invalid argument:

udpClient.Client.SetRawSocketOption(1, 15, MemoryMarshal.AsBytes<int>(new int[] { 1 }));

Still looks like only one instance gets the unicast messages, though.

[tmds]

@philiphenriksen can you try this and see how it works for you?

using System.Net;
using System.Net.NetworkInformation;
using System.Net.Sockets;
using System.Text;

const int SO_BINDTODEVICE = 25;
const int SOL_SOCKET = 1;

int port = 5000;

if (args.Length < 2)
{
    Console.WriteLine($"You must specify two arguments: <interface name> <peer address>.");
    return 1;
}

string networkInterfaceName = args[0];
if (!IPAddress.TryParse(args[1], out IPAddress? peerAddress))
{
    Console.WriteLine($"The peer address is not valid.");
    return 1;
}

NetworkInterface? networkInterface = NetworkInterface.GetAllNetworkInterfaces().FirstOrDefault(interf => interf.Id == networkInterfaceName);
if (networkInterface is null)
{
    Console.WriteLine($"Interface {networkInterfaceName} is not found.");
    return 1;
}
UnicastIPAddressInformation? interfaceAddress = GetIPv4InterfaceAddress(networkInterface);
if (interfaceAddress is null)
{
    Console.WriteLine($"The interface has no associated ip address.");
    return 1;
}

Console.WriteLine($"Receiving packets on {networkInterface.Name}");
Socket receiveSocket = CreateReceiveSocket(networkInterface, port);
// Receive on a thread.
new Thread(Receive).Start(receiveSocket);

Socket sendSocket = CreateSendSocket();
// Send a unicast message
IPEndPoint peerEndpoint = new IPEndPoint(peerAddress, port);
Console.WriteLine($"Send unicast to {peerEndpoint}");
sendSocket.SendTo("Hello unicast"u8, peerEndpoint);
// Send a broadcast message
IPEndPoint broadcastEndpoint = new IPEndPoint(GetBroadcastAddress(interfaceAddress), port);
Console.WriteLine($"Send broadcast to {broadcastEndpoint}");
sendSocket.SendTo("Hello broadcast"u8, broadcastEndpoint);

Thread.Sleep(1000);
Console.WriteLine("Press Ctrl+C to stop the application.");

return 0;

static void Receive(object? state)
{
    Socket receiveSocket = (Socket)state!;
    byte[] receiveBuffer = new byte[1024];
    EndPoint remoteEndPoint = receiveSocket.LocalEndPoint!;
    while (true)
    {
        int receiveLength = receiveSocket.ReceiveFrom(receiveBuffer, ref remoteEndPoint);
        Console.WriteLine($"Receive from {remoteEndPoint}: {Encoding.UTF8.GetString(receiveBuffer.AsSpan(0, receiveLength))}");
    }
}

static Socket CreateReceiveSocket(NetworkInterface interf, int port)
{
    Socket receiveSocket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
    
    // Bind to Any to enable receiving both broadcast as well as unicast packets.
    receiveSocket.Bind(new IPEndPoint(IPAddress.Any, port));

    // Only receive from the interface.
    receiveSocket.SetRawSocketOption(SOL_SOCKET, SO_BINDTODEVICE, Encoding.UTF8.GetBytes(interf.Id));

    return receiveSocket;
}

static Socket CreateSendSocket()
{
    Socket sendSocket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);

    // Allow sending broadcast messages.
    sendSocket.EnableBroadcast = true;

    return sendSocket;
}

UnicastIPAddressInformation? GetIPv4InterfaceAddress(NetworkInterface networkInterface)
    => networkInterface.GetIPProperties().UnicastAddresses.SingleOrDefault(address => address.Address.AddressFamily == AddressFamily.InterNetwork);

static IPAddress GetBroadcastAddress(UnicastIPAddressInformation interfaceAddress)
{
    if (interfaceAddress.Address.AddressFamily == AddressFamily.InterNetworkV6)
    {
        throw new ArgumentException("IPv6 doesn't have broadcast addresses.");
    }

    var addressBytes = interfaceAddress.Address.GetAddressBytes();
    var mask = interfaceAddress.IPv4Mask.GetAddressBytes();

    var broadcastAddress = new byte[addressBytes.Length];
    for (var i = 0; i < broadcastAddress.Length; i++)
    {
        broadcastAddress[i] = (byte)((addressBytes[i] & mask[i]) | (~mask[i]));
    }

    return new IPAddress(broadcastAddress);
}

[philiphenriksen]

This still seems to consume all addresses.

Instance 1 (ethernet adapter "eno1"):

> dotnet test.dll eno1 "10.0.0.15"
Receiving packets on eno1

Instance 2 (wifi adapter "wlp2s0"):

> dotnet test.dll wlp2s0 "10.0.0.15"
Receiving packets on wlp2s0
Unhandled exception. System.Net.Sockets.SocketException (98): Address already in use
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Program.<<Main>$>g__CreateReceiveSocket|0_2(NetworkInterface interf, Int32 port) in <path>\Program.cs:line 96
   at Program.<Main>$(String[] args) in <path>\Program.cs:line 49
Aborted (core dumped)

Moving SetRawSocketOption before Bind avoids the exception but still doesn't allow the second instance to receive broadcasts.

[tmds]

This still seems to consume all addresses.

Try setting SO_REUSEPORT before binding the receive socket.

const int SO_REUSEPORT = 15;
...
receiveSocket.SetRawSocketOption(SOL_SOCKET, SO_REUSEPORT, MemoryMarshal.AsBytes<int>(new int[] { 1 }));
...

You need to verify if you run both apps on the same host, both still receive the expected messages.

Side question: are you required to use the broadcast address? Or can you use multicast instead?

[philiphenriksen]

Now we're back to the last state I managed to reach with my own code; only one instance receives broadcast.

static Socket CreateReceiveSocket(NetworkInterface interf, int port)
{
    Socket receiveSocket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);

    // Only receive from the interface.
    receiveSocket.SetRawSocketOption(SOL_SOCKET, SO_BINDTODEVICE, Encoding.UTF8.GetBytes(interf.Id));
    receiveSocket.SetRawSocketOption(SOL_SOCKET, SO_REUSEPORT, MemoryMarshal.AsBytes<int>(new int[] { 1 }));

    // Bind to Any to enable receiving both broadcast as well as unicast packets.
    receiveSocket.Bind(new IPEndPoint(IPAddress.Any, port));

    return receiveSocket;
}

Instance 1:

> dotnet test.dll eno1 "10.0.0.15"
Receiving packets on eno1
Receive from 10.0.0.15:63259: Hello broadcast

Instance 2:

> dotnet test.dll wlp2s0 "10.0.0.15"
Receiving packets on wlp2s0

After some more testing this may actually be another issue entirely; I don't seem to receive broadcast on the wifi adapter no matter the order I start the services in.
Sending and receiving unicast works, but receiving broadcast doesn't.

As for the side question; my end goal is to create a service that can listen for BACnet messages.
The BACnet protocol uses broadcast for object- and device discovery.
I therefore must be able to listen for broadcast messages to create an appropriate response.

[tmds]

Now we're back to the last state I managed to reach with my own code; only one instance receives broadcast.

The issues with SO_REUSEPORT and SO_REUSEADDR is that they can stop the other socket from receiving packets, either because they steal the endpoint (the other socket no longer receives anything) or they do load balancing (packets are sent to one of the sockets, but not to both).

I would try either option (SO_REUSEPORT and SO_REUSEADDR)

If the sockets aren't both receiving the expected packets, then I think you need to bind these sockets to the interface IP, and have a separate socket that receives broadcast packets (bound to any).

You'll still need to set SO_REUSEADDR to allow the overlapping binds.

I think with that, the broadcast packets will go to the any socket, and the unicast packets will go to the interface ip sockets.

If you start another application that binds the same endpoint, I think that application will probably steal the endpoint.

[philiphenriksen]

It seems I'll have to bind a separate listener to the broadcast address, like you suggested;

Socket 1 bound to IPAddress.Any listens for broadcast messages.
Socket 2 bound directly to the IP-address of the interface listens for and sends unicast messages.

Raw socket options are no longer needed with this setup, as ExclusiveAddressUse seems to work fine.

Something akin to this:

<...>

Console.WriteLine($"Receiving packets on {networkInterface.Name} ({interfaceAddress.Address})");
IPEndPoint receiveEndpoint = new IPEndPoint(interfaceAddress.Address, port);
(Socket unicastSocket, Socket broadcastSocket) = CreateReceiveSockets(receiveEndpoint);

// Receive on a thread.
new Thread(Receive).Start(unicastSocket);
new Thread(Receive).Start(broadcastSocket);

<...>
static (Socket unicastSocket, Socket broadcastSocket) CreateReceiveSockets(IPEndPoint endPoint)
{
    Socket broadcastSocket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
    broadcastSocket.ExclusiveAddressUse = false; // Must be false to avoid SocketException when opening unicast socket.

    // Bind to Any to enable receiving broadcast packets.
    broadcastSocket.Bind(new IPEndPoint(IPAddress.Any, endPoint.Port));

    // Create another socket for receiving unicast packets.
    Socket unicastSocket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
    unicastSocket.ExclusiveAddressUse = false; // Must be false to avoid conflict with broadcast socket.
    unicastSocket.Bind(endPoint);

    return (unicastSocket, broadcastSocket);
}
<...>

Instance 1:

> dotnet test.dll eno1 "10.0.0.15"
Receiving packets on eno1 (10.0.0.10)
Receive from 10.0.0.15:62185: Unicast to 1
Receive from 10.0.0.15:62185: Hello broadcast

Instance 2:

> dotnet test.dll wlp2s0 "10.0.0.15"
Receiving packets on wlp2s0 (10.0.0.11)
Receive from 10.0.0.15:62185: Unicast to 2
Receive from 10.0.0.15:62185: Hello broadcast

10.0.0.15 is a second machine that sends separate unicast packets to each of the instances.
This is to verify that both instances are able to send packets too.

[tmds]

@philiphenriksen I'm happy we have found a solution.

Are we good to close the ticket?

[philiphenriksen]

Yep, my question has been answered and we've found a decent alternative.

[antonfirsov]

Thanks @tmds for chiming in!