Akka.IO.UdpExt.Manager: OverflowException when sending UDP packets to terminated clients

[motmot80]

Akka.Net Version: v1.4.8
Platform: .NET core 3.1 (windows)

Summary:
The UDP actor is sometimes crashing when sending data to UDP clients which already have been terminated.
Sometimes this is followed by an OverflowException. Sometimes by an ArgumentException. Sometimes I receive empty messages from the UDP actor.

Analysis:
While the error occured, I recorded the udp traffic with wire shark.
The exception is occuring when the server tries to send a UDP packet to a non existing client and then receives an ICMP response "Destination unreachable (Port unreachable)" (Type:3, Code:3). And according to the wire shark trace there was no inbound traffic except the ICMP port unreachable packet. The empty messages I received from the UDP actor are also matching timestamps from the ICMP port unreachable packets.

Hint: The original UDP message itself is included within the ICMP response to the server.

127.0.0.1 127.0.0.1 UDP 44 6666 -> 55023
127.0.0.1 127.0.0.1 ICMP 72 Destination unreachable (Port unreachable) [Afterwards the original UDP message is included]

I am not sure if this is really causing the issue and I know this is far from obvious, but according to the call stack (see exceptions below) and the empty messages received - for whatever reason the listener on the port seems to react to the ICMP unreachable packets including the original UDP packets. Interestingly wire shark does something similar: It's showing the associated ICMP unreachable packets even though I filtered "udp and udp.port == 6666".

Reproduction:

1. Bind the UDP manager to a local port f.e. 6666
2. Start a client listening to a local high port fe. 52023
3. Send a UDP packet to the client 6666 => 52023
4. Kill the client
5. Send another UDP packet to the 6666 => 52023
   5.1 (Wait for an ICMP port unreachable packet)

Expected:
UDP actor is not crashing.

Actual:
UDP actor is crashing sporadically.

ArgumentException:

[ERROR][27.11.2020 21:57:52][Thread 0036][akka://SYSTEMNAME/system/IO-UDP-FF/$a] Provided length [1585734688] of array to copy doesn't fit array length [256000] within given offset [254976] (Parameter 'count')
Cause: System.ArgumentException: Provided length [1585734688] of array to copy doesn't fit array length [256000] within given offset [254976] (Parameter 'count')
   at Akka.IO.ByteString.CopyFrom(Byte[] array, Int32 offset, Int32 count)
   at Akka.IO.UdpListener.DoReceive(SocketAsyncEventArgs e, IActorRef handler)
   at Akka.IO.UdpListener.ReadHandlers(Object message)
   at Akka.IO.UdpListener.<PreStart>b__10_0(Object m)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)

OverflowException:

[ERROR][27.11.2020 22:38:35][Thread 0004][akka://SYSTEMNAME/system/IO-UDP-FF/$a] Arithmetic operation resulted in an overflow.
Cause: System.OverflowException: Arithmetic operation resulted in an overflow.
   at Akka.IO.ByteString.CopyFrom(Byte[] array, Int32 offset, Int32 count)
   at Akka.IO.UdpListener.DoReceive(SocketAsyncEventArgs e, IActorRef handler)
   at Akka.IO.UdpListener.ReadHandlers(Object message)
   at Akka.IO.UdpListener.<PreStart>b__10_0(Object m)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)

After this the port is occupied:

[ERROR][27.11.2020 21:57:52][Thread 0036][akka.tcp://SYSTEMNAME@0.0.0.0:7778/system/IO-UDP-FF/$a] Failed to bind UDP channel to endpoint [0.0.0.0:7777]

I haven't had the chance to debug it yet but I think the implementation should be more resilient against these scenario, so no one can bring it down.

Thanks in advance and best regards
Thomas

[motmot80]

According to RFC1122 this seems to be by design: https://tools.ietf.org/html/rfc1122#page-78

[...]
UDP MUST pass to the application layer all ICMP error
messages that it receives from the IP layer.  Conceptually
at least, this may be accomplished with an upcall to the
ERROR_REPORT routine (see Section 4.2.4.1).
[...]

If i had to guess i would assume the Akka.Net UDP implementation is reading ICMP errors as udp messages and that is why I got random buffer issues.

So the Akka.Net UDP implementation should be aware of this.

[motmot80]

Looking at the code from Akka.IO.UdpListener maybe it is just necessary to check the SocketError property.

akka.net/src/core/Akka/IO/UdpListener.cs

Line 117 in 0948525

private void DoReceive(SocketAsyncEventArgs e, IActorRef handler)

if (e.SocketError != SocketError.Success) 
{
    // maybe tell someone an error occured
    // handler.Tell(new ReceivedError(e.SocketError, e.RemoteEndPoint));
    return;
}
handler.Tell(new Received(ByteString.CopyFrom(e.Buffer, e.Offset, e.BytesTransferred), e.RemoteEndPoint));

[Aaronontheweb]

closed via #4643 - going to roll this into Akka.NET v1.4.13