Trying to use MSQUIC with unconnected UDP socket to support NAT hole punching

[jacobgorm]

hi,

I would like to use MSQUIC for peer to peer MacOS app, such that after an initial rendezvous through a central server, peers would be able to connect directly using traditional NAT hole punching. I have this working already with my own UDP-based protocol, but would like to add QUIC to get features like TLS and reliable streams. Basically I use just a single datagram socket and sendto() and recvfrom() instead of doing connect() and bind().

The first thing I noticed was that MSQUIC does a connect() no the UDP socket, which will cause incoming packets to get filtered, so I changed the code in src/platform/datapath_kqueue.c to not do the connect, but instead just make a copy of the address & size arguments to connect(), so that I can set them later in the sendmsg() call, like this:

instead of connect():

memcpy(&SocketContext->ConnectedAddress, &MappedAddress.Ip, sizeof(SocketContext->ConnectedAddress));
SocketContext->ConnectedAddressSize = ForceIpv4 ? sizeof(struct sockaddr_in) : sizeof(struct sockaddr_in6);

and before calling sendmsg():

if (!Socket->Connected) {

replaced with:

if (Socket->Connected) {
    Mhdr.msg_name = &SocketContext->ConnectedAddress;
    Mhdr.msg_namelen = SocketContext->ConnectedAddressSize;
} else {

I've also tried emulating the effect of getsockname() by just faking out a port number, but the code seems happy one way or the other.

However, when trying to run a client that is modified like this against an unmodified server, the packets that get received back from the server seem to be getting silently dropped somewhere higher up in MSQUIC. The packets are sent and I get replies back OK, but a connection never gets established.

Does anyone have hints as to what I may be doing wrong here? I would assume that these changes should not affect anything higher up, but clearly I am wrong.

Thanks in advance!
Jacob

[nibanks]

Ideally, we would grab logs to see where the packet is getting dropped, but with macOS our logging story is a bit sad. I believe you might be able to get console logging working, if you set QUIC_LOGGING_TYPE to stdout (or if using PowerShell set LoggingType). Then the logs will show you were the packet is getting dropped.

[jacobgorm]

Thanks @nibanks for getting back to me. In retrospect I should have probably done this work on linux where the logging-situation seems better. Anyway, I still managed to find the problem, which is that the packet gets dropped in QuicConnGetPathForDatagram() because the local address is unset. I came up with this workaround:

diff --git a/src/core/path.c b/src/core/path.c
index c612c8e77..27bb22c98 100644
--- a/src/core/path.c
+++ b/src/core/path.c
@@ -190,12 +190,17 @@ QuicConnGetPathForDatagram(
     )
 {
     for (uint8_t i = 0; i < Connection->PathsCount; ++i) {
-        if (!QuicAddrCompare(
-                &Datagram->Route->LocalAddress,
-                &Connection->Paths[i].Route.LocalAddress) ||
-            !QuicAddrCompare(
-                &Datagram->Route->RemoteAddress,
-                &Connection->Paths[i].Route.RemoteAddress)) {
+
+        BOOLEAN LocalAddrMatch =
+            QuicAddrIsWildCard(&Connection->Paths[i].Route.LocalAddress) ||
+            QuicAddrCompare(&Datagram->Route->LocalAddress,
+                &Connection->Paths[i].Route.LocalAddress);
+
+        BOOLEAN RemoteAddrMatch =
+            QuicAddrCompare(&Datagram->Route->RemoteAddress,
+                &Connection->Paths[i].Route.RemoteAddress);
+
+        if (!LocalAddrMatch && !RemoteAddrMatch) {
             if (!Connection->State.HandshakeConfirmed) {
                 //
                 // Ignore packets on any other paths until connected/confirmed.

Not sure if accepting incoming packets if local addr is unset would be generally OK, but it seems to solve my immediate problem.

[nibanks]

@jacobgorm would you be willing/able to contribute back NAT punch-through support to MsQuic?

[jacobgorm]

@nibanks I got sidetracked a bit from this project, but before that I did manage to get something running, and will have another go this week at getting it to a state where it is good enough for integration in my app. Once I have something in a not too embarrasing state, I will be happy to share it, however, I found that the existing code's optimizations for maximum throughput on multi-core CPUs were mostly getting in my way, so in the end I found it easier to resort to writing a new and simplified datapath with just a single system socket doing all the work. I am not sure that is how one would do it given more perfect command/understanding of the entire MSQUIC codebase, but at least the code should be useful for reference.