Traversal: Use low TTL for probe packet

[skyfloogle]

The currently popular methods for NAT traversal involve having two clients send each other a UDP packet. This adds an entry to their NAT, linking their endpoint to the other client, which means the other client can reply, allowing communication. However, this doesn't work for everyone: some NATs, upon receiving an unsolicited packet, will add an entry linking that packet's source to the NAT itself, which completely blocks communication. The solution is to ensure the first packet does not reach the other side, by giving it an appropriate TTL value.

Dolphin happens to be quite well set up for this: the host fires the first shot, so it can configure one TTL value to use for all connecting clients. I propose an additional input field on the netplay setup menu, where the desired TTL can be filled in, and a "Test TTL" button which, when clicked, performs a test to check if it's adequate. This test requires three steps:

1. Send a "hello" over UDP to a test server (server A), which returns the IP and/or port of server B.
2. Send a UDP packet with the given TTL value to server B.
3. Ask server A to send a reply via server B. If it arrives, the TTL value used is sufficiently high to traverse all currently relevant NATs.

It's possible to run this test multiple times concurrently for different TTL values in order to automate the process, but the manual test seems more straightforward to implement and less janky to use.

I imagine you guys will want to use a custom server for this kind of thing, but as a proof of concept, I created a Python script to perform this test using publicly accessible STUN servers.

So far, I've implemented the TTL input field, but not the test, as I'd like to hear how you guys would prefer to handle it. I'm also open to suggestions on naming things: while the occasional forum post can be found describing this technique, I've yet to see any examples of it in practice, or even anyone giving the problem or the solution a name. This is despite the issue prompting it being by no means rare: I encounter it when using my phone as a Wi-Fi hotspot, and I've met several people who run into it even on wired connections (this includes almost every Brazilian I know).

[Leseratte10]

Interesting behaviour, I haven't heard of that (a NAT "linking" an incoming packet to itself blocking the port from being used by other applications. Do you happen to know any NAT that does that, or any link containing more information about that?

If there's really some NATs that do that, then this might be a useful patch for Wiimmfi (and non-Netplay online gameplay), too, both for Dolphin and for real Wii consoles.

[skyfloogle]

My most direct source of evidence is the numerous cases of it I've observed myself: my phone hotspot, and the numerous people I've tested it with in Finland, Argentina, China, and especially Brazil. Some of my Brazilian friends need a TTL as high as 4 to break through their badly-behaved NAT, which suggests it's somewhere deep in some ISP's infrastructure over there. I've updated my script so you can check whether your own NAT exhibits this behaviour.

I've found one person emailing the IETF describing this behaviour (also where I got the "connecting to itself" part from, it's difficult to observe empirically), but it got dismissed as a corner case, where standard procedure is to fall back to a relay server. This also got brought up here. He claims the behaviour can be replicated with iptables, adding a MASQUERADE rule but not a DROP rule. I haven't looked into verifying this myself though.

This forum post claims the port simply gets blocked, though this is difficult to distinguish from the NAT self-referencing itself, as the effect is the same.

I've found a number of sources advocating the use of TTLs for other reasons: here, here, and here.

For Wiimmfi you'd definitely want to automate the process a bit more than what I suggested for Dolphin. Dolphin benefits from giving users a lot of knobs to turn, but Wiimmfi you do expect to "just work". I haven't checked how susceptible Wiimmfi is to this behaviour, but it's definitely worth looking into.

[Leseratte10]

Yeah, I agree, for Wiimmfi we need to have a solution that works for everyone and not a setting people would have to tinker with. Which probably means we should start with a fairly high TTL and only reduce it if it still doesn't work.

The benefit Wiimmfi currently has, compared to Dolphin's Netplay is that it's currently not that big a deal if a single NAT traversal fails - there's usually more than 2 people in a MKW match so if there's players A, B and C, and C can for some reason not send any packets to B, he'll just send them to A instead with a note to forward them to B. Not ideal since it increases delay, and increases the bandwidth the proxying player needs to have, but it means that the game could in theory "experiment" with various NAT traversal techniques or various TTL values in the background while the players are already playing. Even if that takes a while, it would still be better compared to the current solution where the NAT traversal doesn't work at all and the connection runs through a proxying client for the whole match.

As for the links you posted, great that apparently the Linux iptables NAT does have this "bug" depending on how it's configured which means it should be somewhat easy to get a test environment for this.

Also, thanks a lot for that script. Next time someone complains about NAT issues with Wiimmfi I'll have them run your script and see what happens.

[skyfloogle]

Yeah, I agree, for Wiimmfi we need to have a solution that works for everyone and not a setting people would have to tinker with. Which probably means we should start with a fairly high TTL and only reduce it if it still doesn't work.

It should be noted that if a client behind this kind of NAT sends even one packet to someone who previously sent unsolicited packets, every new connection that socket makes will be assigned a new port, while maintaining the old port for existing connections. At this point the socket is no longer usable for UDP hole punching.

For Wiimmfi, I would propose to run a few tests during the initial server connection step to determine whether the bug exists, and if so, what TTL is required to get past it. If you have two clients with "good" NATs, status quo is adequate. If you have one with "good" and one with "bad", the "bad" one can shoot first without setting a TTL, or the "good" one can shoot first with a low TTL. If you have two "bads", TTL trickery is required.

[shuffle2]

• it would be nicer if the configuration were automatic and not adding Yet Another UI Config
• any changes to Externals (enet) should be made upstream if possible

[skyfloogle]

ENet has the necessary changes upstream now.
I'm happy to implement automated configuration, but I'd like to know what would be most convenient for you guys in terms of implementing the server side of the tests.
It wouldn't need a particularly complicated server, it'd just have two open UDP ports, each of which can be asked for its counterpart, or for a reply via its counterpart. Like should I write a Python script, or a C++ program like the traversal server, or should it be integrated into the traversal server, or does pretty much anything work?

[shuffle2]

I don't know answer to that. Maybe @delroth or someone can comment?

[delroth]

@danderson I'm curious if you've heard of this NAT traversal technique and if you have thoughts about its usefulness? :)

@skyfloogle easiest would be integrating it into the existing traversal server since it's already infrastructure we maintain, but if this is particularly hard I could be convinced to run some other server -- if we get there I'm happy to write down a list of requirements and/or contribute.

[danderson]

Ooh, a new cursed edge case! :)

I've never encountered NATs that behave as described. When I first read this I thought this was implementing a different NAT traversal technique[1]. If that's not what's being done, then I'm curious why y'all are messing with the TTL at all, aside from the discussion in this PR?

If I understand correctly: the claim is that in the world, there are NATs that will create invalid conntrack entries if they receive a probe packet from the remote peer on their WAN iface, before the local peer sent their own probe packet. Do I have that right?

Can you identify specific ISPs or purchasable NAT devices that do this? I would love to be able to independently reproduce this behavior. Assuming an invalid mapping gets created, I agree that this would persistently break the ability to form p2p links. From the linked discussion on the ice mailing list, it sounds like this might be reproducible with a basic linux box configured just right (basic masquerade setup, but make sure to REJECT unknown WAN flows instead of the more common DROP).

Assuming for now that the issue is real and behaves as described here and in linked discussions, the proposed workaround should work: send some deliberately expiring packets to prime the NATs at both ends first, then start on the actual traversal attempts. Because of the existence of CGNAT these days, you cannot reliably assume that the NAT will be 1 hop away, and worse the CGNAT may be on the far end of an L3 routed network so not only >1 hop away but also >2 hops away.

It would be nice to have it be self-configuring though. The obvious way (and only way that I can think of, because NATs are irritatingly stealthy from the client's POV so it's hard to map the network path) is exponential growth: set TTL=2, transmit a probe, then repeatedly double+transmit until you reach some safe cutoff, say 16 or 32. That's still a probabilistic approach and you can construct a pathological topology where that fails (e.g. two clients that are both behind the same CGNAT, where the CGNAT is one of the rare ones that allows hairpinning but also screws up this conntrack behavior), but for empirical common topologies, that should have a decent chance of doing the right thing.

I'll set up the configuration discussed on the ICE forums in my lab tomorrow and investigate the behavior more, and report back what I find.

[1]: set TTL to make packets expire between the facing NATs, and hope your local NAT doesn't correctly rewrite the payload in the ICMP error response, which lets you discover the mapped ip:port of an endpoint-dependent NAT. This used to work pretty well, but over time NATs have become more RFC 5508 compliant and patch up ICMP payloads, which nerfed the tactic.

[skyfloogle]

While I haven't independently verified why it happens (my best guess is from the ICE mailing list), I have observed the p2p-blocking symptoms in various places (particularly some friends in Brazil), and found that TTL hacking fixes it. Some of the cases I've seen do indeed appear to be CGNAT, requiring TTLs of like 4 to be adequately primed, but some appear to be home routers, needing only the standard 2. One such home router appears to be an Askey RTF3507VW-N2, from a Brazilian ISP called Vivo, and I've seen a relevant CGNAT in an ISP called Syncontel.

It would be nice to have it be self-configuring though. The obvious way (and only way that I can think of, because NATs are irritatingly stealthy from the client's POV so it's hard to map the network path) is exponential growth: set TTL=2, transmit a probe, then repeatedly double+transmit until you reach some safe cutoff, say 16 or 32.

This is exactly what I had in mind, except I was going to just increment it rather than doubling it, since I've only seen outcomes between roughly 2 and 4. You would need to create a new socket with a new port for every test, as a failed test will render the original port useless.

I'll set up the configuration discussed on the ICE forums in my lab tomorrow and investigate the behavior more, and report back what I find.

I look forward to hearing your results! I found this stuff a while ago, but I haven't been able to look into the hardware/firmware side of it much.

easiest would be integrating it into the existing traversal server since it's already infrastructure we maintain

Cool, I'll sort that out in a couple days then.

[mbc07]

One such home router appears to be an Askey RTF3507VW-N2, from a Brazilian ISP called Vivo [...]

That's the ISP I use, but with a different Askey unit (RTF8115VW). It's currently set up in bridge mode since I manage all my internal network with a separate router running OpenWrt, but it should be easy to return to the ISP defaults and use only the Askey unit if needed. Let me know if you need to test something with that particular setup...

[danderson]

Hopefully I can get enough info from a linux lab setup, if that doesn't work out we can try to gather more info, but I don't want to break your internet if we can avoid it :)

Thanks for the refs to Askey and Syncontel, I might be able to find some more details on what software they're running, or find a way to remotely detect this condition and map out how common it is.

[danderson]

This is exactly what I had in mind, except I was going to just increment it rather than doubling it, since I've only seen outcomes between roughly 2 and 4. You would need to create a new socket with a new port for every test, as a failed test will render the original port useless.

Yeah, that probably works fine. I was thinking of some very wide area CGNATs I've seen, where there are 4-5 hops from the home router to the CGNAT layer through the ISP's internal network. But like everything in NAT traversal, what you suggest will probably work for >90% of people with this problem, and it's easier to reason about 👍

[danderson]

An emergency came up today. I might be able to dig into this late tonight, but it'll probably be a bit longer. Sorry!

[skyfloogle]

No worries, take all the time you need.

I'd like to add that I checked my phone (Nokia 7.1 running Android 10) and when running a test on my laptop using my phone as a mobile hotspot, it has the bug, with a TTL of 2 required to get past it, which suggests that my phone specifically is the cause. I got a friend to test with his Honor View 20 with the same results, which leads me to suspect this is an Android-wide thing. My test was this slightly-updated Python script.
I ran a test directly on my phone just to check that it's not my provider doing something weird, and it demonstrated endpoint-independent filtering, which means it really is the phone.

[skyfloogle]

TTL is now determined automatically. The traversal server has an additional socket on port 6226, which functions identically to the first. The test works as follows:

1. Upon receiving the HelloFromServer message, the client creates a new socket on an unspecified port.
2. The new socket sends a ping to 6226 with a specified TTL (starting at 2).
3. The new socket sends a new "TestPlease" request with its host ID to the original port, 6262.
4. Upon receiving the TestPlease packet, the server sends two acks: one via 6226 to the return address, and one via 6262 to the endpoint associated with the host ID.
5. Upon receiving the TestPlease ack on its main socket, the client watches the second socket for the corresponding ack (which should arrive at around the same time) for 50ms. If it's received, stop testing and use the TTL next time someone tries to join. Otherwise, increment the TTL, recreate the socket, and try again. If it hits 32 it gives up.

This should come to a solution within a couple seconds at most, well before the host has given their ID to anyone, let alone had someone try joining.

[skyfloogle]

Rebased on latest master so that #11381 is included in potential test builds.

It's worth noting that having a second port on the traversal server means we can automatically detect port-dependent mapping, which would probably make troubleshooting much easier. I don't know how common address-dependent mapping is in practice (it would require a server that somehow has two public IPs, or maybe two servers with different IPs), but I suspect port-dependent mapping is the more common one by far. That's probably not for this PR though.

[fanick1]

Nice. Changing TTL looks like groundbreaking idea, thanks for sharing your findings.

[lioncash]

Ok, I'm not a networking person, but after a (lot) of reading, this seems fine? (as far as I can tell, unless I've overlooked something). Would like to get the landed (or something of the sort, since this has been sitting for quite a while).

@danderson Sorry for the ping, but were you ever able to get around to digging into it? Don't want to jump the gun if you were still looking into anything.

[danderson]

Sorry, I completely forgot about this. I tried reproducing the failure mode, and failed to do so. So, I can't really offer much more except to say that the change here shouldn't hurt NAT traversal, so if there are known setups where this helps, then yeah you should go ahead with it 👍

[lioncash]

Alright, sounds good, thanks!

[AdmiralCurtiss]

This seemingly got lost in the shuffle, but does the traversal server (as in, this) need port 6226 opened now?

[skyfloogle]

This seemingly got lost in the shuffle, but does the traversal server (as in, this) need port 6226 opened now?

Yes, it does.

[OatmealDome]

Is the modified traversal server backwards compatible with older Dolphin versions?

[skyfloogle]

Is the modified traversal server backwards compatible with older Dolphin versions?

Should be, it only adds commands.

[OatmealDome]

OK, the new version should be live now.