Dual-NAT / Dual-load-balancer scenarios

[MikeBishop]

When thinking about peer-to-peer use-cases, it will obviously be common for both ends to be behind a NAT or (in a datacenter) behind a load balancer. If we're talking about having load-balancer routing information baked into the Connection ID, we might need different connection IDs in each direction.

This would be relatively easy, since we're already talking about putting the Connection ID into the transport parameters -- each party just declares what Connection ID it wants the other party to use when sending packets to it, and the client's initial Connection ID is just a throw-away. (Which might coincidentally be the same as one or both of the Connection IDs they agree to use.) If you don't want/need a Connection ID included in your direction, simply don't provide a value for this setting. At the same time, you'd want to make the NEW_CONNECTION_ID frame symmetric rather than server-only.

Dual NATs would also imply that PING frames need to be sent in both directions for keep-alive purposes, which affects the guidance being added to HTTP in #744.

[martinthomson]

I don't think that you need to worry about PINGs in both directions. A PING-ACK exchange initiated by a client is enough for the client NAT, then it should keep the server NAT open. The only trick being that the keep-alive interval needs to be the lower of the timeouts at the two NATs.

The problem with a connection ID in both directions is that it screws badly with the design of the stateless reset. We'd probably want to make it appear to be a connection migration, which sticks out a little more than before.

[janaiyengar]

I like this idea of having different connection IDs in the two directions, and I think it's useful. Why does it mess with the design of Stateless Reset?

[MikeBishop]

Let's say we configure a connection to use ID A when sending from client-to-server, but ID B when sending from server-to-client. Now the server receives a packet with connection ID A, but has lost all its state.

In the current model, the server generates a Stateless Reset packet that looks like any other packet it would send to the client, re-creates the token by deriving it from A, and pads out the packet with random nonsense.

With dual connection IDs, the server has no idea what connection ID it's supposed to use in reply packets, so it necessarily looks different from a "real" response packet. It could reuse 'A' in the wrong direction, or could omit the Connection ID entirely, but both of those will be different from "real" responses. (And if the Connection ID is actually needed to route back to the client, then neither of those will actually reach the other end anyway.)

[janaiyengar]

Hm, ok, I'm not so worried about the looking different part, but there's the addition issue of the load balancer not routing the stateless reset back to the right receiver.

[igorlord]

I wonder how realistic the need for double load balancing would be.

In a datacenter, all machines will typically have unique globally routable IPs (maybe IPv6-only). You'd want to initiate connections from your unique IP. I would be puzzled to see a design that requires initiating connections sourced from a load balanced IP, since the load balancer would just be in the way on the return path.

For peer-to-peer users, the NAT on the client's side is not likely to be a problem, since it is not a "load balancer". Even if a NAT could also do load balancing and had to use a single IP (why? RFC 6177 suggests for end site to be allocated at least /64), that NAT would "load balance" only some UDP ports (the ports used by known apps) and do ordinary NAT port mapping for other ports.

[mirjak]

We could put two congestion IDs in all packets, or use half the connection ID for each direction...

However, I guess if the stateless reset gets lost/mis-routed somehow, the client might keep sending data but will never get a response and might therefore stop at some point.... a load balancer could even learn to understand a stateless reset and block an additional packets from the client on that connection. Not sure if that is a great idea...

[MikeBishop]

Fixed by #1151.