IP Anycast with server IP+Port renegotiation

[huitema]

Many large DNS server deployments rely on IP Anycast for directing UDP requests to the "closest" server. As we consider QUIC for DNS, it would be nice to support the same kind of deployment. However, DNS over QUIC requires server-side state. The instantaneous nature of IP routing could cause anycast packets to be delivered to the "wrong" server - a server instance that does not hold the state for the connection.

This could be solved if after the initial exchange the server indicated an alternative non-anycast IP address, and maybe an alternative port number. The client could then switch to using that address, assuring then that the QUIC connection would remain stable even if IP routing did change.

There is are obvious security issues, but they can be mitigated. The server indication would need be protected against spoofing, which probably means that it would have to come as a 1-RTT packet. A malicious server could redirect client traffic towards an unsuspecting party, in some kind of DOS attack. This can be mitigated if the client sends a verification packet to the new address, and only starts using the new address for "regular" traffic if the verification packet is acknowledged.

[martinthomson]

I'd like to hear from server operators whether they see the need for this sort of capability. Otherwise I think that we should park this.

It's certainly possible to define a new frame type (call it MIGRATE) that acts much like the HTTP Alt-Svc frame. An endpoint that receives it might move to the new transport endpoint as soon as feasibly possible. For this, you would want to pair MIGRATE with NEW_CONNECTION_ID so that linkability could be broken.

We'd also have to be careful not to accidentally define multipath, because the natural reaction to this is to retain the old path until the new path has a sufficiently wide congestion window.

In any case, the frame that tests the new path for liveness is necessary before we address this. See #161 for more on that.

[huitema]

The anycast support is simpler than general server mobility. Server only need to send the non-unicast address once. They could send the preferred IP address & port as part of the transport parameter TLS extension, probably together with a new connection ID.

I would also like to see a possibility to MIGRATE later in the connection, but that's a different issue.

I agree that this looks a bit like #161, but on the other hand, server initiated DOS is a much less acute problem that client initiated DOS. Currently, a server can easily send the client a script that will trick the client into bombing random locations -- I think the great cannon worked just like that. The general defense is to identify the server...

[ThomasSwindells]

As a CDN operator this proposal looks valuable to me, in a long lived connection context (e.g. HTTP etc) quickly moving clients off the anycast address onto a server specific address provides a reliable means for machines to stop advertising the anycast address when they become busy(loaded) but still continue to serve existing connections.

As Christian has pointed out the option to also use MIGRATE later in the connection is also an interesting concept with value, but that is definitely a different issue.

[mirjak]

One side comment, I think any make-before-break solution is basically multipath, and as such we should make sure if we design such a mechanism that it can also be used for multi path correctly. So yes, we shouldn't design multipath by accident but if multipath is what we need for another feature that we would like to support, we should design it as providing both.

[LPardue]

Echoing Christian's and Thomas' comments.

The early-handoff of clients to a specific address could be a useful capability for deployments. In addtion to anycast, this could also be beneficial for load balanced deployments that use ECMP.

The compromise with the TLS extension is that relocation chaining would not be supported (e.g. a service on one anycast IP, directs to another anycast IP, that directs to a non-anycast IP). A MIGRATE frame could deal with this, along with a multitude of other more complex deployments.

[igorlord]

Would not one be able to use Server-generated Connection ID to encode the server/pop containing state for this connection? In case of BGP reconvergence, the server/load balancer would examine the Connection ID and tunnel the packet to the correct server/pop.

If a true "Alt-Svc-for-QUIC" is needed, I am all for migrating truly useful features to lower layers (like transport). But does this feature HAVE to belong in transport? Since we are already considering not using a 5-tuple to identify a connection (see #714), a MIGRATE signal can be an application-level signal, as long as the QUIC library API allows the application to change its endpoint's IP/port.

[MikeBishop]

I don't see this as being ALTSVC -- ALTSVC by definition implies the other endpoint is an acceptable alternative the client should consider for the future, not a "move now" directive. However, I do see this as being a special case of server-side migration. We can borrow the logic from Teredo of sending appropriate bubble packets to open the NAT, but we should keep things symmetrical with an eye toward peer-to-peer uses of QUIC.

Perhaps the MIGRATE frame simply says, "I expect to shortly begin using this IP/port/Connection ID; please send a PING frame to that destination"? The recipient then begins the process in #732 as if it had just received a packet from that IP/port.

We'd need to think about attacks where the sender knows there's an existing QUIC connection at that IP/port with that Connection ID (the attempt to validate then looks like an address change from the target connection's peer), but I'd hope that it's sufficiently difficult to force your connection to use the same key as the target, so decryption would just drop the packet.

[ianswett]

This is definitely valuable.

In the short term, I'd expect most users to migrate at the end of the handshake. In the long term, there are other use cases, such as migrating from one server to another within a cluster for load balancing/etc.

[janaiyengar]

I seem to have missed this thread entirely -- this is very valuable indeed. This is also useful for connection migration later on in the connection, since a migrating client may hit a different server if the servers are behind anycast IPs.

I'm also in favor of making this part of the server transport params instead of a new frame. It's sent exactly once, at the beginning of the connection.

[igorlord]

I actually like this proposal a lot. Moving connections off anycast address to a unicast address belonging to the same server quickly (at the end of a handshake) is a major use-case.

@ianswett's migration to a different server is a different use case.

[mikkelfj]

wrt. migration to different server, should there be protection against loops, or is this an application configuration issue?

[igorlord]

Loops sound like an app concern. (And it is probably among the simplest of app concerns in case of a mid-connection server migration. The app would also need to migrate app state and quic state out-of-band. It would also need to forward to the new sever quic packets arriving at the old sever after that state migration.)

[LPardue]

This ticket seems relevant to the connection migration topic discussed in Singapore. I'm not sure if it came up during the actual session but may provide another example usage beyond NAT rebinding and mult-interface single path.

It may be beyond the scope of "minimal connection migration" and be a candidate for a v2 label?

[igorlord]

Just to add another related use case -- migrating to another IP immediately after receiving Client Initial, ie before the completion of the handshake.

That can be used for migration off anycast to unicast of the same sever. It can also be used for migration to a better server based on client's IP and SNI of the connection.

[martinthomson]

This depends on a resolution to #880. Editors discussed the inclusion of a preferred IP transport parameter and how that might be used.

[MikeBishop]

Proposal to be written up:

• New transport param with server's "preferred IP"
• Client MUST probe the server's preferred address following the Handshake
• Same constraints about how much data can be sent to an unvalidated IP address apply
• Need to talk about what happens if the client migrates before the probe completes (client SHOULD probe the preferred IP only; MAY probe the old IP if they fail)

[hardie]

Proposal to be written up: - New transport param with server's "preferred IP" - Client MUST probe the server's preferred address following the Handshake

When writing the proposal, can I ask that we be really clear about whether we expect this to be used across address families? If someone reaches server X on an anycast IPv4 address, for example, will this transport param be allowed to redirect them to an IPv6 address? Or vice versa? While I can see very valid reasons for doing so, that would be a change to how happy eyeballs work, and I'd like to get review if it will be permitted.