Reliable identification of the initial packet for a connection

[igorlord]

Some path elements may need to statelessly identify an initial packet of a connection. These path elements could include load balancers and organization's border security systems. Ability to identify the initial packet may also be required for systems implementing Stateless Rejects #60.

Since QUIC allows for packet number truncation, the most straightforward way to identify the initial packet is by checking for packet number 1 and VERSION flag set.

Hence:

• The language of section 6.1 Version Negotiation would need to change SHOULD to MUST in "All subsequent packets sent by the client SHOULD have the VERSION flag unset".
• Section 6.1 Version Negotiation should also gain language like: "If Version negotiation is not complete in 2^6 packets, any endpoint MAY reset this connection". (This is a weaker statement than Repeating Version Negotiation #143.)
• Proposal in issue Starting packet number #35 is not compatible with this issue. If packet numbers are allowed to be deliberately skipped, we should require that packets with VERSION set MUST NOT be skipped.

[MikeBishop]

You're entirely correct that #35 is not compatible, but it's more fundamental: This issue suggests that we make it possible to identify and act on a particular packet when you're not a party to the connection; #35 suggests making it more difficult to act on and therefore ossify an exposed protocol element.

I suspect you'll need to back up and motivate why we want to allow path elements to identify a particular packet in a connection. "Some... may need to" isn't a justification. My gut reaction is that load balancers at least should be statelessly operating on the connection ID itself, whether it's the first packet or the billionth.

[igorlord]

I suspect you'll need to back up and motivate why we want to allow path elements to identify a particular packet in a connection.

Certainly. Let's take a load balancer system. Think LVS or Google Maglev. These systems consist of multiple machines serving as load balancers, each with imperfect availability, and packets belonging to a single connection may arrive at different load balancer servers.

These load balancers are not proxies. They are more like routers. They do not terminate connections but rather forward packets to the appropriate backend servers.

Load balancer's job is to:
a) For a new connection, select an appropriate server to handle it, and
b) Keep sending all packets that are a part of the connection to the same server, and
c) Keep doing (b) even if the set of machines eligible to serve new connections of this type changes dynamically (say, every second)

To do its job, at a minimum, the load balancer needs to identify new connections (a). Identifying first packet (subject of this issue) is trivial with TCP (SYN). It should be trivial with QUIC, too.

[igorlord]

One more reason for changing SHOULD to MUST of section 6.1 Version Negotiation in "All subsequent packets sent by the client SHOULD have the VERSION flag unset".

In draft-ietf-quic-tls-01, 6.1.1. Initial Key Transitions, we require:

Packets protected with 1-RTT keys have a KEY_PHASE bit set to 1. These packets also have a VERSION bit set to 0.

If I understand it correctly, TLS 1-RTT keys should be available at the same time as the completion of the Version negotiation. So changing of SHOULD to MUST is mandated by draft-ietf-quic-tls-01 (6.1.1). Right?

[ianswett]

If we want to make identification easier, I'd suggest when the version bit is set, we also include 4 fixed bytes that identify the protocol as QUIC. I'd suggest the string QUIC, so it is easy to identify in wireshark as well.

I have mixed feelings on making it easier to identify UDP traffic QUIC, but I think it's going to be a thing people will do, so I'd rather have them latch onto an explicit signal than hardcode the version number or something worse.(public flags anyone...)

Two notes:

1. Initial packets from the client cannot have a truncated connection ID, because the peer has to inform you it's acceptable in the handshake.
2. There are proposals to randomize the initial packet number, so relying on it being packet number 1 would conflict with that.

[igorlord]

The issue here is not identifying traffic as QUIC (it could be a different issue).

The issue is being able to identify new client connections statelessly (just by observing the packets).

This is needed for load balancers. As a load balancer I want to know whether I need to:

• pick a backend server for this connection (i.e. processing "TCP SYN" or "QUICK packet 1"), or
• try to identify which backend server is already handling this connection and forward the packet that way ("TCP ACK or RST" or "QUICK packet > 1").

This is especially important, since I would like to keep load balancers stateless and will want to use "Stateless Reject" to encode the load balancing decision for a particular connection. So I need to be able to statelessly identify initial packets and send "Stateless Reject" to them.

draft-ietf-quic-tls-01 already requires (6.1.1. Initial Key Transitions) that packets without 0-RTT protection have KEY_PHASE=0, VERSION=1; with 0-RTT have KEY_PHASE=1, VERSION=1; with 1-RTT have KEY_PHASE=1, VERSION=0.

So I would like to bring draft-ietf-quic-transport-01 is agreement with draft-ietf-quic-tls-01 and make setting VERSION=0 a MUST instead of SHOULD.

If there is a strong desire (for a good reason) to randomize the initial packet number, I think I can live with it, and send "Stateless Rejects" to all packets with KEY_PHASE=0, VERSION=1.

[mirjak]

I still don't see why you need to identify the first packet. Isn't for a load balancer simply the first packet of a connection the first ones it sees when it doesn't have a mapping yet? And if the load balancers change dynamically, don't you have to sync state anyway all the time?

[igorlord]

That would be true for a completely stateful load balancer. And that's exactly the kind of load balancing system I'd like to avoid having to build -- a system that requires all nodes to be completely in sync and aware of all connections real-time.

This system is very expensive (CPU and bandwidth), especially when you think global scale (not just in-datacenter). And it may not work very reliably anyway due to packets for a single connection arriving very quickly, since if packet 1 and 2 arrive at different nodes, there may had been not enough time for all nodes to get in sync. If you try to fix the previous problem by adding a request-response mechanism for connections you do not know about, you need to worry about your vulnerability to an attack with random packets, all of which may need to go through this expensive request-response. And simply setting up rate limiting on request-response to deal with attacks would not work, since there could be legitimate events that could cause a ton of traffic shifting load balancers all together even in a well-designed system (think BGP change for a global load balancer or a server crash for a local one).

A little more on this in issue #205.

(Our current system for TCP is "mostly stateless".)

[martinthomson]

@ianswett, do you want that four bytes AND #167?

[ianswett]

I think it was mentioned today that this is no longer an issue, is that correct @igorlord ?

[igorlord]

@ianswett Yes. It seems resolved for now, although there are proposals (like #203) that may need reopening this issue.

[janaiyengar]

How was this issue resolved?

I don't understand the premise of the problem here. If you want to build something completely stateless, the only way to do it is ECMP with a consistent hash. You can do that on any packet in the connection and it won't matter.

But if you're doing anything smarter, you're going to be stateful. You cannot know which server to direct traffic for existing connections to without maintaining state. The state that you need it a table mapping 4-tuple to server for TCP, and you could do the same with connection ID to server for QUIC.

Since you will have this map (this may be the "mostly stateless" part you mention earlier), any received packet that has a connection ID that's missing from the map can create a new entry. You can be smarter about it, but this is a fairly simple and largely effective algorithm that ought to work.

[igorlord]

No, you _can_ know which server to direct traffic for existing connections to without maintaining state. All you need is something in each packet you are routing that identifies that server.  Hence a server-generated ConnectionID for QUIC. For tcp, there is the sequence number that the server generated during SYC/ACK and which is carried in all TCP packets (and ICMP Error packets), except for TCP RST.  The "mostly stateless" for TCP is due to the nature of the always-mutating sequence number -- if the connection persists long enough, you may need to start keeping state. The map you are suggesting would need to be a distributed map (it is not just one box that is doing all the load balancing).  While such a distributed map is possible within one pop, it would be a complex system that must keep nodes synced to a firehouse of changes and be resilient to packet #2 arriving on a different node, while the information about the routing decision has not yet propagated from the original node. What's worse, for QUIC, this map would not work for Anycast-based CDNs, since when you are migrating from your WiFi network to a Mobile network, you are likely going to be taken to a geographically different Anycast pop. Reliably syncing a firehouse of connection establishment/teardown events across pops makes doing the same within a single pop seem easy. - Igor P.S. Consistent Hash can help here but just a little. You can use consistent hash to pick a load balancer machine within a pop. That allows you to limit sharing of state with fewer than all load balancing nodes. But that cannot be used to route to the backend servers, since you need to consistently route to the correct backend nodes despite of the frequent changes in the set of backend nodes available to serve this particular traffic. On Friday, February 10, 2017 7:28 PM, janaiyengar <notifications@github.com> wrote: How was this issue resolved?I don't understand the premise of the problem here. If you want to build something completely stateless, the only way to do it is ECMP with a consistent hash. You can do that on any packet in the connection and it won't matter.But if you're doing anything smarter, you're going to be stateful. You cannot know which server to direct traffic for existing connections to without maintaining state. The state that you need it a table mapping 4-tuple to server for TCP, and you could do the same with connection ID to server for QUIC.Since you will have this map (this may be the "mostly stateless" part you mention earlier), any received packet that has a connection ID that's missing from the map can create a new entry. You can be smarter about it, but this is a fairly simple and largely effective algorithm that ought to work.— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[MikeBishop]

#361 purports to address this; please re-open or file a new issue if that's incorrect.