From 989207a697ce6e4379da3774876e22e275d252bf Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 15:01:09 -0500 Subject: [PATCH 01/13] Specify MaxAckDelay for TLP and RTO Closes #981 --- draft-ietf-quic-recovery.md | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index 7d7343a677..a664b8fcff 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -301,7 +301,7 @@ The alarm duration, or Probe Timeout (PTO), is set based on the following conditions: * If there is exactly one unacknowledged packet, PTO SHOULD be scheduled for - max(2*SRTT, 1.5*SRTT+kDelayedAckTimeout) + max(2*SRTT, 1.5*SRTT+MaxAckDelay) * If there are more than one unacknowledged packets, PTO SHOULD be scheduled for max(2*SRTT, 10ms). @@ -309,13 +309,11 @@ conditions: * If RTO ({{rto}}) is earlier, schedule a TLP alarm in its place. That is, PTO SHOULD be scheduled for min(RTO, PTO). -kDelayedAckTimeout is the expected delayed ACK timer. When there is exactly one -unacknowledged packet, the alarm duration includes time for a delayed -acknowledgment to be received by including kDelayedAckTimeout. - -The RECOMMENDED value for kDelayedAckTimeout is 25ms. - -(TODO: Add negotiability of delayed ack timeout.) +MaxAckDelay is the maximum ack delay supplied in an incoming ack frame. +Ack delays that are aren't used for RTT sample or reference ack-only packets +are excluded. When there is exactly one unacknowledged packet, the alarm +duration includes time for a delayed acknowledgment to be received by including +MaxAckDelay. A PTO value of at least 2*SRTT ensures that the ACK is overdue. Using a PTO of exactly 1*SRTT may generate spurious probes, and 2*SRTT is simply the next @@ -360,7 +358,7 @@ Similar to TCP {{!RFC6298}}, the RTO period is set based on the following conditions: * When the final TLP packet is sent, the RTO period is set to max(SRTT + - 4*RTTVAR, minRTO) + 4*RTTVAR + MaxAckDelay, minRTO) * When an RTO alarm fires, the RTO period is doubled. @@ -502,6 +500,9 @@ rttvar: min_rtt: : The minimum RTT seen in the connection, ignoring ack delay. +max_ack_delay: +: The maximum ack delay a peer's ack frame used in this connection. + reordering_threshold: : The largest delta between the largest acked retransmittable packet and a packet containing retransmittable frames before @@ -541,6 +542,7 @@ follows: smoothed_rtt = 0 rttvar = 0 min_rtt = 0 + max_ack_delay = 0 largest_sent_before_rto = 0 time_of_last_sent_packet = 0 largest_sent_packet = 0 @@ -602,6 +604,10 @@ Pseudocode for OnAckReceived and UpdateRtt follow: // Adjust for ack delay if it's plausible. if (latest_rtt - min_rtt > ack_delay): latest_rtt -= ack_delay + // Only save into max ack delay if it's used + // for rtt calculation and is not ack only. + if (!sent_packets[ack.largest_acked].ack_only) + max_ack_delay = max(max_ack_delay, ack_delay) // Based on {{?RFC6298}}. if (smoothed_rtt == 0): smoothed_rtt = latest_rtt @@ -703,7 +709,7 @@ Pseudocode for SetLossDetectionAlarm follows: else if (tlp_count < kMaxTLPs): // Tail Loss Probe if (num_retransmittable_packets_outstanding == 1): - alarm_duration = 1.5 * smoothed_rtt + kDelayedAckTimeout + alarm_duration = 1.5 * smoothed_rtt + max_ack_delay else: alarm_duration = kMinTLPTimeout alarm_duration = max(alarm_duration, 2 * smoothed_rtt) From 372c3189eeb4366b51eab91b29ed62d5c2e3aa9e Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 15:06:16 -0500 Subject: [PATCH 02/13] Update draft-ietf-quic-recovery.md --- draft-ietf-quic-recovery.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index a664b8fcff..dba75d811a 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -501,7 +501,9 @@ min_rtt: : The minimum RTT seen in the connection, ignoring ack delay. max_ack_delay: -: The maximum ack delay a peer's ack frame used in this connection. +: The maximum ack delay in an incoming ack frame for this connection. + Excludes ack delays for ack only packets and those that create an + RTT sample less than min_rtt. reordering_threshold: : The largest delta between the largest acked @@ -518,9 +520,9 @@ transmit or exceeding the reordering window in time. sent_packets: : An association of packet numbers to information about them, including a number field indicating the packet number, a time field indicating the time a packet - was sent, and a bytes field indicating the packet's size. sent_packets is - ordered by packet number, and packets remain in sent_packets until - acknowledged or lost. + was sent, a boolean indicating whether the packet is ack only, and a bytes + field indicating the packet's size. sent_packets is ordered by packet + number, and packets remain in sent_packets until acknowledged or lost. ### Initialization @@ -571,6 +573,7 @@ Pseudocode for OnPacketSent follows: largest_sent_packet = packet_number sent_packets[packet_number].packet_number = packet_number sent_packets[packet_number].time = now + sent_packets[packet_number].ack_only = is_ack_only if !is_ack_only: OnPacketSentCC(sent_bytes) sent_packets[packet_number].bytes = sent_bytes From 7c508ce53f9a978c637cb25ba28b030b3e69e62b Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 15:06:53 -0500 Subject: [PATCH 03/13] Update draft-ietf-quic-recovery.md --- draft-ietf-quic-recovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index dba75d811a..21094082a2 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -310,7 +310,7 @@ conditions: PTO SHOULD be scheduled for min(RTO, PTO). MaxAckDelay is the maximum ack delay supplied in an incoming ack frame. -Ack delays that are aren't used for RTT sample or reference ack-only packets +Ack delays that are aren't used for an RTT sample or reference ack-only packets are excluded. When there is exactly one unacknowledged packet, the alarm duration includes time for a delayed acknowledgment to be received by including MaxAckDelay. From e1d7bacc65c3850daaaa47e5d1ddde7c6ef70897 Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 15:37:54 -0500 Subject: [PATCH 04/13] Add RTO rationale --- draft-ietf-quic-recovery.md | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index 21094082a2..1ab93d37e5 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -310,10 +310,13 @@ conditions: PTO SHOULD be scheduled for min(RTO, PTO). MaxAckDelay is the maximum ack delay supplied in an incoming ack frame. -Ack delays that are aren't used for an RTT sample or reference ack-only packets -are excluded. When there is exactly one unacknowledged packet, the alarm -duration includes time for a delayed acknowledgment to be received by including -MaxAckDelay. +MaxAckDelay excludes Ack delays that aren't included in an RTT sample because +they're too large and those which reference an ack-only packet. When there is +exactly one unacknowledged packet, the alarm duration includes time for a +delayed acknowledgment to be received by including MaxAckDelay. + +QUIC diverges from TCP by calculating MaxAckDelay dynamically, instead of +assuming a constant value for all connections. A PTO value of at least 2*SRTT ensures that the ACK is overdue. Using a PTO of exactly 1*SRTT may generate spurious probes, and 2*SRTT is simply the next @@ -370,10 +373,17 @@ one significantly increases resilience to packet drop in both directions, thus reducing the probability of consecutive RTO events. QUIC's RTO algorithm differs from TCP in that the firing of an RTO alarm is not -considered a strong enough signal of packet loss. An RTO alarm fires only when +considered a strong enough signal of packet loss, so does not result in an +immediate change to CWND or recovery state. An RTO alarm fires only when there's a prolonged period of network silence, which could be caused by a change in the underlying network RTT. +QUIC also diverges from TCP by including MaxAckDelay in the RTO period. QUIC is +able to explicitly model the ack delay via the ack delay field in the ack frame. +Because QUIC is subtracting this delay from both SRTT and it is expected to +reduce RTTVar, it is necessary to add it back, in order to compensate for +the smaller SRTT and RTTVar. + When an acknowledgment is received for a packet sent on an RTO event, any unacknowledged packets with lower packet numbers than those acknowledged MUST be marked as lost. From 25784256da325462780bd85ac61fc7d056e15e0f Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 15:47:03 -0500 Subject: [PATCH 05/13] Update draft-ietf-quic-recovery.md --- draft-ietf-quic-recovery.md | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index 1ab93d37e5..e7bccceec0 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -300,11 +300,7 @@ receiver. The alarm duration, or Probe Timeout (PTO), is set based on the following conditions: -* If there is exactly one unacknowledged packet, PTO SHOULD be scheduled for - max(2*SRTT, 1.5*SRTT+MaxAckDelay) - -* If there are more than one unacknowledged packets, PTO SHOULD be scheduled for - max(2*SRTT, 10ms). +* PTO SHOULD be scheduled for max(1.5*SRTT+MaxAckDelay, 10ms) * If RTO ({{rto}}) is earlier, schedule a TLP alarm in its place. That is, PTO SHOULD be scheduled for min(RTO, PTO). @@ -316,15 +312,14 @@ exactly one unacknowledged packet, the alarm duration includes time for a delayed acknowledgment to be received by including MaxAckDelay. QUIC diverges from TCP by calculating MaxAckDelay dynamically, instead of -assuming a constant value for all connections. - -A PTO value of at least 2*SRTT ensures that the ACK is overdue. Using a PTO of -exactly 1*SRTT may generate spurious probes, and 2*SRTT is simply the next -integral value of RTT. - -The values of 2 and 1.5 are based on -{{?LOSS-PROBE=I-D.dukkipati-tcpm-tcp-loss-probe}}, but implementations MAY -experiment with other constants. +assuming a constant value for all connections. It includes this in all probe +timeouts, because it assume the ack delay may come into play, regardless of +the number of packets outstanding, where as TCP's TLP assumes if at least 2 +packets are outstanding, acks will not be delayed. + +A PTO value of at least 1.5*SRTT ensures that the ACK is overdue. The of 1.5 +is based on {{?LOSS-PROBE=I-D.dukkipati-tcpm-tcp-loss-probe}}, but +implementations MAY experiment with other constants. To reduce latency, it is RECOMMENDED that the sender set and allow the TLP alarm to fire twice before setting an RTO alarm. In other words, when the TLP alarm From f14f41f3b49e52cb820ca3000443827d36c090b1 Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 15:47:47 -0500 Subject: [PATCH 06/13] Update draft-ietf-quic-recovery.md --- draft-ietf-quic-recovery.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index e7bccceec0..ec4cd0e605 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -307,9 +307,7 @@ conditions: MaxAckDelay is the maximum ack delay supplied in an incoming ack frame. MaxAckDelay excludes Ack delays that aren't included in an RTT sample because -they're too large and those which reference an ack-only packet. When there is -exactly one unacknowledged packet, the alarm duration includes time for a -delayed acknowledgment to be received by including MaxAckDelay. +they're too large and those which reference an ack-only packet. QUIC diverges from TCP by calculating MaxAckDelay dynamically, instead of assuming a constant value for all connections. It includes this in all probe From a1b67988615e938f59178161b2c0472c3192802d Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 15:48:31 -0500 Subject: [PATCH 07/13] Update draft-ietf-quic-recovery.md --- draft-ietf-quic-recovery.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index ec4cd0e605..4b38ad3a97 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -306,8 +306,8 @@ conditions: PTO SHOULD be scheduled for min(RTO, PTO). MaxAckDelay is the maximum ack delay supplied in an incoming ack frame. -MaxAckDelay excludes Ack delays that aren't included in an RTT sample because -they're too large and those which reference an ack-only packet. +MaxAckDelay excludes ack delays that aren't included in an RTT sample because +they're too large and excludes those which reference an ack-only packet. QUIC diverges from TCP by calculating MaxAckDelay dynamically, instead of assuming a constant value for all connections. It includes this in all probe From 3ba73bed8669b4990a3df9a8b1da5108fa154a6b Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 15:49:30 -0500 Subject: [PATCH 08/13] Update draft-ietf-quic-recovery.md --- draft-ietf-quic-recovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index 4b38ad3a97..92ac931f81 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -312,7 +312,7 @@ they're too large and excludes those which reference an ack-only packet. QUIC diverges from TCP by calculating MaxAckDelay dynamically, instead of assuming a constant value for all connections. It includes this in all probe timeouts, because it assume the ack delay may come into play, regardless of -the number of packets outstanding, where as TCP's TLP assumes if at least 2 +the number of packets outstanding. TCP's TLP assumes if at least 2 packets are outstanding, acks will not be delayed. A PTO value of at least 1.5*SRTT ensures that the ACK is overdue. The of 1.5 From ddcf6d977ca47ea4e363844f5c9f116619719163 Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 15:50:45 -0500 Subject: [PATCH 09/13] Update draft-ietf-quic-recovery.md --- draft-ietf-quic-recovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index 92ac931f81..4053f6f26d 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -315,7 +315,7 @@ timeouts, because it assume the ack delay may come into play, regardless of the number of packets outstanding. TCP's TLP assumes if at least 2 packets are outstanding, acks will not be delayed. -A PTO value of at least 1.5*SRTT ensures that the ACK is overdue. The of 1.5 +A PTO value of at least 1.5*SRTT ensures that the ACK is overdue. The 1.5 is based on {{?LOSS-PROBE=I-D.dukkipati-tcpm-tcp-loss-probe}}, but implementations MAY experiment with other constants. From 6f644ac4c06453dd5653b717d3aa25180829fc41 Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 15:56:12 -0500 Subject: [PATCH 10/13] Update draft-ietf-quic-recovery.md --- draft-ietf-quic-recovery.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index 4053f6f26d..fb977b59d7 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -714,11 +714,8 @@ Pseudocode for SetLossDetectionAlarm follows: alarm_duration = loss_time - time_of_last_sent_packet else if (tlp_count < kMaxTLPs): // Tail Loss Probe - if (num_retransmittable_packets_outstanding == 1): - alarm_duration = 1.5 * smoothed_rtt + max_ack_delay - else: - alarm_duration = kMinTLPTimeout - alarm_duration = max(alarm_duration, 2 * smoothed_rtt) + alarm_duration = max(1.5 * smoothed_rtt + max_ack_delay, + kMinTLPTimeout) else: // RTO alarm alarm_duration = smoothed_rtt + 4 * rttvar From c58974b553d5599045faf1ed2e6496106c7ab45c Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 16:04:03 -0500 Subject: [PATCH 11/13] Comments --- draft-ietf-quic-recovery.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index fb977b59d7..940b343ae0 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -310,9 +310,9 @@ MaxAckDelay excludes ack delays that aren't included in an RTT sample because they're too large and excludes those which reference an ack-only packet. QUIC diverges from TCP by calculating MaxAckDelay dynamically, instead of -assuming a constant value for all connections. It includes this in all probe -timeouts, because it assume the ack delay may come into play, regardless of -the number of packets outstanding. TCP's TLP assumes if at least 2 +assuming a constant delayed ack timeout for all connections. QUIC includes +this in all probe timeouts, because it assume the ack delay may come into play, +regardless of the number of packets outstanding. TCP's TLP assumes if at least 2 packets are outstanding, acks will not be delayed. A PTO value of at least 1.5*SRTT ensures that the ACK is overdue. The 1.5 @@ -367,15 +367,14 @@ reducing the probability of consecutive RTO events. QUIC's RTO algorithm differs from TCP in that the firing of an RTO alarm is not considered a strong enough signal of packet loss, so does not result in an -immediate change to CWND or recovery state. An RTO alarm fires only when -there's a prolonged period of network silence, which could be caused by a change -in the underlying network RTT. +immediate change to congestion window or recovery state. An RTO alarm fires only +when there's a prolonged period of network silence, which could be caused by a +change in the underlying network RTT. QUIC also diverges from TCP by including MaxAckDelay in the RTO period. QUIC is able to explicitly model the ack delay via the ack delay field in the ack frame. -Because QUIC is subtracting this delay from both SRTT and it is expected to -reduce RTTVar, it is necessary to add it back, in order to compensate for -the smaller SRTT and RTTVar. +Since QUIC corrects for this delay in its SRTT and RTTVAR computations, it is +necessary to add this delay explicitly in the RTO computation. When an acknowledgment is received for a packet sent on an RTO event, any unacknowledged packets with lower packet numbers than those acknowledged MUST be From 69547db50be6e4441b8d70bf17a42fd771915565 Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 16:22:02 -0500 Subject: [PATCH 12/13] Update draft-ietf-quic-recovery.md --- draft-ietf-quic-recovery.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index 940b343ae0..b2fcded32f 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -312,8 +312,8 @@ they're too large and excludes those which reference an ack-only packet. QUIC diverges from TCP by calculating MaxAckDelay dynamically, instead of assuming a constant delayed ack timeout for all connections. QUIC includes this in all probe timeouts, because it assume the ack delay may come into play, -regardless of the number of packets outstanding. TCP's TLP assumes if at least 2 -packets are outstanding, acks will not be delayed. +regardless of the number of packets outstanding. TCP's TLP assumes if at least +2 packets are outstanding, acks will not be delayed. A PTO value of at least 1.5*SRTT ensures that the ACK is overdue. The 1.5 is based on {{?LOSS-PROBE=I-D.dukkipati-tcpm-tcp-loss-probe}}, but From ba5534853ad3e685d3d7ddf3f1e83ef3a628a72f Mon Sep 17 00:00:00 2001 From: ianswett Date: Tue, 5 Dec 2017 18:30:03 -0500 Subject: [PATCH 13/13] s/ack/ACK --- draft-ietf-quic-recovery.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/draft-ietf-quic-recovery.md b/draft-ietf-quic-recovery.md index b2fcded32f..1ab5d251d4 100644 --- a/draft-ietf-quic-recovery.md +++ b/draft-ietf-quic-recovery.md @@ -305,7 +305,7 @@ conditions: * If RTO ({{rto}}) is earlier, schedule a TLP alarm in its place. That is, PTO SHOULD be scheduled for min(RTO, PTO). -MaxAckDelay is the maximum ack delay supplied in an incoming ack frame. +MaxAckDelay is the maximum ack delay supplied in an incoming ACK frame. MaxAckDelay excludes ack delays that aren't included in an RTT sample because they're too large and excludes those which reference an ack-only packet. @@ -372,9 +372,10 @@ when there's a prolonged period of network silence, which could be caused by a change in the underlying network RTT. QUIC also diverges from TCP by including MaxAckDelay in the RTO period. QUIC is -able to explicitly model the ack delay via the ack delay field in the ack frame. -Since QUIC corrects for this delay in its SRTT and RTTVAR computations, it is -necessary to add this delay explicitly in the RTO computation. +able to explicitly model delay at the receiver via the ack delay field in the +ACK frame. Since QUIC corrects for this delay in its SRTT and RTTVAR +computations, it is necessary to add this delay explicitly in the TLP and RTO +computation. When an acknowledgment is received for a packet sent on an RTO event, any unacknowledged packets with lower packet numbers than those acknowledged MUST be @@ -486,7 +487,7 @@ largest_sent_packet: : The packet number of the most recently sent packet. largest_acked_packet: -: The largest packet number acknowledged in an ack frame. +: The largest packet number acknowledged in an ACK frame. latest_rtt: : The most recent RTT measurement made when receiving an ack for @@ -503,7 +504,7 @@ min_rtt: : The minimum RTT seen in the connection, ignoring ack delay. max_ack_delay: -: The maximum ack delay in an incoming ack frame for this connection. +: The maximum ack delay in an incoming ACK frame for this connection. Excludes ack delays for ack only packets and those that create an RTT sample less than min_rtt. @@ -914,7 +915,7 @@ bytes_in_flight: : The sum of the size in bytes of all sent packets that contain at least one retransmittable or PADDING frame, and have not been acked or declared lost. The size does not include IP or UDP overhead. - Packets only containing ack frames do not count towards byte_in_flight + Packets only containing ACK frames do not count towards byte_in_flight to ensure congestion control does not impede congestion feedback. congestion_window: