Summary

Envoy HTTP/2 protocol stack is vulnerable to the "Rapid Reset" class of exploits, which send a sequence of HEADERS frames optionally followed by RST_STREAM frames.

Affected Component

HTTP/2 protocol stack

Details

In the "Rapid Reset" family of attacks the attacker sends a sequence of compressed HEADERS frames. The compression allows a high number of HEADERS frames to be sent in a few kilobytes of wire data, causing Envoy to process them all at the same time. The attacker then resets the newly created HTTP/2 streams by rapidly sending a sequence of RST_STREAM frames. The attack is repeated by sending more HEADERS frames. This causes a single connection to exhaust CPU resources, preventing other requests from making progress, which results in either elevated latency or request timeouts.

Other forms of attack the HEADERS frames contain malformed request specifically crafted to elicit an early error response (stream reset) from the proxy. In this case the attacker does not need to send RST_STREAM frames.

Impact

Denial of service through CPU exhaustion

Attack vector(s)

Specifically crafted sequence of HEADERS frames, optionally followed by RST_STREAM frames, from an untrusted downstream client.

Patches

Users should upgrade to versions 1.27.1, 1.26.5, 1.25.10 or 1.24.10 to mitigate the effects of the "Rapid Reset" attack.

Workarounds

None, apart from disabling HTTP/2 protocol for downstream connections.

Detection

The attack can be detected by observing elevated downstream_rq_http2_total or downstream_rq_http3_total counters without a substantial increase in the downstream_cx_active counter. In some known forms of attack the value of downstream_rq_rx_reset will be elevated, while during other known forms of attack the downstream_rq_5xx or downstream_rq_4xx will be elevated.

Mitigation Details

The mitigation adds a budget on the number of RST_STREAM frames received shortly after corresponding HEADERS frame. The overload.premature_reset_total_stream_count runtime value controls how many prematurely reset streams are tolerated before disconnecting the connection. The default value is 500 and it can be lowered based on your service DoS capacity. The overload.premature_reset_min_stream_lifetime_seconds controls the interval during which the RST_STREAM is considered premature. The default value is 1 second and can be increased based on the profile of the attack. Setting the runtime key envoy.restart_features.send_goaway_for_premature_rst_streams to false completely disables this check.

The patch also contains the change that improves fair sharing of CPU resources in the presence of the attack, allowing legitimate requests to make progress and avoid timeouts. It is disabled by default due to its high risk and limited qualification time. This mitigation can be enabled by setting the http.max_requests_per_io_cycle runtime value to 1 (a higher number can be used too, at the cost of effectiveness of mitigation).

Envoy's default limits on the number of outbound frames max_outbound_frames and max_outbound_control_frames can be lowered as well to improve mitigation of the attack that uses Envoy's early error response. The value of max_outbound_frames may be set to 1000 and max_outbound_control_frames to 500. See https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#config-core-v3-http2protocoloptions for more information.

References

https://cloud.google.com/blog/products/identity-security/how-it-works-the-novel-http2-rapid-reset-ddos-attack
https://www.nginx.com/blog/http-2-rapid-reset-attack-impacting-f5-nginx-products/?sf269548684=1
https://www.envoyproxy.io/docs/envoy/latest/configuration/best_practices/edge