Handling of CVE-2023-44487 / HTTP2 Rapid Reset

[lkarsten]

Hi. Brought this up on IRC, but I think there should be some public record of it as well.

How well does Varnish handle the Rapid Reset attack?

Consensus seems to be that some new rate limiting should be added to Varnish.

Links:

• https://ubuntu.com/security/CVE-2023-44487
• https://blog.qualys.com/vulnerabilities-threat-research/2023/10/10/cve-2023-44487-http-2-rapid-reset-attack

Related fixes:

• h2o/h2o@28fe151

[bsdphk]

Or maybe we should simply violate this part of RFC7540 ?

SETTINGS_MAX_CONCURRENT_STREAMS (0x3): Indicates the maximum number
of concurrent streams that the sender will allow. This limit is
directional: it applies to the number of streams that the sender
permits the receiver to create. Initially, there is no limit to
this value. [...]

I'm pretty sure that a limit of 10 or 20 would cater for all non-hostile uses.

[kjetilho]

I'm pretty sure that a limit of 10 or 20 would cater for all non-hostile uses.

I think this was what Cloudflare had to back down on?

While responding to DoS attacks enabled by CVE-2023-44487, Cloudflare reduced maximum stream concurrency to 64. Before making this change, we were unaware that clients don't wait for SETTINGS and instead assume a concurrency of 100. Some web pages, such as an image gallery, do indeed cause a browser to send 100 requests immediately at the start of a connection. Unfortunately, the 36 streams above our limit all needed to be reset, which triggered our counting mitigations. This meant that we closed connections on legitimate clients, leading to a complete page load failure. As soon as we realized this interoperability issue, we changed the maximum stream concurrency to 100.

However, they employed blacklisting for IP of (inaccurately perceived) hostile users. Just sending resets without further punishment will not have the same kind of impact on user experience.

[jailbird777]

Looks like ATS went for rate limiting also: apache/trafficserver@8316188

[daghf]

I'm pretty sure that a limit of 10 or 20 would cater for all non-hostile uses.

SETTINGS_MAX_CONCURRENT_STREAMS is not a viable mitigation. The issue is that only streams that are in the "open" or either of the "half-closed" states will count towards this limit. Receving an RST puts the stream in state "closed", which is not counted towards SETTINGS_MAX_CONCURRENT_STREAMS.

Thus an attacker can endlessly spam us with new streams+RST regardless of the value of SETTINGS_MAX_CONCURRENT_STREAMS, and still be safely within the confines set by the h/2 spec ..

BTW we've done some experimentation with a rate-limiting facility for RSTs that looks very promising as a mitigation. I will open a PR here soon.

[Dridi]

We currently have two complementary mitigations:

• h2: Add a rate limit facility for h/2 RST handling ("Rapid reset" mitigation) #3997
• vcl_vrt: Skip VCL execution if the client is gone #3998

[TomasKorbar]

Thanks a lot guys for your work on this, it is really appreciated. Do you have any idea when these 2 changes could be merged?

[nigoroll]

@TomasKorbar during bugwash today we agreed that we want to review again tomorrow and merge what is going to be merged for patch releases soon after. #3998 is planned to be merged even before the review session.

[nigoroll]

We made good progress today, but we are not done. #3998 should get in next, #3999 will be discussed tomorrow.

[nigoroll]

All code identified as relevant up to now has been merged. See 3de5783 for an overview

[Dridi]

Branches where we will land the mitigations:

• trunk
  • h2: Add a rate limit facility for h/2 RST handling ("Rapid reset" mitigation) #3997
  • vcl_vrt: Skip VCL execution if the client is gone #3998
  • Add vmod_h2 to control http rapid reset rate limits per session #3999
  • Stabilize r03996.vtc
  • Update targeted platforms via circleci config
  • changelog
• 6.0 LTS
  • h2: Integrate h2_rxbuf_storage (6.0) #4004
  • vcl_vrt: Skip VCL execution if the client is gone (6.0) #4006
  • h2: Add a rate limit facility for h/2 RST handling (6.0) #4008
  • Update targeted platforms via circleci config
  • changelog
• 7.4
  • h2: Rapid reset mitigations (7.4) #4009
  • Update targeted platforms via circleci config
  • changelog
• 7.3
  • Collect bug fixes for 7.3.1 #3922
  • h2: Rapid reset mitigations (7.3) #4011
  • Update targeted platforms via circleci config
  • changelog

[Dridi]

I took care of all the ports to older branches.

[nigoroll]

Comments from reviewing #4009:

• Should we also add 325faac and related?

[Dridi]

This is part of a larger patch series, let me have a look.

[TomasKorbar]

Thanks a lot guys, for such a swift reaction, really appreciate it.

[Dridi]

https://varnish-cache.org/#varnish-http-2-rapid-reset-attack