Issue summary: Some non-default TLS server configurations can cause unbounded
memory growth when processing TLSv1.3 sessions

Impact summary: An attacker may exploit certain server configurations to trigger
unbounded memory growth that would lead to a Denial of Service

This problem can occur in TLSv1.3 if the non-default SSL_OP_NO_TICKET option is
being used (but not if early_data support is also configured and the default
anti-replay protection is in use). In this case, under certain conditions, the
session cache can get into an incorrect state and it will fail to flush properly
as it fills. The session cache will continue to grow in an unbounded manner. A
malicious client could deliberately create the scenario for this failure to
force a Denial of Service. It may also happen by accident in normal operation.

This issue only affects TLS servers supporting TLSv1.3. It does not affect TLS
clients.

The FIPS modules in 3.2, 3.1 and 3.0 are not affected by this issue. OpenSSL
1.0.2 is also not affected by this issue.

References

• https://nvd.nist.gov/vuln/detail/CVE-2024-2511
• openssl/openssl@7e4d731
• openssl/openssl@b52867a
• openssl/openssl@e9d7083
• https://github.openssl.org/openssl/extended-releases/commit/5f8d25770ae6437db119dfc951e207271a326640
• https://www.openssl.org/news/secadv/20240408.txt
• http://www.openwall.com/lists/oss-security/2024/04/08/5
• https://security.netapp.com/advisory/ntap-20240503-0013