Fix unconstrained session cache growth in TLSv1.3

In TLSv1.3 we create a new session object for each ticket that we send.
We do this by duplicating the original session. If SSL_OP_NO_TICKET is in
use then the new session will be added to the session cache. However, if
early data is not in use (and therefore anti-replay protection is being
used), then multiple threads could be resuming from the same session
simultaneously. If this happens and a problem occurs on one of the threads,
then the original session object could be marked as not_resumable. When we
duplicate the session object this not_resumable status gets copied into the
new session object. The new session object is then added to the session
cache even though it is not_resumable.

Subsequently, another bug means that the session_id_length is set to 0 for
sessions that are marked as not_resumable - even though that session is
still in the cache. Once this happens the session can never be removed from
the cache. When that object gets to be the session cache tail object the
cache never shrinks again and grows indefinitely.

CVE-2024-2511

Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from openssl#24044)

(cherry picked from commit 7e4d731)

ssl/ssl_lib.c

@@ -3557,9 +3557,10 @@ void ssl_update_cache(SSL *s, int mode)
 
     /*
      * If the session_id_length is 0, we are not supposed to cache it, and it
-     * would be rather hard to do anyway :-)
+     * would be rather hard to do anyway :-). Also if the session has already
+     * been marked as not_resumable we should not cache it for later reuse.
      */
-    if (s->session->session_id_length == 0)
+    if (s->session->session_id_length == 0 || s->session->not_resumable)
         return;
 
     /*

ssl/ssl_sess.c

@@ -195,6 +195,7 @@ SSL_SESSION *ssl_session_dup(SSL_SESSION *src, int ticket)
     } else {
         dest->ext.tick_lifetime_hint = 0;
         dest->ext.ticklen = 0;
+        dest->not_resumable &= ticket;
     }
 
     if (src->ext.alpn_selected != NULL) {

ssl/statem/statem_srvr.c

@@ -2419,9 +2419,8 @@ int tls_construct_server_hello(SSL *s, WPACKET *pkt)
      * so the following won't overwrite an ID that we're supposed
      * to send back.
      */
-    if (s->session->not_resumable ||
-        (!(s->ctx->session_cache_mode & SSL_SESS_CACHE_SERVER)
-         && !s->hit))
+    if (!(s->ctx->session_cache_mode & SSL_SESS_CACHE_SERVER)
+            && !s->hit)
         s->session->session_id_length = 0;
 
     if (usetls13) {