Restrict the size of OBJECT IDENTIFIERs that OBJ_obj2txt will translate

OBJ_obj2txt() would translate any size OBJECT IDENTIFIER to canonical
numeric text form.  For gigantic sub-identifiers, this would take a very
long time, the time complexity being O(n^2) where n is the size of that
sub-identifier.

To mitigate this, a restriction on the size that OBJ_obj2txt() will
translate to canonical numeric text form is added, based on RFC 2578
(STD 58), which says this:

> 3.5. OBJECT IDENTIFIER values
>
> An OBJECT IDENTIFIER value is an ordered list of non-negative numbers.
> For the SMIv2, each number in the list is referred to as a sub-identifier,
> there are at most 128 sub-identifiers in a value, and each sub-identifier
> has a maximum value of 2^32-1 (4294967295 decimal).

Fixes otc/security#96
Fixes CVE-2023-2650

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>

Author: levitte

CHANGES.md

@@ -30,6 +30,32 @@ breaking changes, and mappings for the large list of deprecated functions.
 
 ### Changes between 3.0.8 and 3.0.9 [xx XXX xxxx]
 
+ * Mitigate for the time it takes for `OBJ_obj2txt` to translate gigantic
+   OBJECT IDENTIFIER sub-identifiers to canonical numeric text form.
+
+   OBJ_obj2txt() would translate any size OBJECT IDENTIFIER to canonical
+   numeric text form.  For gigantic sub-identifiers, this would take a very
+   long time, the time complexity being O(n^2) where n is the size of that
+   sub-identifier.  ([CVE-2023-2650])
+
+   To mitigitate this, `OBJ_obj2txt()` will only translate an OBJECT
+   IDENTIFIER to canonical numeric text form if the size of that OBJECT
+   IDENTIFIER is 586 bytes or less, and fail otherwise.
+
+   The basis for this restriction is RFC 2578 (STD 58), section 3.5. OBJECT
+   IDENTIFIER values, which stipulates that OBJECT IDENTIFIERS may have at
+   most 128 sub-identifiers, and that the maximum value that each sub-
+   identifier may have is 2^32-1 (4294967295 decimal).
+
+   For each byte of every sub-identifier, only the 7 lower bits are part of
+   the value, so the maximum amount of bytes that an OBJECT IDENTIFIER with
+   these restrictions may occupy is 32 * 128 / 7, which is approximately 586
+   bytes.
+
+   Ref: https://datatracker.ietf.org/doc/html/rfc2578#section-3.5
+
+   *Richard Levitte*
+
  * Fixed buffer overread in AES-XTS decryption on ARM 64 bit platforms which
    happens if the buffer size is 4 mod 5 in 16 byte AES blocks. This can
    trigger a crash of an application using AES-XTS decryption if the memory
@@ -19626,6 +19652,7 @@ ndif
 
 <!-- Links -->
 
+[CVE-2023-2650]: https://www.openssl.org/news/vulnerabilities.html#CVE-2023-2650
 [CVE-2023-1255]: https://www.openssl.org/news/vulnerabilities.html#CVE-2023-1255
 [CVE-2023-0466]: https://www.openssl.org/news/vulnerabilities.html#CVE-2023-0466
 [CVE-2023-0465]: https://www.openssl.org/news/vulnerabilities.html#CVE-2023-0465

NEWS.md

@@ -20,6 +20,8 @@ OpenSSL 3.0
 
 ### Major changes between OpenSSL 3.0.8 and OpenSSL 3.0.9 [under development]
 
+  * Mitigate for very slow `OBJ_obj2txt()` performance with gigantic OBJECT
+    IDENTIFIER sub-identities.  ([CVE-2023-2650])
   * Fixed buffer overread in AES-XTS decryption on ARM 64 bit platforms
     ([CVE-2023-1255])
   * Fixed documentation of X509_VERIFY_PARAM_add0_policy() ([CVE-2023-0466])
@@ -1439,6 +1441,8 @@ OpenSSL 0.9.x
   * Support for various new platforms
 
 <!-- Links -->
+
+[CVE-2023-2650]: https://www.openssl.org/news/vulnerabilities.html#CVE-2023-2650
 [CVE-2023-1255]: https://www.openssl.org/news/vulnerabilities.html#CVE-2023-1255
 [CVE-2023-0466]: https://www.openssl.org/news/vulnerabilities.html#CVE-2023-0466
 [CVE-2023-0465]: https://www.openssl.org/news/vulnerabilities.html#CVE-2023-0465

crypto/objects/obj_dat.c

@@ -443,6 +443,25 @@ int OBJ_obj2txt(char *buf, int buf_len, const ASN1_OBJECT *a, int no_name)
     first = 1;
     bl = NULL;
 
+    /*
+     * RFC 2578 (STD 58) says this about OBJECT IDENTIFIERs:
+     *
+     * > 3.5. OBJECT IDENTIFIER values
+     * >
+     * > An OBJECT IDENTIFIER value is an ordered list of non-negative
+     * > numbers. For the SMIv2, each number in the list is referred to as a
+     * > sub-identifier, there are at most 128 sub-identifiers in a value,
+     * > and each sub-identifier has a maximum value of 2^32-1 (4294967295
+     * > decimal).
+     *
+     * So a legitimate OID according to this RFC is at most (32 * 128 / 7),
+     * i.e. 586 bytes long.
+     *
+     * Ref: https://datatracker.ietf.org/doc/html/rfc2578#section-3.5
+     */
+    if (len > 586)
+        goto err;
+
     while (len > 0) {
         l = 0;
         use_bn = 0;