Store an alt name with its entry in one allocation

[Frauschi]

One of six independent branches that cut allocations in the TLS and certificate layers. Self-contained and touches no TLS code.

What changes

Parsing the subject alternative names of a certificate allocated a DNS_entry and then a second buffer for the name it points at - a shape the code itself flagged with a "consider one malloc" note. A chain carrying six names cost twelve allocations on every handshake that verifies it.

AltNameNewEx() allocates the entry with room for the name behind it and copies the name in, so a six name chain costs six allocations. wolfSSL_X509_add_altname_ex() uses the same helper.

The failure path in SetDNSEntry() now hands the entry to FreeAltNames() instead of freeing two buffers by hand, so any ipString or ridString a later step allocates is released with it. Nothing leaked before this - no in-tree path can fail after those buffers are built - it is hardening against a future failure path.

API note

An entry built this way reports nameStored == 0, which parsing already used for a name borrowed from the input DER. Either way the name is not a separate allocation and must not be freed on its own. Application code that walks WOLFSSL_X509->altNames or DecodedCert->altNames and frees name only when nameStored is set is unaffected; code that frees it unconditionally was already wrong.

AltNameNewEx() rejects a negative length, and a NULL name with a positive length, rather than leaving len covering bytes that were never written - which the old inline code did. SetDNSEntry() screens those arguments before allocating so a NULL return really does mean out of memory, rather than reporting a bad argument as MEMORY_E and sending a caller that retries on MEMORY_E round forever. wolfSSL_X509_add_altname_ex() bounds its word32 length against INT_MAX explicitly rather than relying on what an out-of-range value narrows to.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11342

Scan targets checked: wolfcrypt-bugs, wolfcrypt-rs-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m0plus

• FLASH: .text +44 B (+0.1%, 66,927 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m3

• FLASH: .text +32 B (+0.0%, 125,699 B / 262,144 B, total: 48% used)

gcc-arm-cortex-m4-crypto-only

• FLASH: .text +64 B (+0.0%, 178,026 B / 262,144 B, total: 68% used)

gcc-arm-cortex-m4-dtls13

• FLASH: .text +64 B (+0.0%, 186,300 B / 1,048,576 B, total: 18% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +64 B (+0.0%, 301,084 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +64 B (+0.0%, 241,697 B / 262,144 B, total: 92% used)

linuxkm-pie

• Data: __patchable_function_entries +8 B (+0.0%, 26,856 B)

linuxkm-standard

• Data: __patchable_function_entries +8 B (+0.0%, 49,672 B)
  No memory changes detected for:
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• stm32-sim-stm32h753

[Frauschi]

Jenkins retest this please - history lost.