Skip to content

Resolve OpenSSL 4.0 build issues - #13482

Closed
cmcfarlen wants to merge 2 commits into
apache:masterfrom
cmcfarlen:ossl4-compat
Closed

Resolve OpenSSL 4.0 build issues#13482
cmcfarlen wants to merge 2 commits into
apache:masterfrom
cmcfarlen:ossl4-compat

Conversation

@cmcfarlen

Copy link
Copy Markdown
Contributor

Fedora 45 ships OpenSSL 4.0, which breaks the ATS build. This supersedes #13440 by
@jeredfloyd, which had the right diagnosis but did not build on OpenSSL 1.1.1 and
missed several affected files. Jered is credited via Co-authored-by.

Resolves #13427.

Root cause

Diffing the actual headers from OpenSSL 1.1.1f and 4.0.1, exactly one API changes in
a way that can break compilation:

API 1.1.1 3.0 / 4.0
X509_NAME_get_index_by_NID takes X509_NAME * takes const X509_NAME *

Others changed only their return type to const in 4.0 (X509_get_subject_name,
X509_get_issuer_name, X509_NAME_get_entry, X509_NAME_ENTRY_get_data,
X509_get0_pubkey_bitstr), and ASN1_STRING became opaque. Receiving those into a
const-qualified variable is legal on every version.

So hardcoding const is required on OpenSSL 4 but illegal on 1.1.1 — which is why
#13440 failed the Ubuntu 20.04 and FreeBSD 13.1 jobs.

Approach

Let the type be deduced from the accessor rather than hardcoded — auto * for
locals, decltype(&X509_get_subject_name) for parameter and function-pointer types.
One source form compiles on 1.1.1, 3.x and 4.0 with no version macros and no
const_cast.

One case makes this mandatory rather than merely tidy: X509_getm_notBefore /
X509_getm_notAfter changed const in the opposite direction (they took
const X509 * in 1.1.1 and take X509 * in 4.0), so no hardcoded spelling works
everywhere.

Beyond #13440

These are also broken on OpenSSL 4 and were not covered by #13440:

  • plugins/lua/ts_lua_client_cert_helpers.h, plugins/lua/ts_lua_client_request.cc
  • src/cripts/Certs.cc, include/cripts/Certs.hpp — same opaque-ASN1_STRING
    issue that Resolve OpenSSL 4.0 build issues #13440 fixed in sslheaders, plus two function-pointer types
  • src/iocore/net/unit_tests/test_SSLDHParams.cc
  • example/plugins/c-api/client_context_dump/client_context_dump.cc
  • tests/tools/plugins/ssl_client_verify_test.cc

Two fixes here are not purely mechanical:

  • plugins/certifier/certifier.cc — the X509_NAME_dup approach needs the copy
    freed on the X509_NAME_add_entry_by_txt failure path and the dup result
    null-checked. Reworked onto the file's existing std::unique_ptr idiom.
  • test_SSLDHParams.cc — this relied on mutating the subject name in place.
    Once the accessor returns const you must dup and then call both
    X509_set_subject_name and X509_set_issuer_name, or the generated self-signed
    cert silently loses its CN.

Verification

Full builds with BUILD_EXPERIMENTAL_PLUGINS=ON and BUILD_REGRESSION_TESTING=ON,
Cripts enabled where the platform's fmt allows it, plus test_tscore and test_net:

Platform OpenSSL Compiler Build Tests
Fedora 45 4.0.1 gcc pass pass
Fedora 43 3.5.7 gcc pass pass
Ubuntu 20.04 1.1.1f clang-12 pass pass

All on linux/arm64.

Caveats:

  • test_SSLDHParams.cc is gated behind SSLLIB_IS_AT_LEAST_OPENSSL3, so that change
    is only exercised on 3.x/4.0.
  • Cripts could not be built on Ubuntu 20.04 — its fmt is 6.1.2 against the 8.1
    minimum — so the Cripts changes are verified on OpenSSL 3.5 and 4.0 only.
  • FreeBSD was covered by proxy (also OpenSSL 1.1.1), not natively.
  • src/tscore/unit_tests/test_X509HostnameValidator.cc exists but is not wired into
    any CMake target, so validate_hostname has no unit coverage. Pre-existing; left
    alone here.

Fedora 45 ships OpenSSL 4.0, which constifies several X509/ASN1 accessors and
makes ASN1_STRING opaque. Hardcoding const breaks OpenSSL 1.1.1, where
X509_NAME_get_index_by_NID() still takes a non-const X509_NAME, so deduce the
pointer and function-pointer types from the accessors instead. That keeps a
single source form building on 1.1.1, 3.x and 4.0 with no version macros.

Co-authored-by: Jered Floyd <jered@redhat.com>
Copilot AI review requested due to automatic review settings August 3, 2026 19:48
@cmcfarlen cmcfarlen added the Build work related to build configuration or environment label Aug 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates multiple ATS core components, plugins, examples, and tests to build cleanly across OpenSSL 1.1.1, 3.x, and 4.0 by avoiding direct access to newly-opaque ASN1 internals and by letting constness/function-pointer types be deduced from OpenSSL accessors (instead of hardcoding types that differ across versions).

Changes:

  • Replace direct ASN1_STRING field access (->data, ->length, ->type) with accessor APIs (ASN1_STRING_get0_data, ASN1_STRING_length, ASN1_STRING_type).
  • Update X509/X509_NAME usage to be const-correct across OpenSSL versions via auto * / decltype(...)-based deduction.
  • Adjust certificate-subject mutation patterns (dup + set back) where OpenSSL now exposes subject names as const.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
tests/tools/plugins/ssl_client_verify_test.cc Deduces subject name constness and updates CN extraction to use ASN1 accessors.
src/tscore/X509HostnameValidator.cc Switches hostname validation string checks to ASN1 accessor APIs and deduced subject name constness.
src/iocore/net/unit_tests/test_SSLDHParams.cc Updates self-signed cert generation to dup/replace subject name for OpenSSL 4 constness changes.
src/iocore/net/SSLUtils.cc Makes ASN1 string duplication const-correct and deduces subject name constness for CN extraction.
src/iocore/net/SSLNetVConnection.cc Adjusts debug helper signature to accept const X509_NAME *.
src/iocore/net/OCSPStapling.cc Replaces direct ASN1 buffer access with accessor APIs.
src/cripts/Certs.cc Updates certificate field extraction to use ASN1 accessors and deduced getter types.
include/cripts/Certs.hpp Defines deduced getter typedefs (decltype(&X509_get_...)) to keep function-pointer types portable across OpenSSL versions.
src/api/InkAPI.cc Updates CN extraction types and ASN1 access for OpenSSL 4 compatibility.
plugins/lua/ts_lua_client_cert_helpers.h Updates X509 name helper constness and uses ASN1 accessors for signature formatting.
plugins/experimental/txn_box/plugin/src/ts_util.cc Uses decltype(X509_get_subject_name(nullptr)) to deduce X509_NAME* constness for nid lookup.
plugins/experimental/sslheaders/expand.cc Makes issuer/subject name pointers const and uses ASN1 accessors for signature dump.
plugins/experimental/cert_reporting_tool/cert_reporting_tool.cc Makes subject name pointer const to match OpenSSL accessor changes.
plugins/certifier/certifier.cc Dups subject name before mutation and introduces an RAII type for X509_NAME.
example/plugins/c-api/client_context_dump/client_context_dump.cc Deduces subject name constness for compatibility.

Comment thread src/api/InkAPI.cc
Comment thread src/tscore/X509HostnameValidator.cc
Comment thread src/iocore/net/unit_tests/test_SSLDHParams.cc
Comment thread plugins/certifier/certifier.cc Outdated
Comment thread tests/tools/plugins/ssl_client_verify_test.cc Outdated
Drop the unnecessary const cast when duplicating an ASN1 string, assert the
subject CN is actually added in the DH params test, build the CN string
directly from the ASN1 buffer instead of strndup, and use an explicit deleter
for the X509_NAME unique_ptr rather than specializing std::default_delete.
Copilot AI review requested due to automatic review settings August 3, 2026 21:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (1)

tests/tools/plugins/ssl_client_verify_test.cc:78

  • The SAN (subjectAltName) GEN_DNS path still uses strndup()/free() and then passes the resulting char* to check_name(const std::string&). If strndup() returns nullptr (allocation failure), constructing a std::string from it is UB and can crash. Since you already have pointer+length from OpenSSL, you can avoid the allocation entirely by constructing a std::string directly (same approach as the CN path above).
      std::string subj_name{reinterpret_cast<const char *>(ASN1_STRING_get0_data(cn)), static_cast<size_t>(ASN1_STRING_length(cn))};
      retval = check_name(subj_name);
    }
  }
  if (!retval) {

@cmcfarlen cmcfarlen closed this Aug 3, 2026
@cmcfarlen cmcfarlen reopened this Aug 3, 2026
maskit added a commit to maskit/trafficserver that referenced this pull request Aug 3, 2026
certifier.cc freed the duplicated X509_NAME manually on every path;
switch to a scoped_X509_NAME unique_ptr matching the file existing
scoped_X509/scoped_EVP_PKEY/scoped_SSL_CTX aliases so no path can
forget to free it.

ts_util.cc templated ssl_value_for just to defer the parameter type
to the caller; decltype(X509_get_subject_name(nullptr)) deduces the
same pointer type directly without turning it into a template.

Also drop the last C-style cast this series introduced in
X509HostnameValidator.cc in favor of reinterpret_cast.
@cmcfarlen cmcfarlen closed this Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Build work related to build configuration or environment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10.2.x branch cannot build on Fedora 45 due to ASN1_STRING opacity

2 participants