client: extend custom CA handling across HTTPS and websocket clients by joshka-oai · Pull Request #14239 · openai/codex

joshka-oai · 2026-03-10T18:46:58Z

Stacked PRs

This work is now effectively split across two steps:

login: add custom CA support for login flows #14178: add custom CA support for browser and device-code login flows, docs, and hermetic subprocess tests
client: extend custom CA handling across HTTPS and websocket clients #14239: extend that shared custom CA handling across Codex HTTPS clients and secure websocket TLS

Note: #14240 was merged into this branch while it was stacked on top of this PR. This PR now subsumes that websocket follow-up and should be treated as the combined change.

Builds on top of #14178.

Problem

Custom CA support landed first in the login path, but the real requirement is broader. Codex constructs outbound TLS clients in multiple places, and both HTTPS and secure websocket paths can fail behind enterprise TLS interception if they do not honor CODEX_CA_CERTIFICATE or SSL_CERT_FILE consistently.

This PR broadens the shared custom-CA logic beyond login and applies the same policy to websocket TLS, so the enterprise-proxy story is no longer split between “HTTPS works” and “websockets still fail”.

What This Delivers

Custom CA support is no longer limited to login. Codex outbound HTTPS clients and secure websocket connections can now honor the same CODEX_CA_CERTIFICATE / SSL_CERT_FILE configuration, so enterprise proxy/intercept setups work more consistently end-to-end.

For users and operators, nothing new needs to be configured beyond the same CA env vars introduced in #14178. The change is that more of Codex now respects them, including websocket-backed flows that were previously still using default trust roots.

I also manually validated the proxy path locally with mitmproxy using:
CODEX_CA_CERTIFICATE=~/.mitmproxy/mitmproxy-ca-cert.pem HTTPS_PROXY=http://127.0.0.1:8080 just codex
with mitmproxy installed via brew install mitmproxy and configured as the macOS system proxy.

Mental model

codex-client is now the owner of shared custom-CA policy for outbound TLS client construction. Reqwest callers start from the builder configuration they already need, then pass that builder through build_reqwest_client_with_custom_ca(...). Websocket callers ask the same module for a rustls client config when a custom CA bundle is configured.

The env precedence is the same everywhere:

CODEX_CA_CERTIFICATE wins
otherwise fall back to SSL_CERT_FILE
otherwise use system roots

The helper is intentionally narrow. It loads every usable certificate from the configured PEM bundle into the appropriate root store and returns either a configured transport or a typed error that explains what went wrong.

Non-goals

This does not add handshake-level integration tests against a live TLS endpoint. It does not validate that the configured bundle forms a meaningful certificate chain. It also does not try to force every transport in the repo through one abstraction; it extends the shared CA policy across the reqwest and websocket paths that actually needed it.

Tradeoffs

The main tradeoff is centralizing CA behavior in codex-client while still leaving adoption up to call sites. That keeps the implementation additive and reviewable, but it means the rule "outbound Codex TLS that should honor enterprise roots must use the shared helper" is still partly enforced socially rather than by types.

For websockets, the shared helper only builds an explicit rustls config when a custom CA bundle is configured. When no override env var is set, websocket callers still use their ordinary default connector path.

Architecture

codex-client::custom_ca now owns CA bundle selection, PEM normalization, mixed-section parsing, certificate extraction, typed CA-loading errors, and optional rustls client-config construction for websocket TLS.

The affected consumers now call into that shared helper directly rather than carrying login-local CA behavior:

backend-client
cloud-tasks
RMCP client paths that use reqwest
TUI voice HTTP paths
codex-core default reqwest client construction
codex-api websocket clients for both responses and realtime websocket connections

The subprocess CA probe, env-sensitive integration tests, and shared PEM fixtures also live in codex-client, which is now the actual owner of the behavior they exercise.

Observability

The shared CA path logs:

which environment variable selected the bundle
which path was loaded
how many certificates were accepted
when TRUSTED CERTIFICATE labels were normalized
when CRLs were ignored
where client construction failed

Returned errors remain user-facing and include the relevant env var, path, and remediation hint. That same error model now applies whether the failure surfaced while building a reqwest client or websocket TLS configuration.

Tests

Pure unit tests in codex-client cover env precedence and PEM normalization behavior. Real client construction remains in subprocess tests so the suite can control process env and avoid the macOS seatbelt panic path that motivated the hermetic test split.

The subprocess coverage verifies:

CODEX_CA_CERTIFICATE precedence over SSL_CERT_FILE
fallback to SSL_CERT_FILE
single-cert and multi-cert bundles
malformed and empty-file errors
OpenSSL TRUSTED CERTIFICATE handling
CRL tolerance for well-formed CRL sections

The websocket side is covered by the existing codex-api / codex-core websocket test suites plus the manual mitmproxy validation above.

xl-openai · 2026-03-12T21:43:01Z

codex-rs/codex-client/src/custom_ca.rs

-    build_login_http_client_with_env(&ProcessEnv, reqwest::Client::builder())
+/// Returns a [`BuildReqwestClientError`] when the configured CA file is unreadable, malformed, or
+/// contains a certificate block that `reqwest` cannot register as a root.
+pub fn build_reqwest_client_with_custom_ca(


Do we want a more generic entry point for constructing a reqwest client, in case we need to add more decorators later?

Not sure what would be a decorator that we'd add there, but we're passing in the ClientBuilder. The builder can be pre-configured with whatever you need in the extra info. Also, code is cheap and this is fairly deep in the core (not really user facing - implementation deatil). I think if we need to augment it with other bits, telling codex to do that would be fairly simple.

I think probably ok to not make this fully open ended on extension. WDYT?

## Stacked PRs This work is split across three stacked PRs: - #14178: add custom CA support for browser and device-code login flows, docs, and hermetic subprocess tests - #14239: broaden the shared custom CA path from login to other outbound `reqwest` clients across Codex - #14240: extend that shared custom CA handling to secure websocket TLS so websocket connections honor the same CA env vars Review order: #14178, then #14239, then #14240. Builds on top of #14239, which itself builds on #14178. ## Problem The shared custom-CA path covered `reqwest` clients after #14239, but secure websocket connections still used tungstenite's default TLS connector. In enterprise MITM setups, that meant HTTPS requests could succeed while websocket connections still failed because they were not loading the same custom root CA bundle. ## What This Delivers Secure websocket connections now honor the same custom CA configuration as the HTTPS clients introduced in the earlier stacked PRs. After this lands, setups that already work for HTTPS behind an intercepting proxy can also work for websocket-backed features instead of failing later during websocket TLS setup. For users and operators, the configuration does not change: `CODEX_CA_CERTIFICATE` wins, then `SSL_CERT_FILE`, then system roots. The difference is that websocket TLS now participates in that same policy. ## Mental model There is now one shared custom-CA policy for both HTTPS and secure websocket connections. `reqwest` callers continue to use `build_reqwest_client_with_custom_ca(...)`. Websocket callers now ask `codex-client` for a rustls client config when a custom CA bundle is configured, then pass that config into tungstenite explicitly. The env precedence remains the same: - `CODEX_CA_CERTIFICATE` wins - otherwise fall back to `SSL_CERT_FILE` - otherwise use system roots ## Non-goals This does not add a live end-to-end TLS interception test. It does not change the fallback behavior when no custom CA env var is set. It also does not try to generalize all websocket transport concerns into `codex-client`; it only extends the shared CA-loading policy to the websocket TLS boundary. ## Tradeoffs The main tradeoff is that websocket callers now have an explicit shared rustls-config path only when a custom CA bundle is configured. That keeps the normal no-override path simple, but it means the websocket implementation still has two TLS setup paths: default connector when no override is set, explicit connector when it is. This PR also renames the shared CA error type to match reality. The error is no longer reqwest-specific, because the same CA-loading failures now surface from websocket TLS configuration too. ## Architecture `codex-client::custom_ca` now exposes an optional rustls client-config builder alongside the existing reqwest client builder. The websocket clients in `codex-api` now consume that shared config: - responses websocket connections - realtime websocket connections When no custom CA env var is set, websocket callers still use their ordinary default connector path. When a custom CA bundle is configured, they build an explicit rustls connector with the same roots that the shared HTTPS path uses. ## Observability The websocket path now inherits the same CA-loading logs and user-facing errors as the shared HTTPS path. Failures to read, parse, or register custom CA certificates are surfaced before websocket TLS is attempted, instead of failing later as an opaque certificate-validation problem. ## Tests This PR relies on the existing `codex-client` CA tests plus the websocket-focused suites in `codex-api` and `codex-core`. Automated coverage run for this stack: - `cargo test -p codex-client -p codex-api` - `cargo test -p codex-core websocket_fallback -- --nocapture` Manual validation: - `CODEX_CA_CERTIFICATE=~/.mitmproxy/mitmproxy-ca-cert.pem HTTPS_PROXY=http://127.0.0.1:8080 just codex` - with `mitmproxy` installed via `brew install mitmproxy` - and mitmproxy also configured as the system proxy in macOS Wi-Fi settings That manual check was specifically useful because it exercises the websocket path behind a real intercepting proxy instead of only the reqwest HTTPS path. Co-authored-by: Codex <noreply@openai.com>

@3axap4eHko

## Stacked PRs This work is split across three stacked PRs: - #14178: add custom CA support for browser and device-code login flows, docs, and hermetic subprocess tests - #14239: broaden the shared custom CA path from login to other outbound `reqwest` clients across Codex - #14240: extend that shared custom CA handling to secure websocket TLS so websocket connections honor the same CA env vars Review order: #14178, then #14239, then #14240. Supersedes #6864. Thanks to @3axap4eHko for the original implementation and investigation here. Although this version rearranges the code and history significantly, the majority of the credit for this work belongs to them. ## Problem Login flows need to work in enterprise environments where outbound TLS is intercepted by an internal proxy or gateway. In those setups, system root certificates alone are often insufficient to validate the OAuth and device-code endpoints used during login. The change adds a login-specific custom CA loading path, but the important contracts around env precedence, PEM compatibility, test boundaries, and probe-only workarounds need to be explicit so reviewers can understand what behavior is intentional. For users and operators, the behavior is simple: if login needs to trust a custom root CA, set `CODEX_CA_CERTIFICATE` to a PEM file containing one or more certificates. If that variable is unset, login falls back to `SSL_CERT_FILE`. If neither is set, login uses system roots. Invalid or empty PEM files now fail with an error that points back to those environment variables and explains how to recover. ## What This Delivers Users can now make Codex login work behind enterprise TLS interception by pointing `CODEX_CA_CERTIFICATE` at a PEM bundle containing the relevant root certificates. If that variable is unset, login falls back to `SSL_CERT_FILE`, then to system roots. This PR applies that behavior to both browser-based and device-code login flows. It also makes login tolerant of the PEM shapes operators actually have in hand: multi-certificate bundles, OpenSSL `TRUSTED CERTIFICATE` labels, and bundles that include well-formed CRLs. ## Mental model `codex-login` is the place where the login flows construct ad hoc outbound HTTP clients. That makes it the right boundary for a narrow CA policy: look for `CODEX_CA_CERTIFICATE`, fall back to `SSL_CERT_FILE`, load every parseable certificate block in that bundle into a `reqwest::Client`, and fail early with a clear user-facing error if the bundle is unreadable or malformed. The implementation is intentionally pragmatic about PEM input shape. It accepts ordinary certificate bundles, multi-certificate bundles, OpenSSL `TRUSTED CERTIFICATE` labels, and bundles that also contain CRLs. It does not validate a certificate chain or prove a handshake; it only constructs the root store used by login. ## Non-goals This change does not introduce a general-purpose transport abstraction for the rest of the product. It does not validate whether the provided bundle forms a real chain, and it does not add handshake-level integration tests against a live TLS server. It also does not change login state management or OAuth semantics beyond ensuring the existing flows share the same CA-loading rules. ## Tradeoffs The main tradeoff is keeping this logic scoped to login-specific client construction rather than lifting it into a broader shared HTTP layer. That keeps the review surface smaller, but it also means future login-adjacent code must continue to use `build_login_http_client()` or it can silently bypass enterprise CA overrides. The `TRUSTED CERTIFICATE` handling is also intentionally a local compatibility shim. The rustls ecosystem does not currently accept that PEM label upstream, so the code normalizes it locally and trims the OpenSSL `X509_AUX` trailer bytes down to the certificate DER that `reqwest` can consume. ## Architecture `custom_ca.rs` is now the single place that owns login CA behavior. It selects the CA file from the environment, reads it, normalizes PEM label shape where needed, iterates mixed PEM sections with `rustls-pki-types`, ignores CRLs, trims OpenSSL trust metadata when necessary, and returns either a configured `reqwest::Client` or a typed error. The browser login server and the device-code flow both call `build_login_http_client()`, so they share the same trust-store policy. Environment-sensitive tests run through the `login_ca_probe` helper binary because those tests must control process-wide env vars and cannot reliably build a real reqwest client in-process on macOS seatbelt runs. ## Observability The custom CA path logs which environment variable selected the bundle, which file path was loaded, how many certificates were accepted, when `TRUSTED CERTIFICATE` labels were normalized, when CRLs were ignored, and where client construction failed. Returned errors remain user-facing and include the relevant path, env var, and remediation hint. This gives enough signal for three audiences: - users can see why login failed and which env/file caused it - sysadmins can confirm which override actually won - developers can tell whether the failure happened during file read, PEM parsing, certificate registration, or final reqwest client construction ## Tests Pure unit tests stay limited to env precedence and empty-value handling. Real client construction lives in subprocess tests so the suite remains hermetic with respect to process env and macOS sandbox behavior. The subprocess tests verify: - `CODEX_CA_CERTIFICATE` precedence over `SSL_CERT_FILE` - fallback to `SSL_CERT_FILE` - single-certificate and multi-certificate bundles - malformed and empty-bundle errors - OpenSSL `TRUSTED CERTIFICATE` handling - CRL tolerance for well-formed CRL sections The named PEM fixtures under `login/tests/fixtures/` are shared by the tests so their purpose stays reviewable. --------- Co-authored-by: Ivan Zakharchanka <3axap4eHko@gmail.com> Co-authored-by: Codex <noreply@openai.com>

Enterprise TLS inspection proxies use custom roots, so OAuth token exchanges fail when we only trust system CAs. This change switches PEM parsing to rustls-pki-types (multi-cert bundles included) and surfaces clearer, user-facing errors that explain how to fix invalid or empty CA files via CODEX_CA_CERTIFICATE/SSL_CERT_FILE. To avoid cross-test races with process-wide env vars, CA path selection now uses a small EnvSource abstraction in unit tests, and environment-dependent behavior is verified via an assert_cmd-driven login_ca_probe helper binary. This keeps normal tests isolated while still validating env precedence and error messaging. Also updates login dev-deps (assert_cmd/pretty_assertions), removes serial_test, and re-exports build_login_http_client for the probe helper. Co-authored-by: Codex <noreply@openai.com>

Route login's reqwest client construction through a shared custom CA loader so browser and device-code authentication can honor CODEX_CA_CERTIFICATE and SSL_CERT_FILE. This supports multi-certificate bundles, OpenSSL TRUSTED CERTIFICATE compatibility, CRL skipping, and named PEM fixtures used by subprocess integration tests. Co-authored-by: Codex <noreply@openai.com>

Route the spawned login_ca_probe binary through a hidden probe_support entry point that disables reqwest proxy autodetection. This keeps the subprocess CA tests focused on custom bundle loading instead of the system-configuration panic that can happen under macOS seatbelt, and documents why the workaround stays out of the normal public API. Co-authored-by: Codex <noreply@openai.com>

Move the shared custom CA loader into codex-client and route external reqwest callers through it. This broadens CODEX_CA_CERTIFICATE / SSL_CERT_FILE support beyond login and moves the env-sensitive tests to the owning crate. Co-authored-by: Codex <noreply@openai.com>

## Stacked PRs This work is split across three stacked PRs: - #14178: add custom CA support for browser and device-code login flows, docs, and hermetic subprocess tests - #14239: broaden the shared custom CA path from login to other outbound `reqwest` clients across Codex - #14240: extend that shared custom CA handling to secure websocket TLS so websocket connections honor the same CA env vars Review order: #14178, then #14239, then #14240. Builds on top of #14239, which itself builds on #14178. ## Problem The shared custom-CA path covered `reqwest` clients after #14239, but secure websocket connections still used tungstenite's default TLS connector. In enterprise MITM setups, that meant HTTPS requests could succeed while websocket connections still failed because they were not loading the same custom root CA bundle. ## What This Delivers Secure websocket connections now honor the same custom CA configuration as the HTTPS clients introduced in the earlier stacked PRs. After this lands, setups that already work for HTTPS behind an intercepting proxy can also work for websocket-backed features instead of failing later during websocket TLS setup. For users and operators, the configuration does not change: `CODEX_CA_CERTIFICATE` wins, then `SSL_CERT_FILE`, then system roots. The difference is that websocket TLS now participates in that same policy. ## Mental model There is now one shared custom-CA policy for both HTTPS and secure websocket connections. `reqwest` callers continue to use `build_reqwest_client_with_custom_ca(...)`. Websocket callers now ask `codex-client` for a rustls client config when a custom CA bundle is configured, then pass that config into tungstenite explicitly. The env precedence remains the same: - `CODEX_CA_CERTIFICATE` wins - otherwise fall back to `SSL_CERT_FILE` - otherwise use system roots ## Non-goals This does not add a live end-to-end TLS interception test. It does not change the fallback behavior when no custom CA env var is set. It also does not try to generalize all websocket transport concerns into `codex-client`; it only extends the shared CA-loading policy to the websocket TLS boundary. ## Tradeoffs The main tradeoff is that websocket callers now have an explicit shared rustls-config path only when a custom CA bundle is configured. That keeps the normal no-override path simple, but it means the websocket implementation still has two TLS setup paths: default connector when no override is set, explicit connector when it is. This PR also renames the shared CA error type to match reality. The error is no longer reqwest-specific, because the same CA-loading failures now surface from websocket TLS configuration too. ## Architecture `codex-client::custom_ca` now exposes an optional rustls client-config builder alongside the existing reqwest client builder. The websocket clients in `codex-api` now consume that shared config: - responses websocket connections - realtime websocket connections When no custom CA env var is set, websocket callers still use their ordinary default connector path. When a custom CA bundle is configured, they build an explicit rustls connector with the same roots that the shared HTTPS path uses. ## Observability The websocket path now inherits the same CA-loading logs and user-facing errors as the shared HTTPS path. Failures to read, parse, or register custom CA certificates are surfaced before websocket TLS is attempted, instead of failing later as an opaque certificate-validation problem. ## Tests This PR relies on the existing `codex-client` CA tests plus the websocket-focused suites in `codex-api` and `codex-core`. Automated coverage run for this stack: - `cargo test -p codex-client -p codex-api` - `cargo test -p codex-core websocket_fallback -- --nocapture` Manual validation: - `CODEX_CA_CERTIFICATE=~/.mitmproxy/mitmproxy-ca-cert.pem HTTPS_PROXY=http://127.0.0.1:8080 just codex` - with `mitmproxy` installed via `brew install mitmproxy` - and mitmproxy also configured as the system proxy in macOS Wi-Fi settings That manual check was specifically useful because it exercises the websocket path behind a real intercepting proxy instead of only the reqwest HTTPS path. Co-authored-by: Codex <noreply@openai.com>

joshka-oai mentioned this pull request Mar 10, 2026

client: apply custom CA handling to websocket TLS #14240

Merged

joshka-oai force-pushed the joshka/login-custom-ca-rework branch from e3487e3 to 3dcb290 Compare March 10, 2026 19:01

joshka-oai mentioned this pull request Mar 10, 2026

login: add custom CA support for login flows #14178

Merged

joshka-oai force-pushed the joshka/login-custom-ca-shared-clients branch 3 times, most recently from 18917a9 to 9490701 Compare March 10, 2026 19:33

joshka-oai force-pushed the joshka/login-custom-ca-rework branch from 3dcb290 to 08a39d7 Compare March 10, 2026 19:33

xl-openai reviewed Mar 12, 2026

View reviewed changes

pakrym-oai approved these changes Mar 12, 2026

View reviewed changes

joshka-oai force-pushed the joshka/login-custom-ca-rework branch from 08a39d7 to 334d227 Compare March 12, 2026 23:34

joshka-oai force-pushed the joshka/login-custom-ca-shared-clients branch from 9490701 to e6446a9 Compare March 12, 2026 23:34

Base automatically changed from joshka/login-custom-ca-rework to main March 13, 2026 00:14

3axap4eHko and others added 7 commits March 12, 2026 17:22

Add custom CA support for login flows

ff4442f

docs: add CA certificate environment variable documentation

31d4e63

joshka-oai force-pushed the joshka/login-custom-ca-shared-clients branch from b5f3e63 to 8566ecf Compare March 13, 2026 00:23

joshka-oai changed the title ~~client: share custom CA handling across outbound reqwest clients~~ client: extend custom CA handling across HTTPS and websocket clients Mar 13, 2026

joshka-oai enabled auto-merge (squash) March 13, 2026 00:25

joshka-oai merged commit 6912da8 into main Mar 13, 2026
52 of 54 checks passed

joshka-oai deleted the joshka/login-custom-ca-shared-clients branch March 13, 2026 00:59

github-actions bot locked and limited conversation to collaborators Mar 13, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

client: extend custom CA handling across HTTPS and websocket clients#14239

client: extend custom CA handling across HTTPS and websocket clients#14239
joshka-oai merged 7 commits intomainfrom
joshka/login-custom-ca-shared-clients

joshka-oai commented Mar 10, 2026 •

edited

Loading

Uh oh!

xl-openai Mar 12, 2026

Uh oh!

joshka-oai Mar 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

joshka-oai commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stacked PRs

Problem

What This Delivers

Mental model

Non-goals

Tradeoffs

Architecture

Observability

Tests

Uh oh!

xl-openai Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

joshka-oai Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

joshka-oai commented Mar 10, 2026 •

edited

Loading