Always-on LD_LIBRARY_PATH stripping causes 11x performance regression for CUDA/MKL users

[johnzfitch]

What version of Codex is running?

0.00

What subscription do you have?

Pro

Which model were you using?

gpt-5.2

What platform is your computer?

Linux 6.17.9-arch1-1 x86_64

What issue are you seeing?

Pre-main hardening in release builds strips LD_LIBRARY_PATH unconditionally, causing severe performance regressions and broken functionality for:

• CUDA workloads: 11-300x slower (GPU libraries not found, fallback to CPU)
• Intel MKL workloads: 11x slower (optimized BLAS not found, slow fallback)
• Conda environments: Legacy installations relying on LD_LIBRARY_PATH fail
• Enterprise deployments: Custom DB drivers, Oracle clients fail to load
• HPC clusters: Module systems and custom toolchains broken

This regression was introduced in PR #4521 (commit b8e1fe6) which made codex_process_hardening::pre_main_hardening() always-on in release builds.

The hardening strips all LD_* environment variables before main(), and because Codex spawns child processes with the parent environment, every child inherits the stripped values.

What steps can reproduce the bug?

1. Install Codex release build (non-debug)
2. Set LD_LIBRARY_PATH to custom CUDA installation:
   export LD_LIBRARY_PATH=/opt/cuda/lib64:$LD_LIBRARY_PATH
3. Run a CUDA workload:
   codex exec "python3 -c 'import torch; print(torch.cuda.is_available())'"
4. Observe: Returns False (CUDA libraries not found)
5. Verify LD_LIBRARY_PATH is stripped:
   codex exec "echo $LD_LIBRARY_PATH"
6. Output: (empty or different from parent)

Performance regression can be measured with:
time codex exec "python3 -c 'import numpy as np; np.dot(np.random.rand(1000,1000), np.random.rand(1000,1000))'"

What is the expected behavior?

LD_LIBRARY_PATH should be preserved by default to support legitimate use cases (CUDA, MKL, custom libraries).

Users who need maximum security can opt-in via environment variable:
CODEX_SECURE_MODE=1 codex exec "..."

This approach:

• Restores performance and functionality for CUDA/MKL users
• Maintains strong security through existing Landlock/Seatbelt boundaries
• Allows opt-in maximum hardening when truly needed
• Has zero breaking changes for existing users

Additional information

I have a tested fix ready for PR submission:

• Makes pre_main_hardening() opt-in via CODEX_SECURE_MODE=1 environment variable
• Default mode: Preserves LD_LIBRARY_PATH (fast, compatible)
• Secure mode: Strips LD_* variables (maximum hardening)
• Only 2 files modified: cli/src/main.rs and responses-api-proxy/src/main.rs
• Security maintained through Landlock/Seatbelt (primary boundaries)

This affects users in ML/AI research, scientific computing, HPC, and enterprise environments for the most part.

---

TL;DR: The pre-main hardening regression is literally a "ghost" - it executes before main(), strips the environment, and then disappears, leaving no trace except confused users reporting slowness.

Reference documentation:

• GITHUB_ISSUE_DETAILED.md (comprehensive analysis)
• Ghost in the Codex Machine (Hypothesis & Theory)

[etraut-openai]

Thanks for the bug report and the detailed analysis. Very helpful.

We want to be secure by default, so making "secure mode" opt-in isn't a very palatable solution. There are other potential solutions we could consider though. An opt-out approach is one. Or maybe we could filter LD_LIBRARY_PATH based on known-good paths.

[bolinfest]

@johnzfitch Thanks for the write-up: this is certainly a surprising result.

Does this mean that you are using codex-aarch64-unknown-linux-gnu instead of codex-aarch64-unknown-linux-musl? Because LD_LIBRARY_PATH should have no effect on musl binaries, correct?

[bolinfest]

@viyatb-oai curious to get your thoughts on this one and #8472

[johnzfitch]

@bolinfest

Does this mean that you are using codex-aarch64-unknown-linux-gnu instead of codex-aarch64-unknown-linux-musl? Because LD_LIBRARY_PATH should have no effect on musl binaries, correct?

Yes, that is correct.

Official Linux release artifacts are *-unknown-linux-musl (source builds typically default to *-unknown-linux-gnu unless you pass a musl target), and you're right: LD_LIBRARY_PATH doesn't affect a static musl Codex binary itself.

The regression is that we strip all LD_* vars from Codex's own process environment before main(), and that stripped environment is what every child process inherits.

Those children (Python/Conda/NumPy/PyTorch, often glibc-linked) can genuinely depend on LD_LIBRARY_PATH for CUDA/MKL/non-RPATH setups, so things break-and it's 100% silent.

---

@etraut-openai

Thanks for the bug report and the detailed analysis. Very helpful.
We want to be secure by default, so making "secure mode" opt-in isn't a very palatable solution. There are other potential solutions we could consider though. An opt-out approach is one. Or maybe we could filter LD_LIBRARY_PATH based on known-good paths.

On "secure by default": agreed.

A good middle ground is to always strip the true injection knobs (LD_PRELOAD/LD_AUDIT/DYLD_INSERT_LIBRARIES, etc.) but preserve LD_LIBRARY_PATH by default (or restore it only for tool-spawned children), and reserve full LD_* stripping for an explicit secure mode.

Filtering LD_LIBRARY_PATH to "known-good" paths is possible, but "good" is highly user/enterprise-specific, so we'd need to be careful and explicit if we modify it.

[johnzfitch]

My theory is that this is the ghost behind the Ghost in the Codex Machine investigation last October.

Timeline Correlation

• 2025-09-30: PR #4521 merges (premainhardening() enabled in release)
• 2025-10-01: rust-v0.43.0 ships (first affected release)
• 2025-10-06: first "painfully slow" regression report
• 2025-10-01 → 2025-10-29: spike in env/PATH inheritance issues (multiple distinct reports)
• 2025-10-29: emergency PATH/environment fix lands
• 2025-10 → 2025-11: continued slowness / reconnecting / hang complaints
• 2025-11-05: "re-connecting" behavior fix lands

Platform Distribution (consistent with the env stripping issues)

macOS: #6012, #5679, #5339, #6243, #6218

• pre_main_hardening() strips DYLD_* (macOS analog to LD_*)
• Can break/alter dynamic linking and toolchains depending on user setup

Linux / WSL2: #4843, #3891, #6200, #5837, #6263

• pre_main_hardening() strips LD_* (including LD_LIBRARY_PATH)
• Can break CUDA/MKL/Conda/custom libs (esp. non‑RPATH installs), often manifesting as silent performance regressions

Why this was so hard to catch

• Runs pre-main() (via #[ctor::ctor]), so it’s easy to miss with standard logging/instrumentation
• Silent: no warning when variables are stripped
• Only hits certain setups (non‑RPATH, custom enterprise libs, CUDA/MKL)
• Users see LD_LIBRARY_PATH set in their shell, but inside codex exec it’s empty
• The shell shows LD_LIBRARY_PATH is set, but inside Codex it's stripped

Supporting Evidence (Issue Reports / Timeline)

• slowdown-reports.md
• env-analysis.md
• env-stripping.md

---

TL;DR: The pre-main hardening regression is literally a "ghost" - it executes before main(), strips the environment, and then disappears, leaving no trace except confused users reporting slowness.

[bolinfest]

@johnzfitch wow, this an amazing piece of investigative work: thank you!

The codex_process_hardening::pre_main_hardening() library function I added to Codex CLI in #4521 was originally designed with another executable in mind, our Responses API proxy:

https://github.com/openai/codex/tree/main/codex-rs/responses-api-proxy

As explained in:

https://github.com/openai/codex/tree/main/codex-rs/responses-api-proxy#hardening-details

Process hardening is "non-negotiable" for that binary because it was written primarily for https://github.com/openai/codex-action, which has the user's OPENAI_API_KEY in memory and we don't want a bad actor to be able to leak it.

Admittedly, Codex CLI likely also has your Codex auth token in memory (which seemed like a good reason to add hardening), though for end-users, it is likely already sitting in ~/.codex/auth.json, which is much easier for an attacker to define.

Based on all of the information you presented, it feels like we should remove codex_process_hardening::pre_main_hardening() ASAP and if we want to revisit hardening later, then we should consider your suggestions above (i.e., remove LD_LIBRARY_PATH from the Codex process, but allow processes spawned via the shell tool to inherit it).

[bolinfest]

Just put up this PR to address this: #8951.

[johnzfitch]

Verified: rust-v0.80.0-alpha.5 fixes the regression

Test Environment:

• Platform: Arch Linux 6.17.9-arch1-1 x86_64
• Binaries tested: codex-x86_64-unknown-linux-musl, codex-x86_64-unknown-linux-gnu

---

Test 1: LD_LIBRARY_PATH Preservation

Binary	LD_LIBRARY_PATH	Result
MUSL	/opt/cuda/lib64:/opt/miniconda3/lib	PRESERVED
GNU	/opt/cuda/lib64:/opt/miniconda3/lib	PRESERVED

---

Test 2: CUDA Library Loading

Library	Status
libcudart.so	LOADED
libcublas.so	LOADED
libcufft.so	LOADED

---

Test 3: MKL Performance Benchmark (10x 2000x2000 matmul)

Configuration	Run 1	Run 2	Run 3	Average	Speedup
MKL via Codex (fixed)	0.308s	0.302s	0.308s	0.306s	53x
System cblas (fallback)	16.435s	16.317s	16.198s	16.317s	1x

---

Test 4: PyTorch CPU Benchmark

Test	Result
PyTorch version	2.9.1
CUDA libs loaded	Yes
CUDA device access	Blocked by sandbox (expected)
CPU matmul (10x 2000x2000)	0.155s

---

Summary:
Environment variables are correctly preserved and inherited by child processes.
MKL-accelerated workloads run 53x faster than fallback.
CUDA libraries load successfully via LD_LIBRARY_PATH.

The fix works.

Full Verification Logs

Version

$ /tmp/alpha5/codex-x86_64-unknown-linux-musl --version
codex-cli 0.80.0-alpha.5

Test 1: LD_LIBRARY_PATH Preservation (MUSL)

$ LD_LIBRARY_PATH=/opt/cuda/lib64:/opt/miniconda3/lib codex exec "echo \  $LD_LIBRARY_PATH"

/usr/bin/zsh -lc 'echo "$LD_LIBRARY_PATH"' succeeded in 2ms:
/opt/cuda/lib64:/opt/miniconda3/lib

Test 2: LD_LIBRARY_PATH Preservation (GNU)

$ LD_LIBRARY_PATH=/opt/cuda/lib64:/opt/miniconda3/lib /tmp/alpha5-gnu/codex-x86_64-unknown-linux-gnu exec "echo \$LD_LIBRARY_PATH"

/usr/bin/zsh -lc 'echo LD_LIBRARY_PATH=$LD_LIBRARY_PATH' succeeded in 3ms:
LD_LIBRARY_PATH=/opt/cuda/lib64:/opt/miniconda3/lib

Test 3: CUDA Library Loading

$ LD_LIBRARY_PATH=/opt/cuda/lib64 codex exec "python3 -c \"import ctypes; ctypes.CDLL('libcudart.so'); ctypes.CDLL('libcublas.so')\""

/usr/bin/zsh -lc "python3 -c \"import ctypes; print('libcudart:', ctypes.CDLL('libcudart.so')); print('libcublas:', ctypes.CDLL('libcublas.so'))\"" succeeded in 39ms:
libcudart: <CDLL 'libcudart.so', handle 55ccf5b0d1d0 at 0x7ffbf7937770>
libcublas: <CDLL 'libcublas.so', handle 55ccf5b47630 at 0x7ffbf7579950>

Test 4: CUDA Library Proof (Full)

$ LD_LIBRARY_PATH=/opt/cuda/lib64:/opt/miniconda3/lib codex exec "python3 cuda_lib_proof.py"

LD_LIBRARY_PATH is SET: /opt/cuda/lib64:/opt/miniconda3/lib

Paths included:
  [EXISTS] /opt/cuda/lib64
  [EXISTS] /opt/miniconda3/lib

CUDA LIBRARY LOADING TEST
  [OK] libcudart.so: LOADED via search path
  [OK] libcublas.so: LOADED via search path
  [OK] libcufft.so: LOADED via search path

Test 5: MKL Performance (via Codex with LD_LIBRARY_PATH)

$ LD_LIBRARY_PATH=/opt/miniconda3/lib codex exec "/opt/miniconda3/bin/python matmul_benchmark.py"

NumPy: 2.3.5
BLAS: mkl-sdl (version: 2025)

Run 1: 0.308s
Run 2: 0.302s
Run 3: 0.308s
Average: 0.306s (10x 2000x2000 matmul)

Test 6: System cblas Performance (baseline - no MKL)

$ /usr/bin/python3 matmul_benchmark.py

NumPy: 2.3.5
BLAS: cblas

Run 1: 16.435s
Run 2: 16.317s
Run 3: 16.198s
Average: 16.317s (10x 2000x2000 matmul)

Test 7: PyTorch CPU Benchmark

$ LD_LIBRARY_PATH=/opt/cuda/lib64:/opt/miniconda3/lib codex exec "python3 pytorch_benchmark.py"

PyTorch: 2.9.1
CUDA available: False (sandbox blocks device access - expected)
CPU (10x 2000x2000 matmul): 0.155s

Summary

- LD_LIBRARY_PATH (MUSL): ✓ PRESERVED
- LD_LIBRARY_PATH (GNU): ✓ PRESERVED
- CUDA libraries: ✓ LOADED (libcudart, libcublas, libcufft)
- MKL performance: ✓ 53x faster than fallback

[bolinfest]

@johnzfitch awesome: glad to hear it! I need to audit the other changes that have landed on main since the last release. Assuming there isn't anything that needs more bake time, we should be able to promote this to 0.80.0-alpha.5 as 0.80.0 tomorrow!

[johnzfitch]

@bolinfest Thanks for prioritizing this - really appreciate the quick turnaround!

The 53x performance improvement for MKL workloads is significant. CUDA/ML users will definitely appreciate this in 0.80.0.