Skip to content

Fix #423: link MKL statically into _scs_mkl; OpenBLAS default backends; clean-container wheel gate - #232

Merged
bodono merged 19 commits into
masterfrom
fix/wheels-openblas-423
Sep 5, 2026
Merged

Fix #423: link MKL statically into _scs_mkl; OpenBLAS default backends; clean-container wheel gate#232
bodono merged 19 commits into
masterfrom
fix/wheels-openblas-423

Conversation

@bodono

@bodono bodono commented Aug 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

The 3.3.0 manylinux x86-64 wheels vendored dynamic MKL and shipped it incomplete: MKL loads its CPU dispatch kernels via dlopen, which auditwheel cannot see, so every solve aborted on machines without MKL on the loader path (#423). Every extension linked MKL, so the default backend died too.

This PR:

  • Default backends link vendored OpenBLAS (_scs_direct, _scs_indirect, _scs_dense), so no MKL problem can ever take down the default solver again.
  • _scs_mkl links MKL statically from Intel's mkl-static PyPI archives (LP64, sequential, symbols hidden). Static MKL dispatches CPU variants internally, so there are no kernels to lose; the interface is fixed at link time, so there is nothing to negotiate at runtime (the core's MKL_Set_Interface_Layer reference is weak, MKL: accept GNU-convention interface layers (compare width bit only) cvxgrp/scs#422); and the sequential threading layer ships no OpenMP runtime that could abort a process already holding one. The wheel is self-contained: about 42 MB, versus 52 MB for the broken 3.3.0 wheel. AUTO selects MKL Pardiso out of the box in the x86-64 manylinux wheels (musllinux has no MKL).
  • Release gates: wheel_smoke_clean_env installs each wheel in a pristine container with nothing on the loader path (asserted) and solves with every backend on a problem with a PSD cone, on Python 3.9, 3.14 and 3.14t; wheel_audit checks every Linux wheel for the shipped inventory (the standard extensions and a vendored OpenBLAS everywhere, _scs_mkl in exactly the x86-64 manylinux wheels, and at least one such wheel), a dynamic MKL dependency, an Intel shared library, or an absolute rpath entry. upload_pypi depends on both and on build_mkl, since the sdist carries the link_mkl source build.
  • Nothing in the cibuildwheel environment points the loader at build-machine libraries; the wheel tests run exactly as clean as a user's machine.
  • Intel's license notice ships in the wheel as LICENSE-INTEL-MKL.txt.
  • Windows wheels pin conda-forge OpenBLAS to 0.3.33: the win-64 0.3.34 build crashes inside its DGEMM kernels on AMD Zen 4/5 CPUs that expose AVX-512 (every Ryzen 7000/9000 desktop), which is what the intermittent access violation in the Windows wheel tests was; reported as win-64 libopenblas 0.3.34 (pthreads): access violation in dgemm_kernel on AMD Zen 4/5 with AVX-512 exposed (0.3.33 fine) conda-forge/openblas-feedstock#196 with a standalone reproducer. Unpin once a fixed build ships.

Validated before landing on a scratch branch: the full core test suite (58 tests wheel-default, 66 with spectral cones, PSD cones included) against the static library in a clean environment.

🤖 Generated with Claude Code

bodono added 2 commits August 31, 2026 15:19
@bodono
Revert x86_64 Linux and Windows wheels to OpenBLAS, add clean-contain…
eef2dad 
…er wheel test

The 3.3.0 manylinux x86_64 wheels shipped MKL without its CPU dispatch
kernels (libmkl_def/avx2/avx512.so.3). libmkl_core dlopens these at
runtime, so they are invisible to auditwheel, which only vendors
DT_NEEDED libraries. On any machine without MKL already on the loader
path the first solve aborted the interpreter with 'Intel oneMKL FATAL
ERROR' and exit(2). All backends were affected because every extension,
including _scs_direct, linked MKL BLAS/LAPACK. Fixes #423 (cvxgrp/scs).

The bug survived CI because cibuildwheel's test phase ran inside the
build container, whose LD_LIBRARY_PATH pointed at the container's
system oneAPI install: the dlopen always succeeded there regardless of
what the wheel vendored.

This reverts the wheel linkage to OpenBLAS, the configuration proven
through 3.2.11 (macOS/aarch64/musllinux wheels were always OpenBLAS or
Accelerate and are unaffected). Windows wheels, MKL-linked since the
same change and repaired by delvewheel, which is equally blind to
dlopen dependencies, revert likewise. MKL remains available in source
builds (-Dlink_mkl=true, e.g. conda).

Adds a wheel_smoke_clean_env CI job that installs the built manylinux
x86_64 wheel in a pristine python:3.12-slim container (no BLAS, no
LD_LIBRARY_PATH) and solves an LP+PSD problem on every shipped backend,
so a vendoring gap of this class can never pass CI again.
@bodono
Gate PyPI upload on the clean-container wheel smoke test
3b00317
bodono added a commit that referenced this pull request Aug 31, 2026
@bodono
README: wheels ship OpenBLAS; AUTO picks up MKL from the environment
95be616 
Aligns the AUTO-preference note with the wheel linkage revert (#232):
the pre-built wheels no longer ship MKL, so AUTO resolves to MKL only
where the environment provides it (e.g. conda source builds).
@bodono bodono mentioned this pull request Aug 31, 2026
Open
@bodono
Restore MKL in Linux x86-64 wheels via the scs[mkl] extra
b23bb4d 
Rather than vendoring MKL (whose dlopen'd CPU dispatch kernels are
invisible to auditwheel -- the root cause of #423) or dropping it, the
wheels now build _scs_mkl linked against MKL without vendoring it, and
Intel's official 'mkl' + 'intel-openmp' PyPI wheels supply the complete,
internally-consistent runtime via the scs[mkl] extra.

- meson: new mkl_backend option builds only _scs_mkl against MKL while
  every other extension links the platform BLAS (OpenBLAS in wheels).
  link_mkl keeps its all-extensions-on-MKL meaning for conda/source.
- scs/py: when importing _scs_mkl fails, preload MKL from the installed
  mkl/intel-openmp distributions (RTLD_LOCAL, dependency order) and
  retry; without them the ImportError propagates and AUTO falls back to
  QDLDL. RTLD_LOCAL avoids interposing MKL onto e.g. NumPy's OpenBLAS.
- cibuildwheel: oneAPI is installed for building only; no LD_LIBRARY_PATH
  in the container environment (the #423 blind spot), the oneAPI paths are
  scoped to the auditwheel repair command, which excludes libmkl_*/libiomp5
  from grafting.
- CI: the clean-container smoke test becomes a default/mkl matrix; the
  mkl variant installs scs[mkl] and requires AUTO to resolve to MKL,
  also enforcing that the wheel's oneMKL generation matches the pinned
  mkl PyPI range (mkl>=2026,<2027, the .so.3 ABI). upload_pypi remains
  gated on both variants.

Windows wheels stay OpenBLAS-only for now: Windows MKL was sequential
anyway (broken iomp pkg-config), and the conda/pip generation-matching
story there needs separate validation.
@bodono bodono changed the title Revert wheels to OpenBLAS linkage, add clean-container wheel smoke test (fixes cvxgrp/scs#423) Fix #423: unvendor MKL — OpenBLAS default backends, MKL via scs[mkl] extra, clean-container wheel gate Aug 31, 2026
bodono added 4 commits August 31, 2026 15:45
@bodono
Restore compiler pkgconfig dir so wheels link threaded (iomp) MKL
d16f2da 
Without /opt/intel/oneapi/compiler/latest/lib/pkgconfig on
PKG_CONFIG_PATH the mkl-dynamic-lp64-iomp dependency fails to resolve
and meson silently falls back to sequential MKL;
test_mkl_module_links_intel_openmp caught the downgrade. Note this env
var carries no LD_LIBRARY_PATH: the mkl tests still run in-container
only via the extension's baked RUNPATH to the oneAPI install, and the
clean-container smoke job remains the authority on what the wheel
actually ships.
@bodono
Smoke test: select the cp312 wheel matching the container python
fb28639
@bodono
Smoke test: identify backends by __file__, add scs[mkl] import diagno…
bdf282b 
…stics

The extensions share the PyModuleDef name "_scs" (single-phase init),
so __name__ cannot distinguish them. Also surface the pip resolution
and the exact ImportError when the mkl variant fails to load.
@bodono
Resolve scs[mkl] via a prefix-relative RUNPATH instead of ctypes prel…
fbbcf23 
…oads

MKL's component libraries reference each other and cannot be eagerly
bound one at a time; ctypes.CDLL always forces RTLD_NOW, so the preload
loop silently loaded nothing and _scs_mkl still failed to import
(diagnosed in CI: all libraries present in <prefix>/lib, every CDLL
swallowed an OSError).

Primary fix: after auditwheel repair, patchelf an
'$ORIGIN/../../../../lib' RUNPATH onto _scs_mkl (site-packages/scs is
four levels below the prefix in every standard layout), so the dynamic
loader resolves the whole MKL group in one dlopen with correct mutual
binding and no symbol-scope widening. auditwheel preserves existing
RUNPATHs (observed in CI), so patching after repair is stable.

The Python shim remains as a fallback for non-standard layouts, now
calling dlopen(3) directly with RTLD_LAZY | RTLD_LOCAL.
@bodono bodono mentioned this pull request Aug 31, 2026
Merged
@bodono
README: recommend scs[mkl] prominently in the install section
41209d6 
Per review direction: MKL is opt-in but should be loud — for most users
on x86-64 Linux it is strictly better than the built-in solver, so the
install section now leads with a GitHub IMPORTANT callout for
pip install "scs[mkl]", and the backends section defers to it.
bodono added a commit that referenced this pull request Aug 31, 2026
@bodono
Fix reviewed #232 issues: normalize RPATH, audit all wheels, widen th…
897cfdb 
…e clean gate

- add_mkl_rpath.sh uses patchelf --set-rpath: the meson build bakes the
  container's /opt/intel paths into _scs_mkl and auditwheel preserves
  them; shipping those would let a build-machine-layout MKL install
  shadow the scs[mkl] runtime and let container tests silently resolve
  the build MKL. The script now also asserts the exact final RUNPATH
  and the exact MKL/iomp NEEDED set.
- New wheel_audit job: every .so in every Linux wheel (all pythons,
  x86_64/aarch64/musllinux) must carry only $ORIGIN-relative RPATH
  entries, and every _scs_mkl must have the exact expected MKL NEEDED
  set. Gates upload_pypi.
- Clean-container smoke matrix now covers oldest (3.9), newest stable
  (3.14) and free-threaded (3.14t via uv) wheels, both with and without
  scs[mkl]; remaining ABIs are covered by the audit.
- README: correct scs[mkl] footprint (~300 MB download / ~1 GB on
  disk), document exactly which install modes the extra supports
  (manylinux x86-64 glibc 2.28+, standard prefixes; not musllinux,
  pip --target, or sdist builds), and stop implying Windows wheels
  contain an MKL backend.
@bodono
Fix reviewed #232 issues: normalize RPATH, audit all wheels, widen th…
4c6f456 
…e clean gate

- add_mkl_rpath.sh uses patchelf --set-rpath: the meson build bakes the
  container's /opt/intel paths into _scs_mkl and auditwheel preserves
  them; shipping those would let a build-machine-layout MKL install
  shadow the scs[mkl] runtime and let container tests silently resolve
  the build MKL. The script now also asserts the exact final RUNPATH
  and the exact MKL/iomp NEEDED set.
- New wheel_audit job: every .so in every Linux wheel (all pythons,
  x86_64/aarch64/musllinux) must carry only $ORIGIN-relative RPATH
  entries, and every _scs_mkl must have the exact expected MKL NEEDED
  set. Gates upload_pypi.
- Clean-container smoke matrix now covers oldest (3.9), newest stable
  (3.14) and free-threaded (3.14t via uv) wheels, both with and without
  scs[mkl]; remaining ABIs are covered by the audit.
- README: correct scs[mkl] footprint (~300 MB download / ~1 GB on
  disk), document exactly which install modes the extra supports
  (manylinux x86-64 glibc 2.28+, standard prefixes; not musllinux,
  pip --target, or sdist builds), and stop implying Windows wheels
  contain an MKL backend.
@bodono
bodono force-pushed the fix/wheels-openblas-423 branch from 897cfdb to 4c6f456 Compare August 31, 2026 17:23
bodono added 4 commits August 31, 2026 20:07
@bodono
Make the wheel audit fail-closed
8cc38e0 
Per review, the audit was fail-open on three axes: a wheel set with no
_scs_mkl at all passed vacuously, readelf failures were silently
ignored, and nothing pinned per-platform content. Now:
- exact extension inventory per platform (x86_64: direct/indirect/mkl;
  aarch64+musllinux: direct/indirect, never mkl),
- the _scs_mkl count must equal the number of x86_64 manylinux wheels,
- readelf exit status is checked; a failure is a finding,
- every wheel must vendor OpenBLAS and must never vendor MKL/iomp,
- _scs_mkl's RUNPATH must be exactly ['$ORIGIN/../../../../lib'].
@bodono
Audit: include _scs_dense, forbid vacuity, split RPATH/RUNPATH, scan …
4d9999b 
…whole wheel

- _scs_dense ships in every Linux wheel and was missing from the
  expected inventories, failing all 27 correct wheels.
- Every platform group must be present in the downloaded artifacts and
  the expected _scs_mkl count must be positive, so an absent artifact
  group can never make the MKL checks vacuous.
- Intel libraries are now rejected anywhere in the wheel, not only
  under .libs paths.
- DT_RPATH and DT_RUNPATH are tracked separately: legacy DT_RPATH is
  forbidden outright, and _scs_mkl must carry exactly one DT_RUNPATH
  equal to $ORIGIN/../../../../lib.
@bodono
Audit: scope the DT_RPATH prohibition to _scs_mkl
af9e091 
auditwheel deliberately writes DT_RPATH on every object it grafts
(RPATH outranks LD_LIBRARY_PATH, isolating vendored libraries), so the
blanket prohibition failed all 27 correct wheels. The general rule is
$ORIGIN-relative entries on either tag type; _scs_mkl -- the object
this repo patches itself -- keeps the strict shape of exactly one
DT_RUNPATH and no DT_RPATH.
@bodono
Audit: require the exact python/ABI tag set per platform
95b9a81 
Deriving expected counts from downloaded wheels let one surviving
wheel per platform stand in for the full set. Each platform group must
now carry exactly {cp39..cp315, cp314t, cp315t}; the ABI tag (field 4)
distinguishes free-threaded builds.
bodono added a commit that referenced this pull request Aug 31, 2026
@bodono
README: wheels ship OpenBLAS; AUTO picks up MKL from the environment
73ed645 
Aligns the AUTO-preference note with the wheel linkage revert (#232):
the pre-built wheels no longer ship MKL, so AUTO resolves to MKL only
where the environment provides it (e.g. conda source builds).
@bodono @claude
Link MKL statically into _scs_mkl: self-contained wheels, no runtime …
bf89d69 
…extra

The 3.3.0 wheels vendored dynamic MKL and shipped it incomplete: its CPU
dispatch kernels are loaded via dlopen, invisible to auditwheel, so every
solve aborted on machines without MKL on the loader path (cvxgrp/scs#423).
Unvendoring (the previous shape of this PR) fixed that at the cost of a
runtime extra, a RUNPATH patch, a dlopen shim and a 276 MB download.

Linking Intel's mkl-static archives into _scs_mkl instead makes the wheel
self-contained: static MKL dispatches CPU variants internally, so there is
nothing to lose; the LP64 interface is fixed at link time, so there is
nothing to negotiate (the core's MKL_Set_Interface_Layer reference is now
weak); every MKL symbol is hidden, so a process-wide MKL is never
interposed; and the sequential threading layer ships no OpenMP runtime
that could abort a process already holding one. Measured on the scratch
branch: 108 MB on disk, 30 MB compressed, the full core test suite
passing in a clean environment (58/58 wheel-default, 66/66 with spectral
cones), versus 52 MB for the broken 3.3.0 wheel.

- meson: mkl_static_prefix replaces mkl_backend; _scs_mkl links the
  intel_lp64/sequential/core archives as a group with --exclude-libs,ALL
  and installs Intel's license notice alongside (ISSL requires it).
- cibuildwheel: install_mkl_static.sh lays the archives out under
  /opt/mkl-static from Intel's PyPI wheels; default auditwheel repair;
  nothing in the environment points the loader at build-machine paths.
- Deleted: the scs[mkl] extra, add_mkl_rpath.sh, the dlopen shim in
  scs/py/__init__.py, and the ldd linkage test.
- wheel_audit keeps the invariants that matter for every Linux wheel: no
  dynamic MKL dependency in any extension, no Intel shared library in the
  wheel, OpenBLAS vendored, no absolute rpath entry.
- wheel_smoke_clean_env, the release gate: one variant, pristine
  container, LD_LIBRARY_PATH asserted unset, every backend solved
  (qdldl, cpu_indirect, dense, mkl, auto) on a problem with a PSD cone;
  on x86-64 the static MKL backend must import and be AUTO's choice.
- scs_source pinned to the core commit carrying the weak reference.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bodono added a commit to cvxgrp/scs that referenced this pull request Sep 3, 2026
@bodono @claude
docs: MKL Pardiso is built into the x86-64 Linux wheels
f26f888 
The wheels now link MKL statically into _scs_mkl (bodono/scs-python#232),
so the scs[mkl] extra no longer exists and there is nothing to recommend
installing: AUTO selects MKL out of the box. Explain why MKL is linked
statically rather than bundled as shared libraries (#423), why the
backend is single-threaded, and where Intel's license notice lives.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@bodono @claude
Smoke test: the dense backend is named cpu_dense
a50807e 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@bodono bodono changed the title Fix #423: unvendor MKL — OpenBLAS default backends, MKL via scs[mkl] extra, clean-container wheel gate Fix #423: link MKL statically into _scs_mkl; OpenBLAS default backends; clean-container wheel gate Sep 3, 2026
@bodono @claude
link_mkl: keep mkl_rt a hard dependency under --as-needed
ca985bf 
The core's MKL_Set_Interface_Layer reference is weak so that static MKL
links without mkl_rt. In the dynamic link_mkl build that made mkl_rt
referenced only weakly, --as-needed dropped it from NEEDED, the symbol
resolved to NULL and the interface-layer guard silently no longer ran
(the ILP64-guard CI lane caught it). Link with --no-as-needed on Linux,
as the core's own CMake and Makefile builds already do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bodono added a commit to cvxgrp/scs that referenced this pull request Sep 3, 2026
@bodono @claude
docs: MKL Pardiso is built into the x86-64 Linux wheels
cdbb01c 
The wheels now link MKL statically into _scs_mkl (bodono/scs-python#232),
so the scs[mkl] extra no longer exists and there is nothing to recommend
installing: AUTO selects MKL out of the box. Explain why MKL is linked
statically rather than bundled as shared libraries (#423), why the
backend is single-threaded, and where Intel's license notice lives.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@bodono
bodono force-pushed the fix/wheels-openblas-423 branch from 788aae7 to 1442623 Compare September 3, 2026 21:07
@bodono @claude
Tidy the static-MKL build: keep rt in link_mkl, say #423 once
8e12289 
- link_mkl builds linked _base_deps + mkl_rt and so dropped the rt that
  every other extension gets through _deps; use _deps.
- The static branch's dl/m/rt dependencies are one required line; the
  install script loses knobs and fallbacks the container never takes.
- The #423 rationale lives in pyproject.toml; meson, the workflow and the
  README point there instead of retelling it.
- wheel_audit keeps the three invariants that matter (no Intel shared
  library, no dynamic MKL dependency, no absolute rpath entry).
- AUTO is resolved once per process.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@bodono
bodono force-pushed the fix/wheels-openblas-423 branch from 1442623 to 8e12289 Compare September 3, 2026 21:17
@bodono @claude
wheel_audit: require the shipped inventory; publish waits for the sou…
720a0f8 
…rce-MKL lanes

The audit only forbade things (dynamic MKL, Intel libraries, absolute
rpaths); it now also requires what a wheel must contain: the standard
extensions and a vendored OpenBLAS in every Linux wheel, _scs_mkl in
exactly the x86-64 manylinux wheels, and at least one such wheel, so a
selector or build change cannot drop the MKL backend unnoticed.
upload_pypi also waits for build_mkl, since the sdist carries the
link_mkl source build. README: MKL ships in the x86-64 manylinux wheels
specifically (musllinux has no MKL).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@bodono bodono mentioned this pull request Sep 4, 2026
Merged
@bodono @claude
Windows wheels: pin conda-forge OpenBLAS to 0.3.33
6355503 
conda-forge's win-64 libopenblas 0.3.34 (pthreads_h877e47f_1) crashes with
an access violation inside dgemm_kernel_HASWELL / dgemm_kernel_ZEN on AMD
Zen 4 and Zen 5 CPUs whose hypervisor or firmware exposes AVX-512, on plain
dsyrk/dpotrf calls; 0.3.33 and earlier builds are fine on the same
machines (conda-forge/openblas-feedstock#196). The wheel CI hit it on the
runners that happen to have such CPUs, and every Ryzen 7000/9000 desktop
would have. Pin until a fixed build ships.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@bodono
bodono merged commit 1724d39 into master Sep 5, 2026
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bodono