From eef2dad62457419241a9dc0988e481e8dc54b785 Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 15:19:14 +0100 Subject: [PATCH 01/19] Revert x86_64 Linux and Windows wheels to OpenBLAS, add clean-container wheel test The 3.3.0 manylinux x86_64 wheels shipped MKL without its CPU dispatch kernels (libmkl_def/avx2/avx512.so.3). libmkl_core dlopens these at runtime, so they are invisible to auditwheel, which only vendors DT_NEEDED libraries. On any machine without MKL already on the loader path the first solve aborted the interpreter with 'Intel oneMKL FATAL ERROR' and exit(2). All backends were affected because every extension, including _scs_direct, linked MKL BLAS/LAPACK. Fixes #423 (cvxgrp/scs). The bug survived CI because cibuildwheel's test phase ran inside the build container, whose LD_LIBRARY_PATH pointed at the container's system oneAPI install: the dlopen always succeeded there regardless of what the wheel vendored. This reverts the wheel linkage to OpenBLAS, the configuration proven through 3.2.11 (macOS/aarch64/musllinux wheels were always OpenBLAS or Accelerate and are unaffected). Windows wheels, MKL-linked since the same change and repaired by delvewheel, which is equally blind to dlopen dependencies, revert likewise. MKL remains available in source builds (-Dlink_mkl=true, e.g. conda). Adds a wheel_smoke_clean_env CI job that installs the built manylinux x86_64 wheel in a pristine python:3.12-slim container (no BLAS, no LD_LIBRARY_PATH) and solves an LP+PSD problem on every shipped backend, so a vendoring gap of this class can never pass CI again. --- .github/workflows/build.yml | 53 ++++++++++++++++++++++++++++++++++--- README.md | 7 ++--- pyproject.toml | 17 +++++++----- 3 files changed, 64 insertions(+), 13 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index e10a1d97..e90472b6 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -249,11 +249,11 @@ jobs: uses: conda-incubator/setup-miniconda@v4 with: miniconda-version: "latest" - channels: https://software.repos.intel.com/python/conda/, conda-forge + channels: conda-forge, anaconda - - name: Install MKL from conda on Windows + - name: Install openblas from conda on Windows if: runner.os == 'Windows' - run: conda install -y -c https://software.repos.intel.com/python/conda/ -c conda-forge mkl mkl-devel intel-openmp pkgconfig + run: conda install -y openblas pkgconfig - name: Build wheels uses: pypa/cibuildwheel@v4.2.0 @@ -279,6 +279,53 @@ jobs: name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }} path: ./wheelhouse/*.whl + wheel_smoke_clean_env: + # Regression gate for #423: install the built manylinux x86_64 wheel in a + # pristine container (no BLAS, no LD_LIBRARY_PATH, no site packages) and + # run a real solve on every backend the wheel ships. The 3.3.0 wheels + # passed cibuildwheel's in-container tests because the build container's + # environment satisfied MKL's dlopen'd dispatch kernels; only a clean + # environment exercises what the wheel actually vendors. + name: Clean-container wheel smoke test + needs: build_wheels + runs-on: ubuntu-latest + steps: + - uses: actions/download-artifact@v7 + with: + pattern: cibw-wheels-ubuntu-latest-* + path: wheelhouse + merge-multiple: true + + - name: Solve in pristine python:3.12-slim + run: | + cat > smoke.py <<'EOF' + import importlib.util, numpy as np, scipy.sparse as sp, scs + print("scs", scs.__version__) + # LP block plus a small PSD cone: the PSD projection goes through + # vendored BLAS/LAPACK, which is exactly what #423 failed to ship. + data = { + "A": sp.csc_matrix(np.vstack([np.array([[1.0, 1.0], [1.0, -1.0], [0.0, 1.0]]), + np.array([[-1.0, 0.0], [0.0, 0.0], [0.0, -1.0]])])), + "b": np.hstack([np.array([1.0, 0.5, 0.7]), np.zeros(3)]), + "c": np.array([1.0, 1.0]), + } + cone = {"z": 1, "l": 2, "s": [2]} + solvers = ["auto"] + [name for name, mod in + [("qdldl", "_scs_direct"), + ("cpu_indirect", "_scs_indirect"), + ("mkl", "_scs_mkl")] + if importlib.util.find_spec("scs." + mod)] + assert "qdldl" in solvers + for name in solvers: + sol = scs.SCS(data, cone, verbose=True, linear_solver=name).solve() + assert sol["info"]["status"] == "solved", (name, sol["info"]["status"]) + print(name, "ok:", sol["info"].get("lin_sys_solver", "n/a")) + EOF + docker run --rm -v "$PWD:/work" -w /tmp python:3.12-slim bash -ec ' + pip install --quiet numpy scipy + pip install --no-index --find-links /work/wheelhouse scs + python /work/smoke.py' + build_sdist: name: Build source distribution runs-on: ubuntu-latest diff --git a/README.md b/README.md index f57d2976..a74d8513 100644 --- a/README.md +++ b/README.md @@ -40,9 +40,10 @@ solver = scs.SCS(data, cone, linear_solver=scs.LinearSolver.QDLDL) Available values: `AUTO`, `QDLDL`, `CPU_INDIRECT`, `MKL`, `ACCELERATE`, `CPU_DENSE`, `GPU_INDIRECT`, `CUDSS`. -The pre-built wheels (`pip install scs`) include MKL on x86_64 Linux and -Windows, and Apple Accelerate on macOS. When installing from source, additional -backends can be enabled with build-time flags: +The pre-built wheels (`pip install scs`) link OpenBLAS on Linux and Windows, +and Apple Accelerate on macOS; the MKL backend is available in source builds +(e.g. in conda environments providing MKL). When installing from source, +additional backends can be enabled with build-time flags: ```bash # MKL Pardiso direct solver diff --git a/pyproject.toml b/pyproject.toml index 41537bcf..d9557c48 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -63,21 +63,24 @@ archs = [ before-build = "pip install delvewheel" # This will probably become default in newer cibuildwheels versions repair-wheel-command = "delvewheel repair -w {dest_dir} {wheel}" -config-settings = {setup-args = "-Dlink_mkl=true"} # Openblas installation for 3 different linux images +# x86_64 Linux and Windows wheels link OpenBLAS. They were switched to +# MKL in #187, which shipped 3.3.0 wheels whose vendored MKL was missing +# its dlopen'd dispatch kernels (libmkl_def/avx2/avx512 -- invisible to +# auditwheel/delvewheel, which only follow DT_NEEDED): on any machine +# without MKL already on the loader path, the first solve aborted the +# interpreter via MKL's own exit(2). See #423. The build-container test +# could not catch it because LD_LIBRARY_PATH pointed at the container's +# system oneAPI install. MKL wheels may return once kernel vendoring is +# solved and gated by the clean-container wheel test. [[tool.cibuildwheel.overrides]] select = "*-manylinux_x86_64" inherit.before-all = "append" before-all = [ - "dnf install -y dnf-plugins-core", - "dnf config-manager --add-repo https://yum.repos.intel.com/oneapi", - "rpm --import https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB", - "dnf install -y intel-oneapi-mkl-devel", + "dnf install -y openblas-devel", ] -environment = {PKG_CONFIG_PATH = "/opt/intel/oneapi/mkl/latest/lib/pkgconfig:/opt/intel/oneapi/compiler/latest/lib/pkgconfig:$PKG_CONFIG_PATH", LD_LIBRARY_PATH = "/opt/intel/oneapi/mkl/latest/lib/intel64:/opt/intel/oneapi/mkl/latest/lib:/opt/intel/oneapi/compiler/latest/lib:$LD_LIBRARY_PATH"} -config-settings = {setup-args = "-Dlink_mkl=true"} [[tool.cibuildwheel.overrides]] select = "*-manylinux_aarch64" From 3b0031722c22d35812b72566432bdcef90eee1c0 Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 15:20:18 +0100 Subject: [PATCH 02/19] Gate PyPI upload on the clean-container wheel smoke test --- .github/workflows/build.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index e90472b6..dc29ef1d 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -343,7 +343,9 @@ jobs: path: dist/*.tar.gz upload_pypi: - needs: [build_wheels, build_sdist] + # wheel_smoke_clean_env gates publishing: wheels that only work in the + # build container must never reach PyPI again (see #423). + needs: [build_wheels, build_sdist, wheel_smoke_clean_env] runs-on: ubuntu-latest environment: pypi permissions: From b23bb4da82155928865cfdeb5b349dabbaa6b897 Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 15:40:04 +0100 Subject: [PATCH 03/19] Restore MKL in Linux x86-64 wheels via the scs[mkl] extra Rather than vendoring MKL (whose dlopen'd CPU dispatch kernels are invisible to auditwheel -- the root cause of #423) or dropping it, the wheels now build _scs_mkl linked against MKL without vendoring it, and Intel's official 'mkl' + 'intel-openmp' PyPI wheels supply the complete, internally-consistent runtime via the scs[mkl] extra. - meson: new mkl_backend option builds only _scs_mkl against MKL while every other extension links the platform BLAS (OpenBLAS in wheels). link_mkl keeps its all-extensions-on-MKL meaning for conda/source. - scs/py: when importing _scs_mkl fails, preload MKL from the installed mkl/intel-openmp distributions (RTLD_LOCAL, dependency order) and retry; without them the ImportError propagates and AUTO falls back to QDLDL. RTLD_LOCAL avoids interposing MKL onto e.g. NumPy's OpenBLAS. - cibuildwheel: oneAPI is installed for building only; no LD_LIBRARY_PATH in the container environment (the #423 blind spot), the oneAPI paths are scoped to the auditwheel repair command, which excludes libmkl_*/libiomp5 from grafting. - CI: the clean-container smoke test becomes a default/mkl matrix; the mkl variant installs scs[mkl] and requires AUTO to resolve to MKL, also enforcing that the wheel's oneMKL generation matches the pinned mkl PyPI range (mkl>=2026,<2027, the .so.3 ABI). upload_pypi remains gated on both variants. Windows wheels stay OpenBLAS-only for now: Windows MKL was sequential anyway (broken iomp pkg-config), and the conda/pip generation-matching story there needs separate validation. --- .github/workflows/build.yml | 64 ++++++++++++++------ README.md | 18 +++++- meson.build | 56 ++++++++++++------ meson.options | 2 + pyproject.toml | 44 ++++++++++---- scs/py/__init__.py | 77 ++++++++++++++++++++++++- test/test_solve_random_cone_prob_mkl.py | 12 ++-- 7 files changed, 219 insertions(+), 54 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index dc29ef1d..adffcc07 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -282,13 +282,25 @@ jobs: wheel_smoke_clean_env: # Regression gate for #423: install the built manylinux x86_64 wheel in a # pristine container (no BLAS, no LD_LIBRARY_PATH, no site packages) and - # run a real solve on every backend the wheel ships. The 3.3.0 wheels - # passed cibuildwheel's in-container tests because the build container's + # run a real solve on every backend. The 3.3.0 wheels passed + # cibuildwheel's in-container tests because the build container's # environment satisfied MKL's dlopen'd dispatch kernels; only a clean - # environment exercises what the wheel actually vendors. - name: Clean-container wheel smoke test + # environment exercises what the wheel actually ships. + # + # Two variants: + # default -- plain `pip install scs`: AUTO must resolve to QDLDL and the + # unvendored _scs_mkl must fail with a clean ImportError, never crash. + # mkl -- `pip install scs[mkl]`: MKL comes from Intel's official wheels; + # AUTO must resolve to MKL and the mkl backend must solve. This also + # enforces that the oneMKL generation the wheel links matches the + # pinned `mkl` PyPI dependency range. + name: Clean-container wheel smoke test (${{ matrix.variant }}) needs: build_wheels runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + variant: [default, mkl] steps: - uses: actions/download-artifact@v7 with: @@ -296,13 +308,15 @@ jobs: path: wheelhouse merge-multiple: true - - name: Solve in pristine python:3.12-slim + - name: Solve in pristine python:3.12-slim (${{ matrix.variant }}) run: | cat > smoke.py <<'EOF' - import importlib.util, numpy as np, scipy.sparse as sp, scs - print("scs", scs.__version__) + import sys + import numpy as np, scipy.sparse as sp, scs + mode = sys.argv[1] + print("scs", scs.__version__, "mode", mode) # LP block plus a small PSD cone: the PSD projection goes through - # vendored BLAS/LAPACK, which is exactly what #423 failed to ship. + # the wheel's BLAS/LAPACK, the exact path #423 failed to ship. data = { "A": sp.csc_matrix(np.vstack([np.array([[1.0, 1.0], [1.0, -1.0], [0.0, 1.0]]), np.array([[-1.0, 0.0], [0.0, 0.0], [0.0, -1.0]])])), @@ -310,21 +324,37 @@ jobs: "c": np.array([1.0, 1.0]), } cone = {"z": 1, "l": 2, "s": [2]} - solvers = ["auto"] + [name for name, mod in - [("qdldl", "_scs_direct"), - ("cpu_indirect", "_scs_indirect"), - ("mkl", "_scs_mkl")] - if importlib.util.find_spec("scs." + mod)] - assert "qdldl" in solvers + sol = scs.SCS(data, cone, verbose=True).solve() + assert sol["info"]["status"] == "solved", sol["info"]["status"] + auto_mod = scs._resolve_auto().__name__ + print("auto resolves to", auto_mod) + solvers = ["qdldl", "cpu_indirect"] + if mode == "mkl": + assert auto_mod.endswith("_scs_mkl"), auto_mod + solvers.append("mkl") + else: + assert auto_mod.endswith("_scs_direct"), auto_mod + # The wheel ships _scs_mkl but its MKL comes from scs[mkl]; + # without the extra it must fail cleanly, never crash. + try: + scs.SCS(data, cone, linear_solver="mkl") + raise SystemExit("unvendored _scs_mkl imported without scs[mkl]") + except ImportError: + print("mkl backend unavailable without scs[mkl], as expected") for name in solvers: sol = scs.SCS(data, cone, verbose=True, linear_solver=name).solve() assert sol["info"]["status"] == "solved", (name, sol["info"]["status"]) print(name, "ok:", sol["info"].get("lin_sys_solver", "n/a")) EOF - docker run --rm -v "$PWD:/work" -w /tmp python:3.12-slim bash -ec ' + docker run --rm -v "$PWD:/work" -w /tmp -e VARIANT=${{ matrix.variant }} python:3.12-slim bash -ec ' pip install --quiet numpy scipy - pip install --no-index --find-links /work/wheelhouse scs - python /work/smoke.py' + WHL=$(ls /work/wheelhouse/scs-*manylinux*x86_64.whl) + if [ "$VARIANT" = mkl ]; then + pip install --quiet "${WHL}[mkl]" + else + pip install --quiet "$WHL" + fi + python /work/smoke.py "$VARIANT"' build_sdist: name: Build source distribution diff --git a/README.md b/README.md index a74d8513..efa3a8ff 100644 --- a/README.md +++ b/README.md @@ -41,9 +41,21 @@ Available values: `AUTO`, `QDLDL`, `CPU_INDIRECT`, `MKL`, `ACCELERATE`, `CPU_DENSE`, `GPU_INDIRECT`, `CUDSS`. The pre-built wheels (`pip install scs`) link OpenBLAS on Linux and Windows, -and Apple Accelerate on macOS; the MKL backend is available in source builds -(e.g. in conda environments providing MKL). When installing from source, -additional backends can be enabled with build-time flags: +and Apple Accelerate on macOS. On x86-64 Linux the MKL Pardiso backend is +available as an extra: + +```bash +pip install "scs[mkl]" +``` + +This pulls Intel's official `mkl` wheels, which provide the complete MKL +runtime (SCS wheels deliberately do not vendor MKL: its CPU dispatch +kernels are loaded via `dlopen`, invisible to wheel-repair tools, and an +incompletely vendored MKL aborts the process at solve time). With the extra +installed `AUTO` selects MKL; without it, `AUTO` falls back to QDLDL. The +MKL backend is also available in source builds (e.g. conda environments +providing MKL). When installing from source, additional backends can be +enabled with build-time flags: ```bash # MKL Pardiso direct solver diff --git a/meson.build b/meson.build index babd9b72..270e482e 100644 --- a/meson.build +++ b/meson.build @@ -21,12 +21,25 @@ print(incdir) '''], check: true).stdout().strip() # Get BLAS +# +# link_mkl links every extension against MKL (conda/source builds, where the +# environment provides a complete MKL). mkl_backend builds only the _scs_mkl +# extension against MKL while every other extension uses the platform BLAS: +# this is the wheel configuration -- MKL is not vendored into wheels because +# its CPU dispatch kernels are loaded via dlopen, invisible to auditwheel +# (cvxgrp/scs#423); instead the scs[mkl] extra supplies Intel's official +# `mkl` wheels at runtime. blas_deps = [] +mkl_blas_deps = [] mkl_pkg_name = '' -if get_option('link_mkl') +want_mkl = get_option('link_mkl') or get_option('mkl_backend') +if want_mkl if get_option('use_blas64') and get_option('int32') error('MKL BLAS64 requires 64-bit SCS integers. Re-run Meson with -Dint32=false.') endif + if get_option('mkl_backend') and not get_option('link_mkl') and get_option('use_blas64') + error('mkl_backend with use_blas64 is unsupported; use link_mkl for ILP64 MKL builds.') + endif # Link against MKL component libraries. The integer width must match: # use_blas64=false (default) -> LP64 (32-bit BLAS integers) # use_blas64=true -> ILP64 (64-bit BLAS integers) @@ -50,27 +63,34 @@ if get_option('link_mkl') if host_machine.system() == 'windows' # On Windows the conda MKL iomp .pc file has broken Cflags (raw path # without -I prefix). Use sequential MKL until Intel ships a fix. - blas_deps = [dependency(_mkl_seq, required : false)] - if blas_deps[0].found() + mkl_blas_deps = [dependency(_mkl_seq, required : false)] + if mkl_blas_deps[0].found() mkl_pkg_name = _mkl_seq endif else - blas_deps = [dependency(_mkl_iomp, required : false)] - if not blas_deps[0].found() - blas_deps = [dependency(_mkl_seq, required : false)] - if blas_deps[0].found() + mkl_blas_deps = [dependency(_mkl_iomp, required : false)] + if not mkl_blas_deps[0].found() + mkl_blas_deps = [dependency(_mkl_seq, required : false)] + if mkl_blas_deps[0].found() mkl_pkg_name = _mkl_seq endif else mkl_pkg_name = _mkl_iomp endif endif - if not blas_deps[0].found() - blas_deps = [cc.find_library('mkl_rt', required : false)] + if not mkl_blas_deps[0].found() + mkl_blas_deps = [cc.find_library('mkl_rt', required : false)] endif - if not blas_deps[0].found() - blas_deps = [dependency('mkl-sdl', required : false)] + if not mkl_blas_deps[0].found() + mkl_blas_deps = [dependency('mkl-sdl', required : false)] endif + if not mkl_blas_deps[0].found() and not get_option('sdist_mode') + error('MKL was requested (link_mkl/mkl_backend) but was not found.') + endif +endif + +if get_option('link_mkl') + blas_deps = mkl_blas_deps else if host_machine.system() == 'darwin' blas_deps = [dependency('Accelerate')] @@ -131,11 +151,11 @@ if get_option('native_arch') endif endif -_deps = [blas_deps] -_deps += dependency('threads') +_base_deps = [dependency('threads')] if get_option('use_openmp') - _deps += dependency('openmp') + _base_deps += dependency('openmp') endif +_deps = [blas_deps] + _base_deps is_linux = host_machine.system() == 'linux' if is_linux @@ -325,7 +345,7 @@ if get_option('use_gpu') ) endif -if get_option('link_mkl') +if want_mkl # The MKL backend calls MKL_Set_Interface_Layer() for a runtime interface # check. This symbol is only exported by mkl_rt (the single dynamic library), # not by component libraries (mkl_intel_lp64 etc.) that pkg-config may link. @@ -336,15 +356,15 @@ if get_option('link_mkl') endif if not mkl_rt_dep.found() and mkl_pkg_name != '' # Search next to the pkg-config-provided MKL component libraries. - _mkl_libdir = blas_deps[0].get_variable(pkgconfig: 'libdir', default_value: '') + _mkl_libdir = mkl_blas_deps[0].get_variable(pkgconfig: 'libdir', default_value: '') if _mkl_libdir != '' mkl_rt_dep = cc.find_library('mkl_rt', dirs: [_mkl_libdir], required : false) endif endif if not mkl_rt_dep.found() - error('link_mkl=true requires mkl_rt (or mkl-sdl). _scs_mkl calls MKL_Set_Interface_Layer() at startup and cannot be built safely without it.') + error('link_mkl/mkl_backend requires mkl_rt (or mkl-sdl). _scs_mkl calls MKL_Set_Interface_Layer() at startup and cannot be built safely without it.') endif - _mkl_deps = _deps + [mkl_rt_dep] + _mkl_deps = [mkl_blas_deps] + _base_deps + [mkl_rt_dep] py.extension_module( '_scs_mkl', 'scs/scspy.c', diff --git a/meson.options b/meson.options index c39e18b8..7a086b20 100644 --- a/meson.options +++ b/meson.options @@ -29,3 +29,5 @@ option('native_arch', type: 'boolean', value: false, description: 'Compile with -march=native for the current CPU. Improves performance but produces non-portable binaries. Enable when building from source for local use.') option('use_spectral_cones', type: 'boolean', value: false, description: 'Build with spectral cone support (logdet, nuclear norm, ell1, sum-of-largest). Requires LAPACK.') +option('mkl_backend', type: 'boolean', + value: false, description: 'Build the _scs_mkl extension against MKL while other extensions use the platform BLAS (wheel configuration; MKL supplied at runtime by the scs[mkl] extra)') diff --git a/pyproject.toml b/pyproject.toml index d9557c48..ef379ae2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -38,6 +38,19 @@ dependencies = [ 'scipy', ] +[project.optional-dependencies] +# MKL Pardiso backend for the pre-built Linux x86-64 wheels. The wheels link +# _scs_mkl against MKL but do not vendor it (MKL's dlopen'd CPU dispatch +# kernels are invisible to wheel-repair tools; see cvxgrp/scs#423): Intel's +# official wheels provide the complete library set at runtime. The version +# range must match the oneMKL generation the wheels are built against +# (currently the .so.3 ABI, oneMKL 2026); the clean-container CI smoke test +# enforces the pairing. Without this extra, AUTO falls back to QDLDL. +mkl = [ + 'mkl >=2026, <2027 ; platform_system == "Linux" and platform_machine == "x86_64"', + 'intel-openmp >=2026, <2027 ; platform_system == "Linux" and platform_machine == "x86_64"', +] + [tool.cibuildwheel] skip = [ "*-win32", # fails on locating Python headers, probably meson.build is misconfigured @@ -66,21 +79,32 @@ repair-wheel-command = "delvewheel repair -w {dest_dir} {wheel}" # Openblas installation for 3 different linux images -# x86_64 Linux and Windows wheels link OpenBLAS. They were switched to -# MKL in #187, which shipped 3.3.0 wheels whose vendored MKL was missing -# its dlopen'd dispatch kernels (libmkl_def/avx2/avx512 -- invisible to -# auditwheel/delvewheel, which only follow DT_NEEDED): on any machine -# without MKL already on the loader path, the first solve aborted the -# interpreter via MKL's own exit(2). See #423. The build-container test -# could not catch it because LD_LIBRARY_PATH pointed at the container's -# system oneAPI install. MKL wheels may return once kernel vendoring is -# solved and gated by the clean-container wheel test. +# x86_64 Linux: default backends (_scs_direct/_scs_indirect) link vendored +# OpenBLAS; _scs_mkl links MKL *without vendoring it* (mkl_backend mode). +# The 3.3.0 wheels vendored MKL and shipped it incomplete -- its dlopen'd +# CPU dispatch kernels are invisible to auditwheel, and every solve aborted +# on machines without MKL on the loader path (#423). At runtime MKL now +# comes from Intel's official wheels via the scs[mkl] extra instead. +# +# Two hard rules, both learned from #423: +# 1. No LD_LIBRARY_PATH in `environment`: the wheel tests must run exactly +# as clean as a user's machine. The oneAPI paths are scoped to the +# repair command only, where auditwheel needs to *find* the MKL libs it +# is told to exclude. +# 2. upload_pypi is gated on the wheel_smoke_clean_env job, which installs +# the wheel in a pristine container with and without scs[mkl]. [[tool.cibuildwheel.overrides]] select = "*-manylinux_x86_64" inherit.before-all = "append" before-all = [ - "dnf install -y openblas-devel", + "dnf install -y openblas-devel dnf-plugins-core", + "dnf config-manager --add-repo https://yum.repos.intel.com/oneapi", + "rpm --import https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB", + "dnf install -y intel-oneapi-mkl-devel", ] +environment = {PKG_CONFIG_PATH = "/opt/intel/oneapi/mkl/latest/lib/pkgconfig:$PKG_CONFIG_PATH"} +config-settings = {setup-args = "-Dmkl_backend=true"} +repair-wheel-command = "LD_LIBRARY_PATH='/opt/intel/oneapi/mkl/latest/lib:/opt/intel/oneapi/mkl/latest/lib/intel64:/opt/intel/oneapi/compiler/latest/lib' auditwheel repair --exclude 'libmkl_*' --exclude 'libiomp5*' -w {dest_dir} {wheel}" [[tool.cibuildwheel.overrides]] select = "*-manylinux_aarch64" diff --git a/scs/py/__init__.py b/scs/py/__init__.py index 07a5e93e..8a2e4d69 100644 --- a/scs/py/__init__.py +++ b/scs/py/__init__.py @@ -37,9 +37,84 @@ class LinearSolver(enum.Enum): CUDSS = "cudss" +def _preload_intel_mkl(): + """Preload MKL from Intel's official ``mkl`` PyPI wheels (``scs[mkl]``). + + The pre-built Linux x86-64 wheels link the _scs_mkl extension against MKL + without vendoring it: MKL loads its CPU dispatch kernels via dlopen, which + wheel-repair tools cannot see, so a vendored MKL is incomplete and aborts + the process at solve time (cvxgrp/scs#423). The ``scs[mkl]`` extra instead + installs Intel's own ``mkl`` and ``intel-openmp`` wheels, whose complete, + internally-consistent libraries land outside the default loader path + (``/lib``). This preloads them so the extension's link-time + dependencies resolve; the dispatch kernels are then found by MKL's own + loader next to its libmkl_core. Returns True if anything was preloaded. + + Libraries are loaded RTLD_LOCAL so MKL's BLAS symbols cannot interpose on + other libraries in the process (e.g. NumPy's vendored OpenBLAS). + """ + if not sys.platform.startswith("linux"): + return False + import ctypes + import glob + import os + from importlib import metadata + + libdirs = [] + for pkg in ("mkl", "intel-openmp"): + try: + dist = metadata.distribution(pkg) + except metadata.PackageNotFoundError: + continue + for f in dist.files or (): + if f.name.startswith(("libmkl_", "libiomp5")): + d = os.path.dirname(os.fspath(dist.locate_file(f))) + if d not in libdirs and os.path.isdir(d): + libdirs.append(d) + if not libdirs: + # Fallback for installers that do not record RECORD data files. + for prefix in dict.fromkeys((sys.prefix, sys.base_prefix, sys.exec_prefix)): + d = os.path.join(prefix, "lib") + if glob.glob(os.path.join(d, "libmkl_core.so*")): + libdirs.append(d) + if not libdirs: + return False + + # Dependency-safe order: OpenMP runtime, then MKL core, threading layer, + # interface layer, and the single-dynamic-library runtime (used by the + # extension's interface-layer check). + patterns = ( + "libiomp5.so", + "libmkl_core.so*", + "libmkl_sequential.so*", + "libmkl_intel_thread.so*", + "libmkl_intel_lp64.so*", + "libmkl_intel_ilp64.so*", + "libmkl_rt.so*", + ) + loaded = False + for pattern in patterns: + for d in libdirs: + for path in sorted(glob.glob(os.path.join(d, pattern))): + try: + ctypes.CDLL(path, mode=ctypes.RTLD_LOCAL) + loaded = True + except OSError: + pass + return loaded + + def _load_module(name): from importlib import import_module - return import_module(f"scs.{name}") + try: + return import_module(f"scs.{name}") + except ImportError: + # The wheel _scs_mkl extension resolves MKL from the `mkl` PyPI package + # (scs[mkl]) rather than vendored libraries; make those loadable and + # retry. Without them the ImportError propagates and AUTO falls back. + if name != "_scs_mkl" or not _preload_intel_mkl(): + raise + return import_module(f"scs.{name}") def _resolve_auto(): diff --git a/test/test_solve_random_cone_prob_mkl.py b/test/test_solve_random_cone_prob_mkl.py index efb343ed..5c475650 100644 --- a/test/test_solve_random_cone_prob_mkl.py +++ b/test/test_solve_random_cone_prob_mkl.py @@ -13,9 +13,11 @@ # Uses scs to solve a random cone problem # ############################################# -# MKL is shipped in manylinux x86_64 and Windows wheels, but not in -# musllinux or macOS or aarch64 wheels. Skip on platforms where MKL -# is never available; on MKL platforms fail hard if the import is missing. +# The MKL backend is available on x86-64 Linux (wheels: via the scs[mkl] +# extra; source builds: link_mkl/mkl_backend) and on Windows source builds. +# Skip on platforms where MKL is never available, and skip when the +# extension is present but its MKL runtime is not (e.g. a wheel install +# without scs[mkl]). if sys.platform == "darwin": pytest.skip("MKL is not available on macOS", allow_module_level=True) if sys.platform == "linux" and platform.machine() != "x86_64": @@ -24,8 +26,8 @@ try: from scs import _scs_mkl # noqa: E402 except ImportError: - # musllinux x86_64 ships openblas, not MKL - pytest.skip("MKL module not installed", allow_module_level=True) + # openblas-only builds (musllinux), or a wheel without the scs[mkl] extra + pytest.skip("MKL backend not importable", allow_module_level=True) # cone: K = { From d16f2dab9a6201fc6abffb28beb7df0a6fabf776 Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 15:45:40 +0100 Subject: [PATCH 04/19] Restore compiler pkgconfig dir so wheels link threaded (iomp) MKL Without /opt/intel/oneapi/compiler/latest/lib/pkgconfig on PKG_CONFIG_PATH the mkl-dynamic-lp64-iomp dependency fails to resolve and meson silently falls back to sequential MKL; test_mkl_module_links_intel_openmp caught the downgrade. Note this env var carries no LD_LIBRARY_PATH: the mkl tests still run in-container only via the extension's baked RUNPATH to the oneAPI install, and the clean-container smoke job remains the authority on what the wheel actually ships. --- pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index ef379ae2..22f209cc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -102,7 +102,7 @@ before-all = [ "rpm --import https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB", "dnf install -y intel-oneapi-mkl-devel", ] -environment = {PKG_CONFIG_PATH = "/opt/intel/oneapi/mkl/latest/lib/pkgconfig:$PKG_CONFIG_PATH"} +environment = {PKG_CONFIG_PATH = "/opt/intel/oneapi/mkl/latest/lib/pkgconfig:/opt/intel/oneapi/compiler/latest/lib/pkgconfig:$PKG_CONFIG_PATH"} config-settings = {setup-args = "-Dmkl_backend=true"} repair-wheel-command = "LD_LIBRARY_PATH='/opt/intel/oneapi/mkl/latest/lib:/opt/intel/oneapi/mkl/latest/lib/intel64:/opt/intel/oneapi/compiler/latest/lib' auditwheel repair --exclude 'libmkl_*' --exclude 'libiomp5*' -w {dest_dir} {wheel}" From fb28639e29915386b8b36bc9c8c918352a2928ab Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 16:04:28 +0100 Subject: [PATCH 05/19] Smoke test: select the cp312 wheel matching the container python --- .github/workflows/build.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index adffcc07..6e4cda08 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -348,7 +348,7 @@ jobs: EOF docker run --rm -v "$PWD:/work" -w /tmp -e VARIANT=${{ matrix.variant }} python:3.12-slim bash -ec ' pip install --quiet numpy scipy - WHL=$(ls /work/wheelhouse/scs-*manylinux*x86_64.whl) + WHL=$(ls /work/wheelhouse/scs-*cp312-cp312-manylinux*x86_64.whl) if [ "$VARIANT" = mkl ]; then pip install --quiet "${WHL}[mkl]" else From bdf282b4b94d2d0b1b02d2b83bff287a87214603 Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 16:23:53 +0100 Subject: [PATCH 06/19] Smoke test: identify backends by __file__, add scs[mkl] import diagnostics The extensions share the PyModuleDef name "_scs" (single-phase init), so __name__ cannot distinguish them. Also surface the pip resolution and the exact ImportError when the mkl variant fails to load. --- .github/workflows/build.yml | 54 ++++++++++++++++++++++++------------- 1 file changed, 36 insertions(+), 18 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 6e4cda08..61085c69 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -311,7 +311,7 @@ jobs: - name: Solve in pristine python:3.12-slim (${{ matrix.variant }}) run: | cat > smoke.py <<'EOF' - import sys + import importlib, os, sys, traceback import numpy as np, scipy.sparse as sp, scs mode = sys.argv[1] print("scs", scs.__version__, "mode", mode) @@ -324,36 +324,54 @@ jobs: "c": np.array([1.0, 1.0]), } cone = {"z": 1, "l": 2, "s": [2]} - sol = scs.SCS(data, cone, verbose=True).solve() - assert sol["info"]["status"] == "solved", sol["info"]["status"] - auto_mod = scs._resolve_auto().__name__ - print("auto resolves to", auto_mod) - solvers = ["qdldl", "cpu_indirect"] + + # Exercise the scs[mkl] loader shim explicitly, with diagnostics. + mkl_err = None + try: + scs._load_module("_scs_mkl") + except ImportError as e: + mkl_err = e + traceback.print_exc() + libdir = os.path.join(sys.prefix, "lib") + libs = sorted(f for f in os.listdir(libdir) if f.startswith(("libmkl", "libiomp"))) + print(libdir, "->", libs[:8], "..." if len(libs) > 8 else "") + try: + import importlib.metadata as md + print("mkl dist:", md.version("mkl"), "intel-openmp dist:", md.version("intel-openmp")) + except Exception as e2: + print("intel dists:", e2) if mode == "mkl": - assert auto_mod.endswith("_scs_mkl"), auto_mod - solvers.append("mkl") + assert mkl_err is None, f"scs[mkl] installed but _scs_mkl failed: {mkl_err}" else: - assert auto_mod.endswith("_scs_direct"), auto_mod # The wheel ships _scs_mkl but its MKL comes from scs[mkl]; # without the extra it must fail cleanly, never crash. - try: - scs.SCS(data, cone, linear_solver="mkl") - raise SystemExit("unvendored _scs_mkl imported without scs[mkl]") - except ImportError: - print("mkl backend unavailable without scs[mkl], as expected") - for name in solvers: - sol = scs.SCS(data, cone, verbose=True, linear_solver=name).solve() + assert mkl_err is not None, "unvendored _scs_mkl imported without scs[mkl]" + print("mkl backend unavailable without scs[mkl], as expected") + + # The extensions share the PyModuleDef name "_scs" (single-phase + # init), so identify the AUTO backend by file, not __name__. + auto_file = os.path.basename(getattr(scs._resolve_auto(), "__file__", "")) + print("auto resolves to", auto_file) + want = "_scs_mkl" if mode == "mkl" else "_scs_direct" + assert auto_file.startswith(want), (auto_file, want) + + solvers = ["qdldl", "cpu_indirect"] + (["mkl"] if mode == "mkl" else []) + for name in solvers + ["auto"]: + kw = {} if name == "auto" else {"linear_solver": name} + sol = scs.SCS(data, cone, verbose=True, **kw).solve() assert sol["info"]["status"] == "solved", (name, sol["info"]["status"]) print(name, "ok:", sol["info"].get("lin_sys_solver", "n/a")) + print("ALL OK:", mode) EOF docker run --rm -v "$PWD:/work" -w /tmp -e VARIANT=${{ matrix.variant }} python:3.12-slim bash -ec ' pip install --quiet numpy scipy WHL=$(ls /work/wheelhouse/scs-*cp312-cp312-manylinux*x86_64.whl) if [ "$VARIANT" = mkl ]; then - pip install --quiet "${WHL}[mkl]" + pip install "${WHL}[mkl]" else - pip install --quiet "$WHL" + pip install "$WHL" fi + pip list | grep -i -E "scs|mkl|openmp" || true python /work/smoke.py "$VARIANT"' build_sdist: From fbbcf234dc38574995989566c779cb106f0b4c46 Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 16:46:03 +0100 Subject: [PATCH 07/19] Resolve scs[mkl] via a prefix-relative RUNPATH instead of ctypes preloads MKL's component libraries reference each other and cannot be eagerly bound one at a time; ctypes.CDLL always forces RTLD_NOW, so the preload loop silently loaded nothing and _scs_mkl still failed to import (diagnosed in CI: all libraries present in /lib, every CDLL swallowed an OSError). Primary fix: after auditwheel repair, patchelf an '$ORIGIN/../../../../lib' RUNPATH onto _scs_mkl (site-packages/scs is four levels below the prefix in every standard layout), so the dynamic loader resolves the whole MKL group in one dlopen with correct mutual binding and no symbol-scope widening. auditwheel preserves existing RUNPATHs (observed in CI), so patching after repair is stable. The Python shim remains as a fallback for non-standard layouts, now calling dlopen(3) directly with RTLD_LAZY | RTLD_LOCAL. --- .github/scripts/add_mkl_rpath.sh | 28 ++++++++++++++++++++++++++++ pyproject.toml | 2 +- scs/py/__init__.py | 30 ++++++++++++++++++------------ 3 files changed, 47 insertions(+), 13 deletions(-) create mode 100755 .github/scripts/add_mkl_rpath.sh diff --git a/.github/scripts/add_mkl_rpath.sh b/.github/scripts/add_mkl_rpath.sh new file mode 100755 index 00000000..69715499 --- /dev/null +++ b/.github/scripts/add_mkl_rpath.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# Add a prefix-relative RUNPATH to the repaired wheel's _scs_mkl extension. +# +# The wheel does not vendor MKL (its dlopen'd CPU dispatch kernels are +# invisible to auditwheel; cvxgrp/scs#423). The scs[mkl] extra installs +# Intel's official wheels into /lib, and site-packages/scs sits +# exactly four levels below the prefix in every standard layout (venv, +# conda, user site, system), so $ORIGIN/../../../../lib lets the dynamic +# loader resolve the whole MKL component group in one dlopen -- with the +# correct mutual binding MKL's libraries require, and without widening +# any symbol scope (NumPy's vendored OpenBLAS is unaffected). +set -euo pipefail +dest_dir="$1" +whl=$(ls -t "$dest_dir"/scs-*.whl | head -1) +tmp=$(mktemp -d) +python -m pip install --quiet wheel +python -m wheel unpack --dest "$tmp" "$whl" +unpacked=$(ls -d "$tmp"/scs-*) +patched=0 +for so in "$unpacked"/scs/_scs_mkl*.so; do + [ -e "$so" ] || continue + patchelf --add-rpath '$ORIGIN/../../../../lib' "$so" + echo "add_mkl_rpath: $(basename "$so") rpath -> $(patchelf --print-rpath "$so")" + patched=1 +done +[ "$patched" -eq 1 ] || { echo "add_mkl_rpath: no _scs_mkl extension found" >&2; exit 1; } +python -m wheel pack --dest-dir "$dest_dir" "$unpacked" +rm -rf "$tmp" diff --git a/pyproject.toml b/pyproject.toml index 22f209cc..1f53e87f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -104,7 +104,7 @@ before-all = [ ] environment = {PKG_CONFIG_PATH = "/opt/intel/oneapi/mkl/latest/lib/pkgconfig:/opt/intel/oneapi/compiler/latest/lib/pkgconfig:$PKG_CONFIG_PATH"} config-settings = {setup-args = "-Dmkl_backend=true"} -repair-wheel-command = "LD_LIBRARY_PATH='/opt/intel/oneapi/mkl/latest/lib:/opt/intel/oneapi/mkl/latest/lib/intel64:/opt/intel/oneapi/compiler/latest/lib' auditwheel repair --exclude 'libmkl_*' --exclude 'libiomp5*' -w {dest_dir} {wheel}" +repair-wheel-command = "LD_LIBRARY_PATH='/opt/intel/oneapi/mkl/latest/lib:/opt/intel/oneapi/mkl/latest/lib/intel64:/opt/intel/oneapi/compiler/latest/lib' auditwheel repair --exclude 'libmkl_*' --exclude 'libiomp5*' -w {dest_dir} {wheel} && bash /project/.github/scripts/add_mkl_rpath.sh {dest_dir}" [[tool.cibuildwheel.overrides]] select = "*-manylinux_aarch64" diff --git a/scs/py/__init__.py b/scs/py/__init__.py index 8a2e4d69..15c96080 100644 --- a/scs/py/__init__.py +++ b/scs/py/__init__.py @@ -44,14 +44,20 @@ def _preload_intel_mkl(): without vendoring it: MKL loads its CPU dispatch kernels via dlopen, which wheel-repair tools cannot see, so a vendored MKL is incomplete and aborts the process at solve time (cvxgrp/scs#423). The ``scs[mkl]`` extra instead - installs Intel's own ``mkl`` and ``intel-openmp`` wheels, whose complete, - internally-consistent libraries land outside the default loader path - (``/lib``). This preloads them so the extension's link-time - dependencies resolve; the dispatch kernels are then found by MKL's own - loader next to its libmkl_core. Returns True if anything was preloaded. - - Libraries are loaded RTLD_LOCAL so MKL's BLAS symbols cannot interpose on - other libraries in the process (e.g. NumPy's vendored OpenBLAS). + installs Intel's own ``mkl`` and ``intel-openmp`` wheels into + ``/lib``. + + The primary lookup mechanism is a ``$ORIGIN``-relative RUNPATH baked into + the wheel's extension (site-packages/scs is four levels below the prefix + in every standard layout), which lets the loader resolve MKL's mutually + referencing component libraries as one group. This fallback covers + non-standard layouts where that relative path misses: it dlopens the + libraries RTLD_LAZY | RTLD_LOCAL so the extension's DT_NEEDED entries + resolve from the link map. LAZY is required -- the components cannot be + eagerly bound one at a time -- and ctypes.CDLL always forces RTLD_NOW, + so this calls dlopen(3) directly. RTLD_LOCAL keeps MKL's BLAS from + interposing on other libraries (e.g. NumPy's vendored OpenBLAS). + Returns True if anything was loaded. """ if not sys.platform.startswith("linux"): return False @@ -92,15 +98,15 @@ def _preload_intel_mkl(): "libmkl_intel_ilp64.so*", "libmkl_rt.so*", ) + dlopen = ctypes.CDLL(None).dlopen + dlopen.restype = ctypes.c_void_p + dlopen.argtypes = (ctypes.c_char_p, ctypes.c_int) loaded = False for pattern in patterns: for d in libdirs: for path in sorted(glob.glob(os.path.join(d, pattern))): - try: - ctypes.CDLL(path, mode=ctypes.RTLD_LOCAL) + if dlopen(os.fsencode(path), os.RTLD_LAZY | os.RTLD_LOCAL): loaded = True - except OSError: - pass return loaded From 41209d6c7412cec66f94941059c49a2df3219f79 Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 17:28:34 +0100 Subject: [PATCH 08/19] README: recommend scs[mkl] prominently in the install section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per review direction: MKL is opt-in but should be loud — for most users on x86-64 Linux it is strictly better than the built-in solver, so the install section now leads with a GitHub IMPORTANT callout for pip install "scs[mkl]", and the backends section defers to it. --- README.md | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index efa3a8ff..155c7511 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,21 @@ The full documentation is available [here](https://www.cvxgrp.org/scs/). pip install scs ``` +> [!IMPORTANT] +> **On x86-64 Linux, install the MKL extra instead — recommended for nearly +> everyone:** +> +> ```bash +> pip install "scs[mkl]" +> ``` +> +> This enables the MKL Pardiso direct linear solver, which is faster than the +> built-in solver for most problems — often dramatically so on larger ones — +> and SCS uses it automatically when it is installed; no code or settings +> changes are needed. The MKL runtime comes from Intel's official `mkl` wheels +> (roughly an extra 240 MB on disk). A plain `pip install scs` works +> everywhere and falls back to the built-in QDLDL solver. + To install from source: ```bash git clone --recursive https://github.com/bodono/scs-python.git @@ -42,20 +57,14 @@ Available values: `AUTO`, `QDLDL`, `CPU_INDIRECT`, `MKL`, `ACCELERATE`, The pre-built wheels (`pip install scs`) link OpenBLAS on Linux and Windows, and Apple Accelerate on macOS. On x86-64 Linux the MKL Pardiso backend is -available as an extra: - -```bash -pip install "scs[mkl]" -``` - -This pulls Intel's official `mkl` wheels, which provide the complete MKL -runtime (SCS wheels deliberately do not vendor MKL: its CPU dispatch -kernels are loaded via `dlopen`, invisible to wheel-repair tools, and an -incompletely vendored MKL aborts the process at solve time). With the extra -installed `AUTO` selects MKL; without it, `AUTO` falls back to QDLDL. The -MKL backend is also available in source builds (e.g. conda environments -providing MKL). When installing from source, additional backends can be -enabled with build-time flags: +available via `pip install "scs[mkl]"` (see [Installation](#installation) — +recommended): Intel's official `mkl` wheels provide the complete MKL runtime, +and `AUTO` selects MKL whenever it is importable. SCS wheels deliberately do +not vendor MKL — its CPU dispatch kernels are loaded via `dlopen`, invisible +to wheel-repair tools, and an incompletely vendored MKL aborts the process at +solve time (cvxgrp/scs#423). The MKL backend is also available in source +builds (e.g. conda environments providing MKL). When installing from source, +additional backends can be enabled with build-time flags: ```bash # MKL Pardiso direct solver From 4c6f456911c3583b6b5bac810f77d46a8e8d64ac Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 18:22:41 +0100 Subject: [PATCH 09/19] Fix reviewed #232 issues: normalize RPATH, audit all wheels, widen the clean gate - add_mkl_rpath.sh uses patchelf --set-rpath: the meson build bakes the container's /opt/intel paths into _scs_mkl and auditwheel preserves them; shipping those would let a build-machine-layout MKL install shadow the scs[mkl] runtime and let container tests silently resolve the build MKL. The script now also asserts the exact final RUNPATH and the exact MKL/iomp NEEDED set. - New wheel_audit job: every .so in every Linux wheel (all pythons, x86_64/aarch64/musllinux) must carry only $ORIGIN-relative RPATH entries, and every _scs_mkl must have the exact expected MKL NEEDED set. Gates upload_pypi. - Clean-container smoke matrix now covers oldest (3.9), newest stable (3.14) and free-threaded (3.14t via uv) wheels, both with and without scs[mkl]; remaining ABIs are covered by the audit. - README: correct scs[mkl] footprint (~300 MB download / ~1 GB on disk), document exactly which install modes the extra supports (manylinux x86-64 glibc 2.28+, standard prefixes; not musllinux, pip --target, or sdist builds), and stop implying Windows wheels contain an MKL backend. --- .github/scripts/add_mkl_rpath.sh | 29 +++++++- .github/workflows/build.yml | 121 +++++++++++++++++++++++++------ README.md | 12 ++- 3 files changed, 132 insertions(+), 30 deletions(-) diff --git a/.github/scripts/add_mkl_rpath.sh b/.github/scripts/add_mkl_rpath.sh index 69715499..60f2c9b5 100755 --- a/.github/scripts/add_mkl_rpath.sh +++ b/.github/scripts/add_mkl_rpath.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Add a prefix-relative RUNPATH to the repaired wheel's _scs_mkl extension. +# Normalize the RUNPATH of the repaired wheel's _scs_mkl extension. # # The wheel does not vendor MKL (its dlopen'd CPU dispatch kernels are # invisible to auditwheel; cvxgrp/scs#423). The scs[mkl] extra installs @@ -9,6 +9,15 @@ # loader resolve the whole MKL component group in one dlopen -- with the # correct mutual binding MKL's libraries require, and without widening # any symbol scope (NumPy's vendored OpenBLAS is unaffected). +# +# --set-rpath (not --add-rpath): the meson build bakes the container's +# /opt/intel/... paths into the extension, and auditwheel preserves them. +# Shipping those would let a build-machine-layout MKL installation win +# over the scs[mkl] runtime, and would let container tests silently +# resolve the build MKL. The RUNPATH must be exactly the one relative +# entry, and the MKL-related NEEDED set must be exactly what the loader +# shim and the mkl PyPI pin (mkl>=2026,<2027 -- the .so.3 ABI) provide; +# both are asserted here so a drift fails the build, not a user. set -euo pipefail dest_dir="$1" whl=$(ls -t "$dest_dir"/scs-*.whl | head -1) @@ -16,11 +25,25 @@ tmp=$(mktemp -d) python -m pip install --quiet wheel python -m wheel unpack --dest "$tmp" "$whl" unpacked=$(ls -d "$tmp"/scs-*) +want_rpath='$ORIGIN/../../../../lib' +want_needed='libiomp5.so libmkl_core.so.3 libmkl_intel_lp64.so.3 libmkl_intel_thread.so.3 libmkl_rt.so.3' patched=0 for so in "$unpacked"/scs/_scs_mkl*.so; do [ -e "$so" ] || continue - patchelf --add-rpath '$ORIGIN/../../../../lib' "$so" - echo "add_mkl_rpath: $(basename "$so") rpath -> $(patchelf --print-rpath "$so")" + patchelf --set-rpath "$want_rpath" "$so" + got_rpath=$(patchelf --print-rpath "$so") + if [ "$got_rpath" != "$want_rpath" ]; then + echo "add_mkl_rpath: unexpected RUNPATH '$got_rpath' on $(basename "$so")" >&2 + exit 1 + fi + got_needed=$(patchelf --print-needed "$so" | grep -E '^lib(mkl|iomp)' | sort | tr '\n' ' ' | sed 's/ $//') + if [ "$got_needed" != "$want_needed" ]; then + echo "add_mkl_rpath: NEEDED drift on $(basename "$so")" >&2 + echo " want: $want_needed" >&2 + echo " got: $got_needed" >&2 + exit 1 + fi + echo "add_mkl_rpath: $(basename "$so") rpath='$got_rpath' needed ok" patched=1 done [ "$patched" -eq 1 ] || { echo "add_mkl_rpath: no _scs_mkl extension found" >&2; exit 1; } diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 61085c69..f3cafe5e 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -279,6 +279,62 @@ jobs: name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }} path: ./wheelhouse/*.whl + wheel_audit: + # Static gate for #423-class defects across EVERY Linux wheel (all + # pythons, x86_64 + aarch64 + musllinux): no .so may carry an absolute + # RPATH/RUNPATH entry (a build-container path in a shipped wheel lets a + # matching system MKL shadow the scs[mkl] runtime), and _scs_mkl's + # MKL-related NEEDED set must be exactly what the mkl>=2026,<2027 pin + # provides (.so.3 ABI). + name: Audit wheel ELF metadata + needs: build_wheels + runs-on: ubuntu-latest + steps: + - uses: actions/download-artifact@v7 + with: + pattern: cibw-wheels-ubuntu-* + path: wheelhouse + merge-multiple: true + + - name: Audit RPATH and NEEDED across all Linux wheels + run: | + cat > audit.py <<'EOF' + import glob, os, re, subprocess, sys, tempfile, zipfile + bad = [] + whls = sorted(glob.glob("wheelhouse/*.whl")) + assert whls, "no wheels downloaded" + nmkl = 0 + for whl in whls: + with tempfile.TemporaryDirectory() as td: + with zipfile.ZipFile(whl) as z: + z.extractall(td) + for so in glob.glob(os.path.join(td, "**", "*.so*"), recursive=True): + out = subprocess.run(["readelf", "-d", so], + capture_output=True, text=True).stdout + rel = os.path.relpath(so, td) + for m in re.finditer(r"\((?:RPATH|RUNPATH)\)[^\[]*\[([^\]]*)\]", out): + for entry in m.group(1).split(":"): + if entry and not entry.startswith("$ORIGIN"): + bad.append((os.path.basename(whl), rel, + f"non-relative rpath entry '{entry}'")) + if "_scs_mkl" in os.path.basename(so): + nmkl += 1 + needed = re.findall(r"\(NEEDED\)[^\[]*\[([^\]]+)\]", out) + mkl = sorted(n for n in needed + if n.startswith(("libmkl", "libiomp"))) + want = ["libiomp5.so", "libmkl_core.so.3", + "libmkl_intel_lp64.so.3", + "libmkl_intel_thread.so.3", "libmkl_rt.so.3"] + if mkl != want: + bad.append((os.path.basename(whl), rel, + f"NEEDED drift: {' '.join(mkl)}")) + print(f"audited {len(whls)} wheels, {nmkl} _scs_mkl extensions") + for whl, so, msg in bad: + print(f"BAD: {whl} :: {so} :: {msg}") + sys.exit(1 if bad else 0) + EOF + python3 audit.py + wheel_smoke_clean_env: # Regression gate for #423: install the built manylinux x86_64 wheel in a # pristine container (no BLAS, no LD_LIBRARY_PATH, no site packages) and @@ -287,20 +343,23 @@ jobs: # environment satisfied MKL's dlopen'd dispatch kernels; only a clean # environment exercises what the wheel actually ships. # - # Two variants: + # Variants: # default -- plain `pip install scs`: AUTO must resolve to QDLDL and the # unvendored _scs_mkl must fail with a clean ImportError, never crash. # mkl -- `pip install scs[mkl]`: MKL comes from Intel's official wheels; # AUTO must resolve to MKL and the mkl backend must solve. This also # enforces that the oneMKL generation the wheel links matches the # pinned `mkl` PyPI dependency range. - name: Clean-container wheel smoke test (${{ matrix.variant }}) + # Pythons: oldest (3.9), newest stable (3.14), free-threaded (3.14t via + # uv, python-build-standalone). Remaining ABIs are covered by wheel_audit. + name: Clean-container wheel smoke test (${{ matrix.variant }}, ${{ matrix.python }}) needs: build_wheels runs-on: ubuntu-latest strategy: fail-fast: false matrix: variant: [default, mkl] + python: ["3.9", "3.14", "3.14t"] steps: - uses: actions/download-artifact@v7 with: @@ -308,13 +367,13 @@ jobs: path: wheelhouse merge-multiple: true - - name: Solve in pristine python:3.12-slim (${{ matrix.variant }}) + - name: Solve in pristine container (${{ matrix.variant }}, ${{ matrix.python }}) run: | cat > smoke.py <<'EOF' import importlib, os, sys, traceback import numpy as np, scipy.sparse as sp, scs mode = sys.argv[1] - print("scs", scs.__version__, "mode", mode) + print("scs", scs.__version__, "mode", mode, "python", sys.version) # LP block plus a small PSD cone: the PSD projection goes through # the wheel's BLAS/LAPACK, the exact path #423 failed to ship. data = { @@ -325,21 +384,18 @@ jobs: } cone = {"z": 1, "l": 2, "s": [2]} - # Exercise the scs[mkl] loader shim explicitly, with diagnostics. + # Exercise the scs[mkl] loader path explicitly, with diagnostics. mkl_err = None try: scs._load_module("_scs_mkl") except ImportError as e: mkl_err = e - traceback.print_exc() - libdir = os.path.join(sys.prefix, "lib") - libs = sorted(f for f in os.listdir(libdir) if f.startswith(("libmkl", "libiomp"))) - print(libdir, "->", libs[:8], "..." if len(libs) > 8 else "") - try: - import importlib.metadata as md - print("mkl dist:", md.version("mkl"), "intel-openmp dist:", md.version("intel-openmp")) - except Exception as e2: - print("intel dists:", e2) + if mode == "mkl": + traceback.print_exc() + libdir = os.path.join(sys.prefix, "lib") + libs = sorted(f for f in os.listdir(libdir) + if f.startswith(("libmkl", "libiomp"))) if os.path.isdir(libdir) else [] + print(libdir, "->", libs[:8], "..." if len(libs) > 8 else "") if mode == "mkl": assert mkl_err is None, f"scs[mkl] installed but _scs_mkl failed: {mkl_err}" else: @@ -363,16 +419,32 @@ jobs: print(name, "ok:", sol["info"].get("lin_sys_solver", "n/a")) print("ALL OK:", mode) EOF - docker run --rm -v "$PWD:/work" -w /tmp -e VARIANT=${{ matrix.variant }} python:3.12-slim bash -ec ' - pip install --quiet numpy scipy - WHL=$(ls /work/wheelhouse/scs-*cp312-cp312-manylinux*x86_64.whl) + case "${{ matrix.python }}" in + 3.9) IMAGE=python:3.9-slim; TAG=cp39-cp39 ;; + 3.14) IMAGE=python:3.14-slim; TAG=cp314-cp314 ;; + 3.14t) IMAGE=ghcr.io/astral-sh/uv:debian-slim; TAG=cp314-cp314t ;; + esac + docker run --rm -v "$PWD:/work" -w /tmp \ + -e VARIANT=${{ matrix.variant }} -e PY=${{ matrix.python }} -e TAG=$TAG \ + "$IMAGE" bash -ec ' + if command -v uv >/dev/null; then + uv python install "$PY" + uv venv -p "$PY" /venv + PIP="uv pip install --python /venv/bin/python" + PYBIN=/venv/bin/python + else + PIP="pip install --quiet" + PYBIN=python + fi + $PIP numpy scipy + WHL=$(ls /work/wheelhouse/scs-*-${TAG}-manylinux*x86_64.whl) if [ "$VARIANT" = mkl ]; then - pip install "${WHL}[mkl]" + $PIP "${WHL}[mkl]" else - pip install "$WHL" + $PIP "$WHL" fi - pip list | grep -i -E "scs|mkl|openmp" || true - python /work/smoke.py "$VARIANT"' + $PYBIN -m pip list 2>/dev/null | grep -i -E "scs|mkl|openmp" || true + $PYBIN /work/smoke.py "$VARIANT"' build_sdist: name: Build source distribution @@ -391,9 +463,10 @@ jobs: path: dist/*.tar.gz upload_pypi: - # wheel_smoke_clean_env gates publishing: wheels that only work in the - # build container must never reach PyPI again (see #423). - needs: [build_wheels, build_sdist, wheel_smoke_clean_env] + # The clean-container smoke tests and the ELF audit gate publishing: + # wheels that only work in the build container, carry build-machine + # paths, or drift their MKL linkage must never reach PyPI (see #423). + needs: [build_wheels, build_sdist, wheel_smoke_clean_env, wheel_audit] runs-on: ubuntu-latest environment: pypi permissions: diff --git a/README.md b/README.md index 155c7511..74edb3ae 100644 --- a/README.md +++ b/README.md @@ -27,8 +27,14 @@ pip install scs > built-in solver for most problems — often dramatically so on larger ones — > and SCS uses it automatically when it is installed; no code or settings > changes are needed. The MKL runtime comes from Intel's official `mkl` wheels -> (roughly an extra 240 MB on disk). A plain `pip install scs` works -> everywhere and falls back to the built-in QDLDL solver. +> (roughly a 300 MB download, about 1 GB on disk). A plain `pip install scs` +> works everywhere and falls back to the built-in QDLDL solver. +> +> The extra is supported on the prebuilt manylinux x86-64 wheels (glibc +> 2.28+) in standard prefix layouts (venv, conda, system, user site). It does +> not work on musllinux/Alpine (Intel publishes no musl wheels), with +> `pip install --target`, or with source/sdist builds (build against your own +> MKL with `-Dlink_mkl=true` instead). To install from source: ```bash @@ -84,7 +90,7 @@ pip install . -Csetup-args=-Duse_spectral_cones=true ``` Notes: -- Linux x86_64 wheels are built and tested against threaded MKL, and CI asserts a `libiomp5` dependency on the packaged `_scs_mkl` extension. Windows currently falls back to sequential MKL because Intel's conda `pkg-config` metadata for the threaded variant is still broken. +- Linux x86_64 wheels ship a `_scs_mkl` extension linked against threaded MKL (CI asserts its exact MKL/`libiomp5` linkage); the runtime comes from `scs[mkl]`. Windows wheels do not include the MKL backend; Windows source builds use sequential MKL because Intel's conda `pkg-config` metadata for the threaded variant is still broken. - `BLAS64` is a general SCS build mode for ILP64 BLAS/LAPACK libraries, not an MKL-only feature. - For the MKL Pardiso backend specifically, `BLAS64` must be paired with 64-bit SCS integers (`DLONG` / `int32=false`), and SCS now fails early if another library in the process has already fixed MKL to an incompatible LP64/ILP64 interface layer. From 8cc38e08e0f29023140b412d27ee003d9cd15f12 Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 20:07:35 +0100 Subject: [PATCH 10/19] Make the wheel audit fail-closed Per review, the audit was fail-open on three axes: a wheel set with no _scs_mkl at all passed vacuously, readelf failures were silently ignored, and nothing pinned per-platform content. Now: - exact extension inventory per platform (x86_64: direct/indirect/mkl; aarch64+musllinux: direct/indirect, never mkl), - the _scs_mkl count must equal the number of x86_64 manylinux wheels, - readelf exit status is checked; a failure is a finding, - every wheel must vendor OpenBLAS and must never vendor MKL/iomp, - _scs_mkl's RUNPATH must be exactly ['$ORIGIN/../../../../lib']. --- .github/workflows/build.yml | 82 ++++++++++++++++++++++++++++++++----- 1 file changed, 72 insertions(+), 10 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index f3cafe5e..576961d1 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -303,31 +303,93 @@ jobs: bad = [] whls = sorted(glob.glob("wheelhouse/*.whl")) assert whls, "no wheels downloaded" + + def platform_of(whl): + name = os.path.basename(whl) + if "musllinux" in name: + return "musllinux" + if "aarch64" in name: + return "aarch64" + assert "manylinux" in name and "x86_64" in name, name + return "manylinux_x86_64" + + # Exact per-platform extension inventory: a missing backend is as + # much a packaging defect as a broken one (a vacuously green audit + # hid exactly this class of failure). + EXPECTED_EXTS = { + "manylinux_x86_64": {"_scs_direct", "_scs_indirect", "_scs_mkl"}, + "aarch64": {"_scs_direct", "_scs_indirect"}, + "musllinux": {"_scs_direct", "_scs_indirect"}, + } + WANT_MKL_NEEDED = ["libiomp5.so", "libmkl_core.so.3", + "libmkl_intel_lp64.so.3", + "libmkl_intel_thread.so.3", "libmkl_rt.so.3"] nmkl = 0 + expected_nmkl = sum(1 for w in whls + if platform_of(w) == "manylinux_x86_64") for whl in whls: + plat = platform_of(whl) with tempfile.TemporaryDirectory() as td: with zipfile.ZipFile(whl) as z: z.extractall(td) - for so in glob.glob(os.path.join(td, "**", "*.so*"), recursive=True): - out = subprocess.run(["readelf", "-d", so], - capture_output=True, text=True).stdout + sos = glob.glob(os.path.join(td, "**", "*.so*"), + recursive=True) + if not sos: + bad.append((os.path.basename(whl), "-", + "wheel contains no shared objects")) + continue + exts = {m.group(1) for so in sos + for m in [re.match(r"(_scs_[a-z]+)\.", + os.path.basename(so))] if m} + if exts != EXPECTED_EXTS[plat]: + bad.append((os.path.basename(whl), "-", + f"extension inventory {sorted(exts)} != " + f"expected {sorted(EXPECTED_EXTS[plat])}")) + # every wheel must vendor a BLAS for the default + # backends; MKL must never be vendored anywhere + vendored = [os.path.basename(so) for so in sos + if ".libs" in so] + if not any(v.startswith("libopenblas") for v in vendored): + bad.append((os.path.basename(whl), "-", + f"no vendored OpenBLAS: {vendored}")) + if any(v.startswith(("libmkl", "libiomp")) for v in vendored): + bad.append((os.path.basename(whl), "-", + f"MKL vendored into wheel: {vendored}")) + for so in sos: + proc = subprocess.run(["readelf", "-d", so], + capture_output=True, text=True) rel = os.path.relpath(so, td) - for m in re.finditer(r"\((?:RPATH|RUNPATH)\)[^\[]*\[([^\]]*)\]", out): + if proc.returncode != 0: + bad.append((os.path.basename(whl), rel, + f"readelf failed: {proc.stderr.strip()}")) + continue + out = proc.stdout + for m in re.finditer( + r"\((?:RPATH|RUNPATH)\)[^\[]*\[([^\]]*)\]", out): for entry in m.group(1).split(":"): if entry and not entry.startswith("$ORIGIN"): bad.append((os.path.basename(whl), rel, - f"non-relative rpath entry '{entry}'")) + f"non-relative rpath entry " + f"'{entry}'")) if "_scs_mkl" in os.path.basename(so): nmkl += 1 - needed = re.findall(r"\(NEEDED\)[^\[]*\[([^\]]+)\]", out) + needed = re.findall( + r"\(NEEDED\)[^\[]*\[([^\]]+)\]", out) mkl = sorted(n for n in needed if n.startswith(("libmkl", "libiomp"))) - want = ["libiomp5.so", "libmkl_core.so.3", - "libmkl_intel_lp64.so.3", - "libmkl_intel_thread.so.3", "libmkl_rt.so.3"] - if mkl != want: + if mkl != WANT_MKL_NEEDED: bad.append((os.path.basename(whl), rel, f"NEEDED drift: {' '.join(mkl)}")) + rp = re.findall( + r"\((?:RPATH|RUNPATH)\)[^\[]*\[([^\]]*)\]", out) + if rp != ["$ORIGIN/../../../../lib"]: + bad.append((os.path.basename(whl), rel, + f"_scs_mkl RUNPATH {rp} != " + f"['$ORIGIN/../../../../lib']")) + if nmkl != expected_nmkl: + bad.append(("(all)", "-", + f"found {nmkl} _scs_mkl extensions, expected " + f"{expected_nmkl}")) print(f"audited {len(whls)} wheels, {nmkl} _scs_mkl extensions") for whl, so, msg in bad: print(f"BAD: {whl} :: {so} :: {msg}") From 4d9999bf93708c3cf7c9aa9cbab18af7fb93a07f Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 20:50:54 +0100 Subject: [PATCH 11/19] Audit: include _scs_dense, forbid vacuity, split RPATH/RUNPATH, scan whole wheel - _scs_dense ships in every Linux wheel and was missing from the expected inventories, failing all 27 correct wheels. - Every platform group must be present in the downloaded artifacts and the expected _scs_mkl count must be positive, so an absent artifact group can never make the MKL checks vacuous. - Intel libraries are now rejected anywhere in the wheel, not only under .libs paths. - DT_RPATH and DT_RUNPATH are tracked separately: legacy DT_RPATH is forbidden outright, and _scs_mkl must carry exactly one DT_RUNPATH equal to $ORIGIN/../../../../lib. --- .github/workflows/build.yml | 50 +++++++++++++++++++++++++------------ 1 file changed, 34 insertions(+), 16 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 576961d1..2fefd9fa 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -317,16 +317,27 @@ jobs: # much a packaging defect as a broken one (a vacuously green audit # hid exactly this class of failure). EXPECTED_EXTS = { - "manylinux_x86_64": {"_scs_direct", "_scs_indirect", "_scs_mkl"}, - "aarch64": {"_scs_direct", "_scs_indirect"}, - "musllinux": {"_scs_direct", "_scs_indirect"}, + "manylinux_x86_64": {"_scs_direct", "_scs_indirect", + "_scs_dense", "_scs_mkl"}, + "aarch64": {"_scs_direct", "_scs_indirect", "_scs_dense"}, + "musllinux": {"_scs_direct", "_scs_indirect", "_scs_dense"}, } WANT_MKL_NEEDED = ["libiomp5.so", "libmkl_core.so.3", "libmkl_intel_lp64.so.3", "libmkl_intel_thread.so.3", "libmkl_rt.so.3"] nmkl = 0 - expected_nmkl = sum(1 for w in whls - if platform_of(w) == "manylinux_x86_64") + group_counts = {} + for w in whls: + group_counts[platform_of(w)] = group_counts.get(platform_of(w), 0) + 1 + for plat in EXPECTED_EXTS: + if group_counts.get(plat, 0) == 0: + bad.append(("(all)", "-", + f"no wheels downloaded for platform {plat}")) + expected_nmkl = group_counts.get("manylinux_x86_64", 0) + if expected_nmkl == 0: + bad.append(("(all)", "-", + "no manylinux x86_64 wheels: the MKL audit would " + "be vacuous")) for whl in whls: plat = platform_of(whl) with tempfile.TemporaryDirectory() as td: @@ -346,15 +357,19 @@ jobs: f"extension inventory {sorted(exts)} != " f"expected {sorted(EXPECTED_EXTS[plat])}")) # every wheel must vendor a BLAS for the default - # backends; MKL must never be vendored anywhere + # backends; Intel libraries must not ship anywhere in + # the wheel, whatever directory they landed in + all_names = [os.path.basename(so) for so in sos] vendored = [os.path.basename(so) for so in sos if ".libs" in so] if not any(v.startswith("libopenblas") for v in vendored): bad.append((os.path.basename(whl), "-", f"no vendored OpenBLAS: {vendored}")) - if any(v.startswith(("libmkl", "libiomp")) for v in vendored): + intel = [n for n in all_names + if n.startswith(("libmkl", "libiomp"))] + if intel: bad.append((os.path.basename(whl), "-", - f"MKL vendored into wheel: {vendored}")) + f"Intel libraries shipped in wheel: {intel}")) for so in sos: proc = subprocess.run(["readelf", "-d", so], capture_output=True, text=True) @@ -364,9 +379,14 @@ jobs: f"readelf failed: {proc.stderr.strip()}")) continue out = proc.stdout - for m in re.finditer( - r"\((?:RPATH|RUNPATH)\)[^\[]*\[([^\]]*)\]", out): - for entry in m.group(1).split(":"): + rpaths = re.findall(r"\(RPATH\)[^\[]*\[([^\]]*)\]", out) + runpaths = re.findall( + r"\(RUNPATH\)[^\[]*\[([^\]]*)\]", out) + if rpaths: + bad.append((os.path.basename(whl), rel, + f"legacy DT_RPATH present: {rpaths}")) + for rp in runpaths: + for entry in rp.split(":"): if entry and not entry.startswith("$ORIGIN"): bad.append((os.path.basename(whl), rel, f"non-relative rpath entry " @@ -380,12 +400,10 @@ jobs: if mkl != WANT_MKL_NEEDED: bad.append((os.path.basename(whl), rel, f"NEEDED drift: {' '.join(mkl)}")) - rp = re.findall( - r"\((?:RPATH|RUNPATH)\)[^\[]*\[([^\]]*)\]", out) - if rp != ["$ORIGIN/../../../../lib"]: + if runpaths != ["$ORIGIN/../../../../lib"]: bad.append((os.path.basename(whl), rel, - f"_scs_mkl RUNPATH {rp} != " - f"['$ORIGIN/../../../../lib']")) + f"_scs_mkl DT_RUNPATH {runpaths} " + f"!= ['$ORIGIN/../../../../lib']")) if nmkl != expected_nmkl: bad.append(("(all)", "-", f"found {nmkl} _scs_mkl extensions, expected " From af9e0912d007d5d0ff6c9eca3fccffaf90773219 Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 22:15:09 +0100 Subject: [PATCH 12/19] Audit: scope the DT_RPATH prohibition to _scs_mkl auditwheel deliberately writes DT_RPATH on every object it grafts (RPATH outranks LD_LIBRARY_PATH, isolating vendored libraries), so the blanket prohibition failed all 27 correct wheels. The general rule is $ORIGIN-relative entries on either tag type; _scs_mkl -- the object this repo patches itself -- keeps the strict shape of exactly one DT_RUNPATH and no DT_RPATH. --- .github/workflows/build.yml | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 2fefd9fa..4ed9e212 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -382,10 +382,13 @@ jobs: rpaths = re.findall(r"\(RPATH\)[^\[]*\[([^\]]*)\]", out) runpaths = re.findall( r"\(RUNPATH\)[^\[]*\[([^\]]*)\]", out) - if rpaths: - bad.append((os.path.basename(whl), rel, - f"legacy DT_RPATH present: {rpaths}")) - for rp in runpaths: + # auditwheel deliberately writes DT_RPATH on grafted + # objects (RPATH outranks LD_LIBRARY_PATH, isolating + # the vendored libraries); both tag types are fine + # in general as long as every entry is + # $ORIGIN-relative. _scs_mkl, which we patch + # ourselves, is held to the stricter shape below. + for rp in rpaths + runpaths: for entry in rp.split(":"): if entry and not entry.startswith("$ORIGIN"): bad.append((os.path.basename(whl), rel, @@ -400,6 +403,10 @@ jobs: if mkl != WANT_MKL_NEEDED: bad.append((os.path.basename(whl), rel, f"NEEDED drift: {' '.join(mkl)}")) + if rpaths: + bad.append((os.path.basename(whl), rel, + f"_scs_mkl must not carry legacy " + f"DT_RPATH: {rpaths}")) if runpaths != ["$ORIGIN/../../../../lib"]: bad.append((os.path.basename(whl), rel, f"_scs_mkl DT_RUNPATH {runpaths} " From 95b9a816dedece51c03d4fa49b1a244c212ba1fc Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Mon, 31 Aug 2026 23:18:26 +0100 Subject: [PATCH 13/19] Audit: require the exact python/ABI tag set per platform Deriving expected counts from downloaded wheels let one surviving wheel per platform stand in for the full set. Each platform group must now carry exactly {cp39..cp315, cp314t, cp315t}; the ABI tag (field 4) distinguishes free-threaded builds. --- .github/workflows/build.yml | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 4ed9e212..e3988a95 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -325,19 +325,23 @@ jobs: WANT_MKL_NEEDED = ["libiomp5.so", "libmkl_core.so.3", "libmkl_intel_lp64.so.3", "libmkl_intel_thread.so.3", "libmkl_rt.so.3"] + # Exact per-platform python/ABI tag coverage: deriving the + # expected count from whatever happened to download would let a + # single surviving wheel per platform pass while two dozen were + # missing. Update this set when python versions are added. + EXPECTED_TAGS = {"cp39", "cp310", "cp311", "cp312", "cp313", + "cp314", "cp314t", "cp315", "cp315t"} nmkl = 0 - group_counts = {} + group_tags = {plat: set() for plat in EXPECTED_EXTS} for w in whls: - group_counts[platform_of(w)] = group_counts.get(platform_of(w), 0) + 1 - for plat in EXPECTED_EXTS: - if group_counts.get(plat, 0) == 0: + tag = os.path.basename(w).split("-")[3] + group_tags[platform_of(w)].add(tag) + for plat, tags in group_tags.items(): + if tags != EXPECTED_TAGS: bad.append(("(all)", "-", - f"no wheels downloaded for platform {plat}")) - expected_nmkl = group_counts.get("manylinux_x86_64", 0) - if expected_nmkl == 0: - bad.append(("(all)", "-", - "no manylinux x86_64 wheels: the MKL audit would " - "be vacuous")) + f"{plat} tag set {sorted(tags)} != expected " + f"{sorted(EXPECTED_TAGS)}")) + expected_nmkl = len(EXPECTED_TAGS) for whl in whls: plat = platform_of(whl) with tempfile.TemporaryDirectory() as td: From bf89d697ba6fc32f9c843cc14d54863c2a491bdb Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Thu, 3 Sep 2026 20:09:32 +0100 Subject: [PATCH 14/19] Link MKL statically into _scs_mkl: self-contained wheels, no runtime extra The 3.3.0 wheels vendored dynamic MKL and shipped it incomplete: its CPU dispatch kernels are loaded via dlopen, invisible to auditwheel, so every solve aborted on machines without MKL on the loader path (cvxgrp/scs#423). Unvendoring (the previous shape of this PR) fixed that at the cost of a runtime extra, a RUNPATH patch, a dlopen shim and a 276 MB download. Linking Intel's mkl-static archives into _scs_mkl instead makes the wheel self-contained: static MKL dispatches CPU variants internally, so there is nothing to lose; the LP64 interface is fixed at link time, so there is nothing to negotiate (the core's MKL_Set_Interface_Layer reference is now weak); every MKL symbol is hidden, so a process-wide MKL is never interposed; and the sequential threading layer ships no OpenMP runtime that could abort a process already holding one. Measured on the scratch branch: 108 MB on disk, 30 MB compressed, the full core test suite passing in a clean environment (58/58 wheel-default, 66/66 with spectral cones), versus 52 MB for the broken 3.3.0 wheel. - meson: mkl_static_prefix replaces mkl_backend; _scs_mkl links the intel_lp64/sequential/core archives as a group with --exclude-libs,ALL and installs Intel's license notice alongside (ISSL requires it). - cibuildwheel: install_mkl_static.sh lays the archives out under /opt/mkl-static from Intel's PyPI wheels; default auditwheel repair; nothing in the environment points the loader at build-machine paths. - Deleted: the scs[mkl] extra, add_mkl_rpath.sh, the dlopen shim in scs/py/__init__.py, and the ldd linkage test. - wheel_audit keeps the invariants that matter for every Linux wheel: no dynamic MKL dependency in any extension, no Intel shared library in the wheel, OpenBLAS vendored, no absolute rpath entry. - wheel_smoke_clean_env, the release gate: one variant, pristine container, LD_LIBRARY_PATH asserted unset, every backend solved (qdldl, cpu_indirect, dense, mkl, auto) on a problem with a PSD cone; on x86-64 the static MKL backend must import and be AUTO's choice. - scs_source pinned to the core commit carrying the weak reference. Co-Authored-By: Claude Fable 5.1 --- .github/scripts/add_mkl_rpath.sh | 51 ------ .github/scripts/install_mkl_static.sh | 23 +++ .github/workflows/build.yml | 216 +++++------------------- LICENSE-INTEL-MKL.txt | 25 +++ README.md | 45 ++--- meson.build | 59 +++++-- meson.options | 4 +- pyproject.toml | 47 ++---- scs/py/__init__.py | 83 +-------- scs_source | 2 +- test/test_solve_random_cone_prob_mkl.py | 21 +-- 11 files changed, 169 insertions(+), 407 deletions(-) delete mode 100755 .github/scripts/add_mkl_rpath.sh create mode 100755 .github/scripts/install_mkl_static.sh create mode 100644 LICENSE-INTEL-MKL.txt diff --git a/.github/scripts/add_mkl_rpath.sh b/.github/scripts/add_mkl_rpath.sh deleted file mode 100755 index 60f2c9b5..00000000 --- a/.github/scripts/add_mkl_rpath.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash -# Normalize the RUNPATH of the repaired wheel's _scs_mkl extension. -# -# The wheel does not vendor MKL (its dlopen'd CPU dispatch kernels are -# invisible to auditwheel; cvxgrp/scs#423). The scs[mkl] extra installs -# Intel's official wheels into /lib, and site-packages/scs sits -# exactly four levels below the prefix in every standard layout (venv, -# conda, user site, system), so $ORIGIN/../../../../lib lets the dynamic -# loader resolve the whole MKL component group in one dlopen -- with the -# correct mutual binding MKL's libraries require, and without widening -# any symbol scope (NumPy's vendored OpenBLAS is unaffected). -# -# --set-rpath (not --add-rpath): the meson build bakes the container's -# /opt/intel/... paths into the extension, and auditwheel preserves them. -# Shipping those would let a build-machine-layout MKL installation win -# over the scs[mkl] runtime, and would let container tests silently -# resolve the build MKL. The RUNPATH must be exactly the one relative -# entry, and the MKL-related NEEDED set must be exactly what the loader -# shim and the mkl PyPI pin (mkl>=2026,<2027 -- the .so.3 ABI) provide; -# both are asserted here so a drift fails the build, not a user. -set -euo pipefail -dest_dir="$1" -whl=$(ls -t "$dest_dir"/scs-*.whl | head -1) -tmp=$(mktemp -d) -python -m pip install --quiet wheel -python -m wheel unpack --dest "$tmp" "$whl" -unpacked=$(ls -d "$tmp"/scs-*) -want_rpath='$ORIGIN/../../../../lib' -want_needed='libiomp5.so libmkl_core.so.3 libmkl_intel_lp64.so.3 libmkl_intel_thread.so.3 libmkl_rt.so.3' -patched=0 -for so in "$unpacked"/scs/_scs_mkl*.so; do - [ -e "$so" ] || continue - patchelf --set-rpath "$want_rpath" "$so" - got_rpath=$(patchelf --print-rpath "$so") - if [ "$got_rpath" != "$want_rpath" ]; then - echo "add_mkl_rpath: unexpected RUNPATH '$got_rpath' on $(basename "$so")" >&2 - exit 1 - fi - got_needed=$(patchelf --print-needed "$so" | grep -E '^lib(mkl|iomp)' | sort | tr '\n' ' ' | sed 's/ $//') - if [ "$got_needed" != "$want_needed" ]; then - echo "add_mkl_rpath: NEEDED drift on $(basename "$so")" >&2 - echo " want: $want_needed" >&2 - echo " got: $got_needed" >&2 - exit 1 - fi - echo "add_mkl_rpath: $(basename "$so") rpath='$got_rpath' needed ok" - patched=1 -done -[ "$patched" -eq 1 ] || { echo "add_mkl_rpath: no _scs_mkl extension found" >&2; exit 1; } -python -m wheel pack --dest-dir "$dest_dir" "$unpacked" -rm -rf "$tmp" diff --git a/.github/scripts/install_mkl_static.sh b/.github/scripts/install_mkl_static.sh new file mode 100755 index 00000000..42d1b294 --- /dev/null +++ b/.github/scripts/install_mkl_static.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# Lay out Intel's static oneMKL archives under a prefix for the static +# _scs_mkl link (see mkl_static_prefix in meson.options). The archives come +# from Intel's own mkl-static PyPI wheel; pip download with an explicit +# platform tag so the build container's python and glibc are irrelevant. +set -euo pipefail +prefix=${1:?usage: install_mkl_static.sh } +ver=${MKL_STATIC_VERSION:-2026.1.0} +py=$(ls -d /opt/python/cp3*-cp3*/bin/python 2>/dev/null | head -1 || command -v python3) +tmp=$(mktemp -d) +"$py" -m pip download --quiet --no-deps --only-binary=:all: \ + --platform manylinux_2_28_x86_64 -d "$tmp" "mkl-static==$ver" "mkl-include==$ver" +for whl in "$tmp"/*.whl; do + "$py" -m zipfile -e "$whl" "$tmp/unpacked" +done +mkdir -p "$prefix" +# wheel data files live under .data/data/{lib,include} +cp -r "$tmp"/unpacked/*.data/data/. "$prefix"/ +for a in libmkl_intel_lp64.a libmkl_sequential.a libmkl_core.a; do + test -f "$prefix/lib/$a" || { echo "install_mkl_static: missing $prefix/lib/$a" >&2; exit 1; } +done +rm -rf "$tmp" +echo "install_mkl_static: oneMKL $ver archives under $prefix" diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index e3988a95..9078e6a3 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -280,12 +280,10 @@ jobs: path: ./wheelhouse/*.whl wheel_audit: - # Static gate for #423-class defects across EVERY Linux wheel (all - # pythons, x86_64 + aarch64 + musllinux): no .so may carry an absolute - # RPATH/RUNPATH entry (a build-container path in a shipped wheel lets a - # matching system MKL shadow the scs[mkl] runtime), and _scs_mkl's - # MKL-related NEEDED set must be exactly what the mkl>=2026,<2027 pin - # provides (.so.3 ABI). + # Static gate for #423-class defects across every Linux wheel: no wheel + # may depend on a dynamic MKL (the backend is linked statically), ship + # an Intel shared library, or carry a build-machine path in an rpath + # (which let the 3.3.0 container tests pass while user machines failed). name: Audit wheel ELF metadata needs: build_wheels runs-on: ubuntu-latest @@ -296,130 +294,40 @@ jobs: path: wheelhouse merge-multiple: true - - name: Audit RPATH and NEEDED across all Linux wheels + - name: Audit NEEDED, rpath and shipped libraries run: | cat > audit.py <<'EOF' import glob, os, re, subprocess, sys, tempfile, zipfile bad = [] whls = sorted(glob.glob("wheelhouse/*.whl")) assert whls, "no wheels downloaded" - - def platform_of(whl): - name = os.path.basename(whl) - if "musllinux" in name: - return "musllinux" - if "aarch64" in name: - return "aarch64" - assert "manylinux" in name and "x86_64" in name, name - return "manylinux_x86_64" - - # Exact per-platform extension inventory: a missing backend is as - # much a packaging defect as a broken one (a vacuously green audit - # hid exactly this class of failure). - EXPECTED_EXTS = { - "manylinux_x86_64": {"_scs_direct", "_scs_indirect", - "_scs_dense", "_scs_mkl"}, - "aarch64": {"_scs_direct", "_scs_indirect", "_scs_dense"}, - "musllinux": {"_scs_direct", "_scs_indirect", "_scs_dense"}, - } - WANT_MKL_NEEDED = ["libiomp5.so", "libmkl_core.so.3", - "libmkl_intel_lp64.so.3", - "libmkl_intel_thread.so.3", "libmkl_rt.so.3"] - # Exact per-platform python/ABI tag coverage: deriving the - # expected count from whatever happened to download would let a - # single surviving wheel per platform pass while two dozen were - # missing. Update this set when python versions are added. - EXPECTED_TAGS = {"cp39", "cp310", "cp311", "cp312", "cp313", - "cp314", "cp314t", "cp315", "cp315t"} - nmkl = 0 - group_tags = {plat: set() for plat in EXPECTED_EXTS} - for w in whls: - tag = os.path.basename(w).split("-")[3] - group_tags[platform_of(w)].add(tag) - for plat, tags in group_tags.items(): - if tags != EXPECTED_TAGS: - bad.append(("(all)", "-", - f"{plat} tag set {sorted(tags)} != expected " - f"{sorted(EXPECTED_TAGS)}")) - expected_nmkl = len(EXPECTED_TAGS) for whl in whls: - plat = platform_of(whl) + name = os.path.basename(whl) with tempfile.TemporaryDirectory() as td: with zipfile.ZipFile(whl) as z: z.extractall(td) - sos = glob.glob(os.path.join(td, "**", "*.so*"), - recursive=True) - if not sos: - bad.append((os.path.basename(whl), "-", - "wheel contains no shared objects")) - continue - exts = {m.group(1) for so in sos - for m in [re.match(r"(_scs_[a-z]+)\.", - os.path.basename(so))] if m} - if exts != EXPECTED_EXTS[plat]: - bad.append((os.path.basename(whl), "-", - f"extension inventory {sorted(exts)} != " - f"expected {sorted(EXPECTED_EXTS[plat])}")) - # every wheel must vendor a BLAS for the default - # backends; Intel libraries must not ship anywhere in - # the wheel, whatever directory they landed in - all_names = [os.path.basename(so) for so in sos] - vendored = [os.path.basename(so) for so in sos - if ".libs" in so] - if not any(v.startswith("libopenblas") for v in vendored): - bad.append((os.path.basename(whl), "-", - f"no vendored OpenBLAS: {vendored}")) - intel = [n for n in all_names - if n.startswith(("libmkl", "libiomp"))] + sos = glob.glob(os.path.join(td, "**", "*.so*"), recursive=True) + names = [os.path.basename(so) for so in sos] + if not any(n.startswith("libopenblas") for n in names): + bad.append((name, "-", f"no vendored OpenBLAS: {names}")) + intel = [n for n in names if n.startswith(("libmkl", "libiomp"))] if intel: - bad.append((os.path.basename(whl), "-", - f"Intel libraries shipped in wheel: {intel}")) + bad.append((name, "-", f"Intel shared libraries in wheel: {intel}")) for so in sos: - proc = subprocess.run(["readelf", "-d", so], - capture_output=True, text=True) rel = os.path.relpath(so, td) + proc = subprocess.run(["readelf", "-d", so], capture_output=True, text=True) if proc.returncode != 0: - bad.append((os.path.basename(whl), rel, - f"readelf failed: {proc.stderr.strip()}")) + bad.append((name, rel, f"readelf failed: {proc.stderr.strip()}")) continue - out = proc.stdout - rpaths = re.findall(r"\(RPATH\)[^\[]*\[([^\]]*)\]", out) - runpaths = re.findall( - r"\(RUNPATH\)[^\[]*\[([^\]]*)\]", out) - # auditwheel deliberately writes DT_RPATH on grafted - # objects (RPATH outranks LD_LIBRARY_PATH, isolating - # the vendored libraries); both tag types are fine - # in general as long as every entry is - # $ORIGIN-relative. _scs_mkl, which we patch - # ourselves, is held to the stricter shape below. - for rp in rpaths + runpaths: + needed = re.findall(r"\(NEEDED\)[^\[]*\[([^\]]+)\]", proc.stdout) + mkl = [n for n in needed if n.startswith(("libmkl", "libiomp"))] + if mkl: + bad.append((name, rel, f"dynamic MKL dependency: {mkl}")) + for rp in re.findall(r"\((?:RPATH|RUNPATH)\)[^\[]*\[([^\]]*)\]", proc.stdout): for entry in rp.split(":"): if entry and not entry.startswith("$ORIGIN"): - bad.append((os.path.basename(whl), rel, - f"non-relative rpath entry " - f"'{entry}'")) - if "_scs_mkl" in os.path.basename(so): - nmkl += 1 - needed = re.findall( - r"\(NEEDED\)[^\[]*\[([^\]]+)\]", out) - mkl = sorted(n for n in needed - if n.startswith(("libmkl", "libiomp"))) - if mkl != WANT_MKL_NEEDED: - bad.append((os.path.basename(whl), rel, - f"NEEDED drift: {' '.join(mkl)}")) - if rpaths: - bad.append((os.path.basename(whl), rel, - f"_scs_mkl must not carry legacy " - f"DT_RPATH: {rpaths}")) - if runpaths != ["$ORIGIN/../../../../lib"]: - bad.append((os.path.basename(whl), rel, - f"_scs_mkl DT_RUNPATH {runpaths} " - f"!= ['$ORIGIN/../../../../lib']")) - if nmkl != expected_nmkl: - bad.append(("(all)", "-", - f"found {nmkl} _scs_mkl extensions, expected " - f"{expected_nmkl}")) - print(f"audited {len(whls)} wheels, {nmkl} _scs_mkl extensions") + bad.append((name, rel, f"non-relative rpath entry '{entry}'")) + print(f"audited {len(whls)} wheels") for whl, so, msg in bad: print(f"BAD: {whl} :: {so} :: {msg}") sys.exit(1 if bad else 0) @@ -427,29 +335,16 @@ jobs: python3 audit.py wheel_smoke_clean_env: - # Regression gate for #423: install the built manylinux x86_64 wheel in a - # pristine container (no BLAS, no LD_LIBRARY_PATH, no site packages) and - # run a real solve on every backend. The 3.3.0 wheels passed - # cibuildwheel's in-container tests because the build container's - # environment satisfied MKL's dlopen'd dispatch kernels; only a clean - # environment exercises what the wheel actually ships. - # - # Variants: - # default -- plain `pip install scs`: AUTO must resolve to QDLDL and the - # unvendored _scs_mkl must fail with a clean ImportError, never crash. - # mkl -- `pip install scs[mkl]`: MKL comes from Intel's official wheels; - # AUTO must resolve to MKL and the mkl backend must solve. This also - # enforces that the oneMKL generation the wheel links matches the - # pinned `mkl` PyPI dependency range. - # Pythons: oldest (3.9), newest stable (3.14), free-threaded (3.14t via - # uv, python-build-standalone). Remaining ABIs are covered by wheel_audit. - name: Clean-container wheel smoke test (${{ matrix.variant }}, ${{ matrix.python }}) + # The release gate for #423-class failures: install the built wheel in a + # pristine container with nothing on the loader path and solve with every + # backend, including a PSD cone so BLAS/LAPACK run. On x86-64 the static + # MKL backend must import and be AUTO's choice. + name: Clean-container wheel smoke test (${{ matrix.python }}) needs: build_wheels runs-on: ubuntu-latest strategy: fail-fast: false matrix: - variant: [default, mkl] python: ["3.9", "3.14", "3.14t"] steps: - uses: actions/download-artifact@v7 @@ -458,15 +353,14 @@ jobs: path: wheelhouse merge-multiple: true - - name: Solve in pristine container (${{ matrix.variant }}, ${{ matrix.python }}) + - name: Solve in pristine container (${{ matrix.python }}) run: | cat > smoke.py <<'EOF' - import importlib, os, sys, traceback + import os, sys import numpy as np, scipy.sparse as sp, scs - mode = sys.argv[1] - print("scs", scs.__version__, "mode", mode, "python", sys.version) - # LP block plus a small PSD cone: the PSD projection goes through - # the wheel's BLAS/LAPACK, the exact path #423 failed to ship. + # the loader must be exactly as clean as a user's machine + assert not os.environ.get("LD_LIBRARY_PATH"), os.environ["LD_LIBRARY_PATH"] + print("scs", scs.__version__, "python", sys.version) data = { "A": sp.csc_matrix(np.vstack([np.array([[1.0, 1.0], [1.0, -1.0], [0.0, 1.0]]), np.array([[-1.0, 0.0], [0.0, 0.0], [0.0, -1.0]])])), @@ -474,49 +368,23 @@ jobs: "c": np.array([1.0, 1.0]), } cone = {"z": 1, "l": 2, "s": [2]} - - # Exercise the scs[mkl] loader path explicitly, with diagnostics. - mkl_err = None - try: - scs._load_module("_scs_mkl") - except ImportError as e: - mkl_err = e - if mode == "mkl": - traceback.print_exc() - libdir = os.path.join(sys.prefix, "lib") - libs = sorted(f for f in os.listdir(libdir) - if f.startswith(("libmkl", "libiomp"))) if os.path.isdir(libdir) else [] - print(libdir, "->", libs[:8], "..." if len(libs) > 8 else "") - if mode == "mkl": - assert mkl_err is None, f"scs[mkl] installed but _scs_mkl failed: {mkl_err}" - else: - # The wheel ships _scs_mkl but its MKL comes from scs[mkl]; - # without the extra it must fail cleanly, never crash. - assert mkl_err is not None, "unvendored _scs_mkl imported without scs[mkl]" - print("mkl backend unavailable without scs[mkl], as expected") - - # The extensions share the PyModuleDef name "_scs" (single-phase - # init), so identify the AUTO backend by file, not __name__. - auto_file = os.path.basename(getattr(scs._resolve_auto(), "__file__", "")) + scs._load_module("_scs_mkl") # static MKL: must import with nothing installed + auto_file = os.path.basename(scs._resolve_auto().__file__) + assert auto_file.startswith("_scs_mkl"), auto_file print("auto resolves to", auto_file) - want = "_scs_mkl" if mode == "mkl" else "_scs_direct" - assert auto_file.startswith(want), (auto_file, want) - - solvers = ["qdldl", "cpu_indirect"] + (["mkl"] if mode == "mkl" else []) - for name in solvers + ["auto"]: + for name in ["qdldl", "cpu_indirect", "dense", "mkl", "auto"]: kw = {} if name == "auto" else {"linear_solver": name} sol = scs.SCS(data, cone, verbose=True, **kw).solve() assert sol["info"]["status"] == "solved", (name, sol["info"]["status"]) print(name, "ok:", sol["info"].get("lin_sys_solver", "n/a")) - print("ALL OK:", mode) + print("ALL OK") EOF case "${{ matrix.python }}" in 3.9) IMAGE=python:3.9-slim; TAG=cp39-cp39 ;; 3.14) IMAGE=python:3.14-slim; TAG=cp314-cp314 ;; 3.14t) IMAGE=ghcr.io/astral-sh/uv:debian-slim; TAG=cp314-cp314t ;; esac - docker run --rm -v "$PWD:/work" -w /tmp \ - -e VARIANT=${{ matrix.variant }} -e PY=${{ matrix.python }} -e TAG=$TAG \ + docker run --rm -v "$PWD:/work" -w /tmp -e PY=${{ matrix.python }} -e TAG=$TAG \ "$IMAGE" bash -ec ' if command -v uv >/dev/null; then uv python install "$PY" @@ -528,14 +396,8 @@ jobs: PYBIN=python fi $PIP numpy scipy - WHL=$(ls /work/wheelhouse/scs-*-${TAG}-manylinux*x86_64.whl) - if [ "$VARIANT" = mkl ]; then - $PIP "${WHL}[mkl]" - else - $PIP "$WHL" - fi - $PYBIN -m pip list 2>/dev/null | grep -i -E "scs|mkl|openmp" || true - $PYBIN /work/smoke.py "$VARIANT"' + $PIP /work/wheelhouse/scs-*-${TAG}-manylinux*x86_64.whl + $PYBIN /work/smoke.py' build_sdist: name: Build source distribution diff --git a/LICENSE-INTEL-MKL.txt b/LICENSE-INTEL-MKL.txt new file mode 100644 index 00000000..0783c051 --- /dev/null +++ b/LICENSE-INTEL-MKL.txt @@ -0,0 +1,25 @@ +Intel Simplified Software License (Version October 2022) + +Use and Redistribution. You may use and redistribute the software, which is provided in binary form only, (the “Software”), without modification, provided the following conditions are met: + +* Redistributions must reproduce the above copyright notice and these terms of use in the Software and in the documentation and/or other materials provided with the distribution. +* Neither the name of Intel nor the names of its suppliers may be used to endorse or promote products derived from this Software without specific prior written permission. +* No reverse engineering, decompilation, or disassembly of the Software is permitted, nor any modification or alteration of the Software or its operation at any time, including during execution. + +No other licenses. Except as provided in the preceding section, Intel grants no licenses or other rights by implication, estoppel or otherwise to, patent, copyright, trademark, trade name, service mark or other intellectual property licenses or rights of Intel. + +Third party software. “Third Party Software” means the files (if any) listed in the “third-party-software.txt” or other similarly-named text file that may be included with the Software. Third Party Software, even if included with the distribution of the Software, may be governed by separate license terms, including without limitation, third party license terms, open source software notices and terms, and/or other Intel software license terms. These separate license terms solely govern Your use of the Third Party Software. + +DISCLAIMER. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT ARE DISCLAIMED. THIS SOFTWARE IS NOT INTENDED FOR USE IN SYSTEMS OR APPLICATIONS WHERE FAILURE OF THE SOFTWARE MAY CAUSE PERSONAL INJURY OR DEATH AND YOU AGREE THAT YOU ARE FULLY RESPONSIBLE FOR ANY CLAIMS, COSTS, DAMAGES, EXPENSES, AND ATTORNEYS’ FEES ARISING OUT OF ANY SUCH USE, EVEN IF ANY CLAIM ALLEGES THAT INTEL WAS NEGLIGENT REGARDING THE DESIGN OR MANUFACTURE OF THE SOFTWARE. + +LIMITATION OF LIABILITY. IN NO EVENT WILL INTEL BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +No support. Intel may make changes to the Software, at any time without notice, and is not obligated to support, update or provide training for the Software. + +Termination. Your right to use the Software is terminated in the event of your breach of this license. + +Feedback. Should you provide Intel with comments, modifications, corrections, enhancements or other input (“Feedback”) related to the Software, Intel will be free to use, disclose, reproduce, license or otherwise distribute or exploit the Feedback in its sole discretion without any obligations or restrictions of any kind, including without limitation, intellectual property rights or licensing obligations. + +Compliance with laws. You agree to comply with all relevant laws and regulations governing your use, transfer, import or export (or prohibition thereof) of the Software. + +Governing law. All disputes will be governed by the laws of the United States of America and the State of Delaware without reference to conflict of law principles and subject to the exclusive jurisdiction of the state or federal courts sitting in the State of Delaware, and each party agrees that it submits to the personal jurisdiction and venue of those courts and waives any objections. THE UNITED NATIONS CONVENTION ON CONTRACTS FOR THE INTERNATIONAL SALE OF GOODS (1980) IS SPECIFICALLY EXCLUDED AND WILL NOT APPLY TO THE SOFTWARE. diff --git a/README.md b/README.md index 74edb3ae..c05a3702 100644 --- a/README.md +++ b/README.md @@ -15,26 +15,14 @@ The full documentation is available [here](https://www.cvxgrp.org/scs/). pip install scs ``` -> [!IMPORTANT] -> **On x86-64 Linux, install the MKL extra instead — recommended for nearly -> everyone:** -> -> ```bash -> pip install "scs[mkl]" -> ``` -> -> This enables the MKL Pardiso direct linear solver, which is faster than the -> built-in solver for most problems — often dramatically so on larger ones — -> and SCS uses it automatically when it is installed; no code or settings -> changes are needed. The MKL runtime comes from Intel's official `mkl` wheels -> (roughly a 300 MB download, about 1 GB on disk). A plain `pip install scs` -> works everywhere and falls back to the built-in QDLDL solver. -> -> The extra is supported on the prebuilt manylinux x86-64 wheels (glibc -> 2.28+) in standard prefix layouts (venv, conda, system, user site). It does -> not work on musllinux/Alpine (Intel publishes no musl wheels), with -> `pip install --target`, or with source/sdist builds (build against your own -> MKL with `-Dlink_mkl=true` instead). +On x86-64 Linux the wheels include the MKL Pardiso direct linear solver, +linked statically into the `_scs_mkl` extension, and use it automatically: +it is faster than the built-in QDLDL solver for most problems, often +dramatically so on larger ones, and nothing extra needs to be installed. The +MKL backend is single-threaded (a bundled OpenMP runtime can abort a process +that already has one, and SCS's per-iteration work does not parallelize +well); Intel's license notice ships in the wheel as `LICENSE-INTEL-MKL.txt`. +Every other wheel falls back to QDLDL. To install from source: ```bash @@ -62,14 +50,13 @@ Available values: `AUTO`, `QDLDL`, `CPU_INDIRECT`, `MKL`, `ACCELERATE`, `CPU_DENSE`, `GPU_INDIRECT`, `CUDSS`. The pre-built wheels (`pip install scs`) link OpenBLAS on Linux and Windows, -and Apple Accelerate on macOS. On x86-64 Linux the MKL Pardiso backend is -available via `pip install "scs[mkl]"` (see [Installation](#installation) — -recommended): Intel's official `mkl` wheels provide the complete MKL runtime, -and `AUTO` selects MKL whenever it is importable. SCS wheels deliberately do -not vendor MKL — its CPU dispatch kernels are loaded via `dlopen`, invisible -to wheel-repair tools, and an incompletely vendored MKL aborts the process at -solve time (cvxgrp/scs#423). The MKL backend is also available in source -builds (e.g. conda environments providing MKL). When installing from source, +and Apple Accelerate on macOS. The x86-64 Linux wheels also ship the MKL +Pardiso backend with MKL linked statically into `_scs_mkl`, and `AUTO` +selects it; MKL is deliberately not vendored as shared libraries, whose +dlopen'd CPU dispatch kernels are invisible to wheel-repair tools and whose +absence aborts the process at solve time (cvxgrp/scs#423). The MKL backend +is also available in source builds (e.g. conda environments providing MKL). +When installing from source, additional backends can be enabled with build-time flags: ```bash @@ -90,7 +77,7 @@ pip install . -Csetup-args=-Duse_spectral_cones=true ``` Notes: -- Linux x86_64 wheels ship a `_scs_mkl` extension linked against threaded MKL (CI asserts its exact MKL/`libiomp5` linkage); the runtime comes from `scs[mkl]`. Windows wheels do not include the MKL backend; Windows source builds use sequential MKL because Intel's conda `pkg-config` metadata for the threaded variant is still broken. +- Linux x86_64 wheels ship a `_scs_mkl` extension with sequential MKL linked statically (CI asserts that no wheel carries a dynamic MKL dependency). Windows wheels do not include the MKL backend; Windows source builds use sequential MKL because Intel's conda `pkg-config` metadata for the threaded variant is still broken. - `BLAS64` is a general SCS build mode for ILP64 BLAS/LAPACK libraries, not an MKL-only feature. - For the MKL Pardiso backend specifically, `BLAS64` must be paired with 64-bit SCS integers (`DLONG` / `int32=false`), and SCS now fails early if another library in the process has already fixed MKL to an incompatible LP64/ILP64 interface layer. diff --git a/meson.build b/meson.build index 270e482e..04d1d327 100644 --- a/meson.build +++ b/meson.build @@ -22,24 +22,24 @@ print(incdir) # Get BLAS # -# link_mkl links every extension against MKL (conda/source builds, where the -# environment provides a complete MKL). mkl_backend builds only the _scs_mkl -# extension against MKL while every other extension uses the platform BLAS: -# this is the wheel configuration -- MKL is not vendored into wheels because -# its CPU dispatch kernels are loaded via dlopen, invisible to auditwheel -# (cvxgrp/scs#423); instead the scs[mkl] extra supplies Intel's official -# `mkl` wheels at runtime. +# link_mkl links every extension against a dynamic MKL from the environment +# (conda/source builds). mkl_static_prefix builds only _scs_mkl, with MKL +# linked statically and its symbols hidden, while every other extension uses +# the platform BLAS: this is the x86-64 Linux wheel configuration. Static +# linking is what makes the wheel self-contained -- dynamic MKL loads its CPU +# dispatch kernels via dlopen, invisible to wheel-repair tools, and the 3.3.0 +# wheels shipped without them (cvxgrp/scs#423). blas_deps = [] mkl_blas_deps = [] mkl_pkg_name = '' -want_mkl = get_option('link_mkl') or get_option('mkl_backend') -if want_mkl +mkl_static_prefix = get_option('mkl_static_prefix') +if get_option('link_mkl') and mkl_static_prefix != '' + error('link_mkl and mkl_static_prefix are mutually exclusive.') +endif +if get_option('link_mkl') if get_option('use_blas64') and get_option('int32') error('MKL BLAS64 requires 64-bit SCS integers. Re-run Meson with -Dint32=false.') endif - if get_option('mkl_backend') and not get_option('link_mkl') and get_option('use_blas64') - error('mkl_backend with use_blas64 is unsupported; use link_mkl for ILP64 MKL builds.') - endif # Link against MKL component libraries. The integer width must match: # use_blas64=false (default) -> LP64 (32-bit BLAS integers) # use_blas64=true -> ILP64 (64-bit BLAS integers) @@ -85,7 +85,7 @@ if want_mkl mkl_blas_deps = [dependency('mkl-sdl', required : false)] endif if not mkl_blas_deps[0].found() and not get_option('sdist_mode') - error('MKL was requested (link_mkl/mkl_backend) but was not found.') + error('link_mkl was requested but MKL was not found.') endif endif @@ -345,7 +345,7 @@ if get_option('use_gpu') ) endif -if want_mkl +if get_option('link_mkl') # The MKL backend calls MKL_Set_Interface_Layer() for a runtime interface # check. This symbol is only exported by mkl_rt (the single dynamic library), # not by component libraries (mkl_intel_lp64 etc.) that pkg-config may link. @@ -362,9 +362,37 @@ if want_mkl endif endif if not mkl_rt_dep.found() - error('link_mkl/mkl_backend requires mkl_rt (or mkl-sdl). _scs_mkl calls MKL_Set_Interface_Layer() at startup and cannot be built safely without it.') + error('link_mkl requires mkl_rt (or mkl-sdl). _scs_mkl calls MKL_Set_Interface_Layer() at startup and cannot be built safely without it.') endif _mkl_deps = [mkl_blas_deps] + _base_deps + [mkl_rt_dep] + _mkl_link_args = [] +elif mkl_static_prefix != '' + # Static MKL from Intel's mkl-static archives: LP64 interface, sequential + # threading (no OpenMP runtime to duplicate), linked as a group because the + # archives reference each other, with every MKL symbol hidden so an MKL + # already in the process (an MKL-backed NumPy, say) is never interposed. + # The core's MKL_Set_Interface_Layer reference is weak, so the mkl_rt-only + # runtime check is skipped automatically. + if get_option('use_blas64') + error('mkl_static_prefix links the LP64 interface; use link_mkl for ILP64 MKL builds.') + endif + _mkl_lib = mkl_static_prefix / 'lib' + _mkl_archives = [] + foreach _a : ['libmkl_intel_lp64.a', 'libmkl_sequential.a', 'libmkl_core.a'] + if not fs.exists(_mkl_lib / _a) + error('mkl_static_prefix: ' + (_mkl_lib / _a) + ' not found (pip install mkl-static).') + endif + _mkl_archives += _mkl_lib / _a + endforeach + _mkl_deps = _base_deps + [cc.find_library('dl', required : false), cc.find_library('m', required : false)] + if is_linux + _mkl_deps += cc.find_library('rt', required : true) + endif + _mkl_link_args = ['-Wl,--start-group'] + _mkl_archives + ['-Wl,--end-group', '-Wl,--exclude-libs,ALL'] + # Intel Simplified Software License: the notice ships with the binaries. + install_data('LICENSE-INTEL-MKL.txt', install_dir: scs_dir) +endif +if get_option('link_mkl') or mkl_static_prefix != '' py.extension_module( '_scs_mkl', 'scs/scspy.c', @@ -375,6 +403,7 @@ if want_mkl c_args: common_c_args + ['-DPY_MKL', '-DSCS_MKL=1'], include_directories: common_includes + ['scs_source/linsys/mkl/direct'], dependencies: _mkl_deps, + link_args: _mkl_link_args, install_dir: scs_dir, install: true, ) diff --git a/meson.options b/meson.options index 7a086b20..07cde3a0 100644 --- a/meson.options +++ b/meson.options @@ -29,5 +29,5 @@ option('native_arch', type: 'boolean', value: false, description: 'Compile with -march=native for the current CPU. Improves performance but produces non-portable binaries. Enable when building from source for local use.') option('use_spectral_cones', type: 'boolean', value: false, description: 'Build with spectral cone support (logdet, nuclear norm, ell1, sum-of-largest). Requires LAPACK.') -option('mkl_backend', type: 'boolean', - value: false, description: 'Build the _scs_mkl extension against MKL while other extensions use the platform BLAS (wheel configuration; MKL supplied at runtime by the scs[mkl] extra)') +option('mkl_static_prefix', type: 'string', value: '', + description: 'Prefix holding Intel MKL static archives (/lib/libmkl_*.a, e.g. from the mkl-static PyPI wheel). Builds _scs_mkl with MKL linked statically and hidden while every other extension uses the platform BLAS: the x86-64 Linux wheel configuration.') diff --git a/pyproject.toml b/pyproject.toml index 1f53e87f..2e5d56bb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -38,19 +38,6 @@ dependencies = [ 'scipy', ] -[project.optional-dependencies] -# MKL Pardiso backend for the pre-built Linux x86-64 wheels. The wheels link -# _scs_mkl against MKL but do not vendor it (MKL's dlopen'd CPU dispatch -# kernels are invisible to wheel-repair tools; see cvxgrp/scs#423): Intel's -# official wheels provide the complete library set at runtime. The version -# range must match the oneMKL generation the wheels are built against -# (currently the .so.3 ABI, oneMKL 2026); the clean-container CI smoke test -# enforces the pairing. Without this extra, AUTO falls back to QDLDL. -mkl = [ - 'mkl >=2026, <2027 ; platform_system == "Linux" and platform_machine == "x86_64"', - 'intel-openmp >=2026, <2027 ; platform_system == "Linux" and platform_machine == "x86_64"', -] - [tool.cibuildwheel] skip = [ "*-win32", # fails on locating Python headers, probably meson.build is misconfigured @@ -79,32 +66,26 @@ repair-wheel-command = "delvewheel repair -w {dest_dir} {wheel}" # Openblas installation for 3 different linux images -# x86_64 Linux: default backends (_scs_direct/_scs_indirect) link vendored -# OpenBLAS; _scs_mkl links MKL *without vendoring it* (mkl_backend mode). -# The 3.3.0 wheels vendored MKL and shipped it incomplete -- its dlopen'd -# CPU dispatch kernels are invisible to auditwheel, and every solve aborted -# on machines without MKL on the loader path (#423). At runtime MKL now -# comes from Intel's official wheels via the scs[mkl] extra instead. +# x86_64 Linux: the default backends link vendored OpenBLAS; _scs_mkl links +# MKL *statically* (Intel's mkl-static archives, sequential, symbols hidden) +# so the wheel is self-contained. The 3.3.0 wheels vendored dynamic MKL and +# shipped it incomplete -- its dlopen'd CPU dispatch kernels are invisible to +# auditwheel -- so every solve aborted on machines without MKL on the loader +# path (#423). Static linking has no dispatch libraries to lose. # -# Two hard rules, both learned from #423: -# 1. No LD_LIBRARY_PATH in `environment`: the wheel tests must run exactly -# as clean as a user's machine. The oneAPI paths are scoped to the -# repair command only, where auditwheel needs to *find* the MKL libs it -# is told to exclude. -# 2. upload_pypi is gated on the wheel_smoke_clean_env job, which installs -# the wheel in a pristine container with and without scs[mkl]. +# Two rules, both learned from #423: nothing in `environment` may point the +# loader at build-machine libraries (the wheel tests must run exactly as +# clean as a user's machine), and upload_pypi is gated on the +# wheel_smoke_clean_env job, which installs the wheel in a pristine +# container and solves with every backend. [[tool.cibuildwheel.overrides]] select = "*-manylinux_x86_64" inherit.before-all = "append" before-all = [ - "dnf install -y openblas-devel dnf-plugins-core", - "dnf config-manager --add-repo https://yum.repos.intel.com/oneapi", - "rpm --import https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB", - "dnf install -y intel-oneapi-mkl-devel", + "dnf install -y openblas-devel", + "bash /project/.github/scripts/install_mkl_static.sh /opt/mkl-static", ] -environment = {PKG_CONFIG_PATH = "/opt/intel/oneapi/mkl/latest/lib/pkgconfig:/opt/intel/oneapi/compiler/latest/lib/pkgconfig:$PKG_CONFIG_PATH"} -config-settings = {setup-args = "-Dmkl_backend=true"} -repair-wheel-command = "LD_LIBRARY_PATH='/opt/intel/oneapi/mkl/latest/lib:/opt/intel/oneapi/mkl/latest/lib/intel64:/opt/intel/oneapi/compiler/latest/lib' auditwheel repair --exclude 'libmkl_*' --exclude 'libiomp5*' -w {dest_dir} {wheel} && bash /project/.github/scripts/add_mkl_rpath.sh {dest_dir}" +config-settings = {setup-args = "-Dmkl_static_prefix=/opt/mkl-static"} [[tool.cibuildwheel.overrides]] select = "*-manylinux_aarch64" diff --git a/scs/py/__init__.py b/scs/py/__init__.py index 15c96080..07a5e93e 100644 --- a/scs/py/__init__.py +++ b/scs/py/__init__.py @@ -37,90 +37,9 @@ class LinearSolver(enum.Enum): CUDSS = "cudss" -def _preload_intel_mkl(): - """Preload MKL from Intel's official ``mkl`` PyPI wheels (``scs[mkl]``). - - The pre-built Linux x86-64 wheels link the _scs_mkl extension against MKL - without vendoring it: MKL loads its CPU dispatch kernels via dlopen, which - wheel-repair tools cannot see, so a vendored MKL is incomplete and aborts - the process at solve time (cvxgrp/scs#423). The ``scs[mkl]`` extra instead - installs Intel's own ``mkl`` and ``intel-openmp`` wheels into - ``/lib``. - - The primary lookup mechanism is a ``$ORIGIN``-relative RUNPATH baked into - the wheel's extension (site-packages/scs is four levels below the prefix - in every standard layout), which lets the loader resolve MKL's mutually - referencing component libraries as one group. This fallback covers - non-standard layouts where that relative path misses: it dlopens the - libraries RTLD_LAZY | RTLD_LOCAL so the extension's DT_NEEDED entries - resolve from the link map. LAZY is required -- the components cannot be - eagerly bound one at a time -- and ctypes.CDLL always forces RTLD_NOW, - so this calls dlopen(3) directly. RTLD_LOCAL keeps MKL's BLAS from - interposing on other libraries (e.g. NumPy's vendored OpenBLAS). - Returns True if anything was loaded. - """ - if not sys.platform.startswith("linux"): - return False - import ctypes - import glob - import os - from importlib import metadata - - libdirs = [] - for pkg in ("mkl", "intel-openmp"): - try: - dist = metadata.distribution(pkg) - except metadata.PackageNotFoundError: - continue - for f in dist.files or (): - if f.name.startswith(("libmkl_", "libiomp5")): - d = os.path.dirname(os.fspath(dist.locate_file(f))) - if d not in libdirs and os.path.isdir(d): - libdirs.append(d) - if not libdirs: - # Fallback for installers that do not record RECORD data files. - for prefix in dict.fromkeys((sys.prefix, sys.base_prefix, sys.exec_prefix)): - d = os.path.join(prefix, "lib") - if glob.glob(os.path.join(d, "libmkl_core.so*")): - libdirs.append(d) - if not libdirs: - return False - - # Dependency-safe order: OpenMP runtime, then MKL core, threading layer, - # interface layer, and the single-dynamic-library runtime (used by the - # extension's interface-layer check). - patterns = ( - "libiomp5.so", - "libmkl_core.so*", - "libmkl_sequential.so*", - "libmkl_intel_thread.so*", - "libmkl_intel_lp64.so*", - "libmkl_intel_ilp64.so*", - "libmkl_rt.so*", - ) - dlopen = ctypes.CDLL(None).dlopen - dlopen.restype = ctypes.c_void_p - dlopen.argtypes = (ctypes.c_char_p, ctypes.c_int) - loaded = False - for pattern in patterns: - for d in libdirs: - for path in sorted(glob.glob(os.path.join(d, pattern))): - if dlopen(os.fsencode(path), os.RTLD_LAZY | os.RTLD_LOCAL): - loaded = True - return loaded - - def _load_module(name): from importlib import import_module - try: - return import_module(f"scs.{name}") - except ImportError: - # The wheel _scs_mkl extension resolves MKL from the `mkl` PyPI package - # (scs[mkl]) rather than vendored libraries; make those loadable and - # retry. Without them the ImportError propagates and AUTO falls back. - if name != "_scs_mkl" or not _preload_intel_mkl(): - raise - return import_module(f"scs.{name}") + return import_module(f"scs.{name}") def _resolve_auto(): diff --git a/scs_source b/scs_source index 9362c04b..3532da32 160000 --- a/scs_source +++ b/scs_source @@ -1 +1 @@ -Subproject commit 9362c04b7f40bb56f3d166217abc3715c503ce31 +Subproject commit 3532da329f53125e9ae9c6d1fb62e5d4ffa0deb0 diff --git a/test/test_solve_random_cone_prob_mkl.py b/test/test_solve_random_cone_prob_mkl.py index 5c475650..1feeb257 100644 --- a/test/test_solve_random_cone_prob_mkl.py +++ b/test/test_solve_random_cone_prob_mkl.py @@ -1,6 +1,4 @@ from __future__ import print_function, division -import os -import subprocess import sys import platform import scs @@ -13,11 +11,10 @@ # Uses scs to solve a random cone problem # ############################################# -# The MKL backend is available on x86-64 Linux (wheels: via the scs[mkl] -# extra; source builds: link_mkl/mkl_backend) and on Windows source builds. -# Skip on platforms where MKL is never available, and skip when the -# extension is present but its MKL runtime is not (e.g. a wheel install -# without scs[mkl]). +# The MKL backend ships in the x86-64 Linux wheels (MKL linked statically +# into _scs_mkl) and in source builds with link_mkl / mkl_static_prefix. +# Skip on platforms where it is never available, and when the extension is +# absent (musllinux, aarch64, macOS). if sys.platform == "darwin": pytest.skip("MKL is not available on macOS", allow_module_level=True) if sys.platform == "linux" and platform.machine() != "x86_64": @@ -26,7 +23,6 @@ try: from scs import _scs_mkl # noqa: E402 except ImportError: - # openblas-only builds (musllinux), or a wheel without the scs[mkl] extra pytest.skip("MKL backend not importable", allow_module_level=True) # cone: @@ -43,15 +39,6 @@ params = {"verbose": True, "eps_abs": 1e-7, "eps_rel": 1e-7, "eps_infeas": 1e-7} -@pytest.mark.skipif( - not (sys.platform == "linux" and platform.machine() == "x86_64"), - reason="Threaded MKL linkage is only checked on Linux x86_64", -) -def test_mkl_module_links_intel_openmp(): - out = subprocess.check_output(["ldd", os.fspath(_scs_mkl.__file__)], text=True) - assert "libiomp5" in out - - def test_solve_feasible(): rng = np.random.RandomState(3000) data, p_star = tools.gen_feasible(K, n=m // 3, density=0.1, rng=rng) From a50807eb70b6007b208e36421930c6d23b819d9a Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Thu, 3 Sep 2026 20:29:18 +0100 Subject: [PATCH 15/19] Smoke test: the dense backend is named cpu_dense Co-Authored-By: Claude Fable 5.1 --- .github/workflows/build.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 9078e6a3..e28fea41 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -372,7 +372,7 @@ jobs: auto_file = os.path.basename(scs._resolve_auto().__file__) assert auto_file.startswith("_scs_mkl"), auto_file print("auto resolves to", auto_file) - for name in ["qdldl", "cpu_indirect", "dense", "mkl", "auto"]: + for name in ["qdldl", "cpu_indirect", "cpu_dense", "mkl", "auto"]: kw = {} if name == "auto" else {"linear_solver": name} sol = scs.SCS(data, cone, verbose=True, **kw).solve() assert sol["info"]["status"] == "solved", (name, sol["info"]["status"]) From ca985bf71b11095d6c86a039051ca89fc2cfb2fb Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Thu, 3 Sep 2026 21:10:31 +0100 Subject: [PATCH 16/19] link_mkl: keep mkl_rt a hard dependency under --as-needed The core's MKL_Set_Interface_Layer reference is weak so that static MKL links without mkl_rt. In the dynamic link_mkl build that made mkl_rt referenced only weakly, --as-needed dropped it from NEEDED, the symbol resolved to NULL and the interface-layer guard silently no longer ran (the ILP64-guard CI lane caught it). Link with --no-as-needed on Linux, as the core's own CMake and Makefile builds already do. Co-Authored-By: Claude Fable 5.1 --- meson.build | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/meson.build b/meson.build index 04d1d327..65ab0031 100644 --- a/meson.build +++ b/meson.build @@ -365,7 +365,11 @@ if get_option('link_mkl') error('link_mkl requires mkl_rt (or mkl-sdl). _scs_mkl calls MKL_Set_Interface_Layer() at startup and cannot be built safely without it.') endif _mkl_deps = [mkl_blas_deps] + _base_deps + [mkl_rt_dep] - _mkl_link_args = [] + # The core's reference to MKL_Set_Interface_Layer is weak (static MKL has + # no mkl_rt), so with --as-needed the linker would drop mkl_rt, the + # symbol would resolve to NULL and the interface-layer guard would be + # skipped. Keep mkl_rt a hard dependency of the dynamic build. + _mkl_link_args = host_machine.system() == 'linux' ? ['-Wl,--no-as-needed'] : [] elif mkl_static_prefix != '' # Static MKL from Intel's mkl-static archives: LP64 interface, sequential # threading (no OpenMP runtime to duplicate), linked as a group because the From 8e12289cbd387975d4f6f94910e07a0c33269c2f Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Thu, 3 Sep 2026 22:00:56 +0100 Subject: [PATCH 17/19] Tidy the static-MKL build: keep rt in link_mkl, say #423 once - link_mkl builds linked _base_deps + mkl_rt and so dropped the rt that every other extension gets through _deps; use _deps. - The static branch's dl/m/rt dependencies are one required line; the install script loses knobs and fallbacks the container never takes. - The #423 rationale lives in pyproject.toml; meson, the workflow and the README point there instead of retelling it. - wheel_audit keeps the three invariants that matter (no Intel shared library, no dynamic MKL dependency, no absolute rpath entry). - AUTO is resolved once per process. Co-Authored-By: Claude Fable 5.1 --- .github/scripts/install_mkl_static.sh | 19 +++++---------- .github/workflows/build.yml | 34 ++++++++++----------------- README.md | 26 +++++++++----------- meson.build | 26 +++++++------------- pyproject.toml | 18 ++++++-------- 5 files changed, 46 insertions(+), 77 deletions(-) diff --git a/.github/scripts/install_mkl_static.sh b/.github/scripts/install_mkl_static.sh index 42d1b294..b70dd79c 100755 --- a/.github/scripts/install_mkl_static.sh +++ b/.github/scripts/install_mkl_static.sh @@ -1,23 +1,16 @@ #!/bin/bash -# Lay out Intel's static oneMKL archives under a prefix for the static -# _scs_mkl link (see mkl_static_prefix in meson.options). The archives come -# from Intel's own mkl-static PyPI wheel; pip download with an explicit -# platform tag so the build container's python and glibc are irrelevant. +# Lay out Intel's static oneMKL archives (the mkl-static and mkl-include PyPI +# wheels) under a prefix for the static _scs_mkl link; see mkl_static_prefix +# in meson.options. Runs inside the cibuildwheel manylinux container. set -euo pipefail prefix=${1:?usage: install_mkl_static.sh } -ver=${MKL_STATIC_VERSION:-2026.1.0} -py=$(ls -d /opt/python/cp3*-cp3*/bin/python 2>/dev/null | head -1 || command -v python3) +py=$(ls -d /opt/python/cp3*-cp3*/bin/python | head -1) tmp=$(mktemp -d) "$py" -m pip download --quiet --no-deps --only-binary=:all: \ - --platform manylinux_2_28_x86_64 -d "$tmp" "mkl-static==$ver" "mkl-include==$ver" + --platform manylinux_2_28_x86_64 -d "$tmp" mkl-static==2026.1.0 mkl-include==2026.1.0 for whl in "$tmp"/*.whl; do "$py" -m zipfile -e "$whl" "$tmp/unpacked" done mkdir -p "$prefix" -# wheel data files live under .data/data/{lib,include} -cp -r "$tmp"/unpacked/*.data/data/. "$prefix"/ -for a in libmkl_intel_lp64.a libmkl_sequential.a libmkl_core.a; do - test -f "$prefix/lib/$a" || { echo "install_mkl_static: missing $prefix/lib/$a" >&2; exit 1; } -done +cp -r "$tmp"/unpacked/*.data/data/. "$prefix"/ # {lib,include} rm -rf "$tmp" -echo "install_mkl_static: oneMKL $ver archives under $prefix" diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index e28fea41..7cfd8b5d 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -280,10 +280,9 @@ jobs: path: ./wheelhouse/*.whl wheel_audit: - # Static gate for #423-class defects across every Linux wheel: no wheel - # may depend on a dynamic MKL (the backend is linked statically), ship - # an Intel shared library, or carry a build-machine path in an rpath - # (which let the 3.3.0 container tests pass while user machines failed). + # No Linux wheel may depend on a dynamic MKL (it is linked statically), + # ship an Intel shared library, or carry an absolute rpath entry; see + # pyproject.toml (#423). name: Audit wheel ELF metadata needs: build_wheels runs-on: ubuntu-latest @@ -307,23 +306,19 @@ jobs: with zipfile.ZipFile(whl) as z: z.extractall(td) sos = glob.glob(os.path.join(td, "**", "*.so*"), recursive=True) - names = [os.path.basename(so) for so in sos] - if not any(n.startswith("libopenblas") for n in names): - bad.append((name, "-", f"no vendored OpenBLAS: {names}")) - intel = [n for n in names if n.startswith(("libmkl", "libiomp"))] + intel = [os.path.basename(so) for so in sos + if os.path.basename(so).startswith(("libmkl", "libiomp"))] if intel: bad.append((name, "-", f"Intel shared libraries in wheel: {intel}")) for so in sos: rel = os.path.relpath(so, td) - proc = subprocess.run(["readelf", "-d", so], capture_output=True, text=True) - if proc.returncode != 0: - bad.append((name, rel, f"readelf failed: {proc.stderr.strip()}")) - continue - needed = re.findall(r"\(NEEDED\)[^\[]*\[([^\]]+)\]", proc.stdout) + out = subprocess.run(["readelf", "-d", so], capture_output=True, + text=True, check=True).stdout + needed = re.findall(r"\(NEEDED\)[^\[]*\[([^\]]+)\]", out) mkl = [n for n in needed if n.startswith(("libmkl", "libiomp"))] if mkl: bad.append((name, rel, f"dynamic MKL dependency: {mkl}")) - for rp in re.findall(r"\((?:RPATH|RUNPATH)\)[^\[]*\[([^\]]*)\]", proc.stdout): + for rp in re.findall(r"\((?:RPATH|RUNPATH)\)[^\[]*\[([^\]]*)\]", out): for entry in rp.split(":"): if entry and not entry.startswith("$ORIGIN"): bad.append((name, rel, f"non-relative rpath entry '{entry}'")) @@ -335,10 +330,9 @@ jobs: python3 audit.py wheel_smoke_clean_env: - # The release gate for #423-class failures: install the built wheel in a - # pristine container with nothing on the loader path and solve with every - # backend, including a PSD cone so BLAS/LAPACK run. On x86-64 the static - # MKL backend must import and be AUTO's choice. + # The release gate: install the built wheel in a pristine container with + # nothing on the loader path and solve with every backend (a PSD cone so + # BLAS/LAPACK run). On x86-64 the static MKL backend must be AUTO's choice. name: Clean-container wheel smoke test (${{ matrix.python }}) needs: build_wheels runs-on: ubuntu-latest @@ -416,9 +410,7 @@ jobs: path: dist/*.tar.gz upload_pypi: - # The clean-container smoke tests and the ELF audit gate publishing: - # wheels that only work in the build container, carry build-machine - # paths, or drift their MKL linkage must never reach PyPI (see #423). + # publishing waits for the clean-container smoke tests and the ELF audit needs: [build_wheels, build_sdist, wheel_smoke_clean_env, wheel_audit] runs-on: ubuntu-latest environment: pypi diff --git a/README.md b/README.md index c05a3702..3d7d65dd 100644 --- a/README.md +++ b/README.md @@ -15,14 +15,12 @@ The full documentation is available [here](https://www.cvxgrp.org/scs/). pip install scs ``` -On x86-64 Linux the wheels include the MKL Pardiso direct linear solver, -linked statically into the `_scs_mkl` extension, and use it automatically: -it is faster than the built-in QDLDL solver for most problems, often -dramatically so on larger ones, and nothing extra needs to be installed. The -MKL backend is single-threaded (a bundled OpenMP runtime can abort a process -that already has one, and SCS's per-iteration work does not parallelize -well); Intel's license notice ships in the wheel as `LICENSE-INTEL-MKL.txt`. -Every other wheel falls back to QDLDL. +On x86-64 Linux the wheels include the MKL Pardiso direct linear solver +(MKL linked statically into `_scs_mkl`, single-threaded) and use it +automatically: it is faster than the built-in QDLDL solver for most problems, +often dramatically so on larger ones, and nothing extra needs to be +installed. Intel's license notice ships in the wheel as +`LICENSE-INTEL-MKL.txt`. Every other wheel falls back to QDLDL. To install from source: ```bash @@ -50,13 +48,11 @@ Available values: `AUTO`, `QDLDL`, `CPU_INDIRECT`, `MKL`, `ACCELERATE`, `CPU_DENSE`, `GPU_INDIRECT`, `CUDSS`. The pre-built wheels (`pip install scs`) link OpenBLAS on Linux and Windows, -and Apple Accelerate on macOS. The x86-64 Linux wheels also ship the MKL -Pardiso backend with MKL linked statically into `_scs_mkl`, and `AUTO` -selects it; MKL is deliberately not vendored as shared libraries, whose -dlopen'd CPU dispatch kernels are invisible to wheel-repair tools and whose -absence aborts the process at solve time (cvxgrp/scs#423). The MKL backend -is also available in source builds (e.g. conda environments providing MKL). -When installing from source, +and Apple Accelerate on macOS; the x86-64 Linux wheels also ship the MKL +Pardiso backend (see Installation). MKL is linked statically rather than +bundled as shared libraries, whose dlopen'd CPU dispatch kernels wheel-repair +tools cannot see (cvxgrp/scs#423). The MKL backend is also available in +source builds (e.g. conda environments providing MKL), where additional backends can be enabled with build-time flags: ```bash diff --git a/meson.build b/meson.build index 65ab0031..304c9621 100644 --- a/meson.build +++ b/meson.build @@ -24,11 +24,8 @@ print(incdir) # # link_mkl links every extension against a dynamic MKL from the environment # (conda/source builds). mkl_static_prefix builds only _scs_mkl, with MKL -# linked statically and its symbols hidden, while every other extension uses -# the platform BLAS: this is the x86-64 Linux wheel configuration. Static -# linking is what makes the wheel self-contained -- dynamic MKL loads its CPU -# dispatch kernels via dlopen, invisible to wheel-repair tools, and the 3.3.0 -# wheels shipped without them (cvxgrp/scs#423). +# linked statically, while every other extension uses the platform BLAS: the +# x86-64 Linux wheel configuration (see pyproject.toml for why). blas_deps = [] mkl_blas_deps = [] mkl_pkg_name = '' @@ -364,19 +361,17 @@ if get_option('link_mkl') if not mkl_rt_dep.found() error('link_mkl requires mkl_rt (or mkl-sdl). _scs_mkl calls MKL_Set_Interface_Layer() at startup and cannot be built safely without it.') endif - _mkl_deps = [mkl_blas_deps] + _base_deps + [mkl_rt_dep] + _mkl_deps = _deps + [mkl_rt_dep] # The core's reference to MKL_Set_Interface_Layer is weak (static MKL has # no mkl_rt), so with --as-needed the linker would drop mkl_rt, the # symbol would resolve to NULL and the interface-layer guard would be # skipped. Keep mkl_rt a hard dependency of the dynamic build. - _mkl_link_args = host_machine.system() == 'linux' ? ['-Wl,--no-as-needed'] : [] + _mkl_link_args = is_linux ? ['-Wl,--no-as-needed'] : [] elif mkl_static_prefix != '' - # Static MKL from Intel's mkl-static archives: LP64 interface, sequential - # threading (no OpenMP runtime to duplicate), linked as a group because the - # archives reference each other, with every MKL symbol hidden so an MKL - # already in the process (an MKL-backed NumPy, say) is never interposed. - # The core's MKL_Set_Interface_Layer reference is weak, so the mkl_rt-only - # runtime check is skipped automatically. + # Intel's mkl-static archives: LP64, sequential, linked as a group (they + # reference each other) with every MKL symbol hidden so an MKL already in + # the process is never interposed. The core's MKL_Set_Interface_Layer + # reference is weak, so the mkl_rt-only runtime check is skipped. if get_option('use_blas64') error('mkl_static_prefix links the LP64 interface; use link_mkl for ILP64 MKL builds.') endif @@ -388,10 +383,7 @@ elif mkl_static_prefix != '' endif _mkl_archives += _mkl_lib / _a endforeach - _mkl_deps = _base_deps + [cc.find_library('dl', required : false), cc.find_library('m', required : false)] - if is_linux - _mkl_deps += cc.find_library('rt', required : true) - endif + _mkl_deps = _base_deps + [cc.find_library('dl'), cc.find_library('m'), cc.find_library('rt')] _mkl_link_args = ['-Wl,--start-group'] + _mkl_archives + ['-Wl,--end-group', '-Wl,--exclude-libs,ALL'] # Intel Simplified Software License: the notice ships with the binaries. install_data('LICENSE-INTEL-MKL.txt', install_dir: scs_dir) diff --git a/pyproject.toml b/pyproject.toml index 2e5d56bb..986d68ac 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -67,17 +67,13 @@ repair-wheel-command = "delvewheel repair -w {dest_dir} {wheel}" # Openblas installation for 3 different linux images # x86_64 Linux: the default backends link vendored OpenBLAS; _scs_mkl links -# MKL *statically* (Intel's mkl-static archives, sequential, symbols hidden) -# so the wheel is self-contained. The 3.3.0 wheels vendored dynamic MKL and -# shipped it incomplete -- its dlopen'd CPU dispatch kernels are invisible to -# auditwheel -- so every solve aborted on machines without MKL on the loader -# path (#423). Static linking has no dispatch libraries to lose. -# -# Two rules, both learned from #423: nothing in `environment` may point the -# loader at build-machine libraries (the wheel tests must run exactly as -# clean as a user's machine), and upload_pypi is gated on the -# wheel_smoke_clean_env job, which installs the wheel in a pristine -# container and solves with every backend. +# MKL statically (Intel's mkl-static archives, sequential, symbols hidden) so +# the wheel is self-contained. The 3.3.0 wheels vendored dynamic MKL without +# its dlopen'd CPU dispatch kernels, which auditwheel cannot see, and every +# solve aborted on machines without MKL on the loader path (cvxgrp/scs#423). +# Hence: nothing in `environment` may point the loader at build-machine +# libraries, and publishing waits for wheel_smoke_clean_env (a pristine +# container solving with every backend) and wheel_audit. [[tool.cibuildwheel.overrides]] select = "*-manylinux_x86_64" inherit.before-all = "append" From 720a0f855e5e9cf0ff258ec72a2fd37b961ecea5 Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Fri, 4 Sep 2026 12:33:13 +0100 Subject: [PATCH 18/19] wheel_audit: require the shipped inventory; publish waits for the source-MKL lanes The audit only forbade things (dynamic MKL, Intel libraries, absolute rpaths); it now also requires what a wheel must contain: the standard extensions and a vendored OpenBLAS in every Linux wheel, _scs_mkl in exactly the x86-64 manylinux wheels, and at least one such wheel, so a selector or build change cannot drop the MKL backend unnoticed. upload_pypi also waits for build_mkl, since the sdist carries the link_mkl source build. README: MKL ships in the x86-64 manylinux wheels specifically (musllinux has no MKL). Co-Authored-By: Claude Fable 5.1 --- .github/workflows/build.yml | 23 ++++++++++++++++++----- README.md | 6 +++--- 2 files changed, 21 insertions(+), 8 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 7cfd8b5d..b941e1eb 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -280,9 +280,10 @@ jobs: path: ./wheelhouse/*.whl wheel_audit: - # No Linux wheel may depend on a dynamic MKL (it is linked statically), - # ship an Intel shared library, or carry an absolute rpath entry; see - # pyproject.toml (#423). + # Every Linux wheel must ship the standard extensions and a vendored + # OpenBLAS, _scs_mkl exactly in the x86-64 manylinux wheels; no wheel may + # depend on a dynamic MKL (it is linked statically), ship an Intel shared + # library, or carry an absolute rpath entry; see pyproject.toml (#423). name: Audit wheel ELF metadata needs: build_wheels runs-on: ubuntu-latest @@ -306,6 +307,16 @@ jobs: with zipfile.ZipFile(whl) as z: z.extractall(td) sos = glob.glob(os.path.join(td, "**", "*.so*"), recursive=True) + mods = {os.path.basename(so).split(".")[0] for so in sos} + want = {"_scs_direct", "_scs_indirect", "_scs_dense"} + if "manylinux" in name and "x86_64" in name: + want.add("_scs_mkl") + elif "_scs_mkl" in mods: + bad.append((name, "-", "_scs_mkl belongs only in x86-64 manylinux wheels")) + if want - mods: + bad.append((name, "-", f"missing extensions: {sorted(want - mods)}")) + if not any(os.path.basename(so).startswith("libopenblas") for so in sos): + bad.append((name, "-", "no vendored OpenBLAS")) intel = [os.path.basename(so) for so in sos if os.path.basename(so).startswith(("libmkl", "libiomp"))] if intel: @@ -322,6 +333,7 @@ jobs: for entry in rp.split(":"): if entry and not entry.startswith("$ORIGIN"): bad.append((name, rel, f"non-relative rpath entry '{entry}'")) + assert any("manylinux" in w and "x86_64" in w for w in map(os.path.basename, whls)), "no x86-64 manylinux wheel" print(f"audited {len(whls)} wheels") for whl, so, msg in bad: print(f"BAD: {whl} :: {so} :: {msg}") @@ -410,8 +422,9 @@ jobs: path: dist/*.tar.gz upload_pypi: - # publishing waits for the clean-container smoke tests and the ELF audit - needs: [build_wheels, build_sdist, wheel_smoke_clean_env, wheel_audit] + # publishing waits for the clean-container smoke tests, the wheel audit + # and the source-MKL lanes (the sdist carries the link_mkl build path) + needs: [build_wheels, build_sdist, wheel_smoke_clean_env, wheel_audit, build_mkl] runs-on: ubuntu-latest environment: pypi permissions: diff --git a/README.md b/README.md index 3d7d65dd..2f64539a 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ The full documentation is available [here](https://www.cvxgrp.org/scs/). pip install scs ``` -On x86-64 Linux the wheels include the MKL Pardiso direct linear solver +On x86-64 Linux the manylinux (glibc) wheels include the MKL Pardiso direct linear solver (MKL linked statically into `_scs_mkl`, single-threaded) and use it automatically: it is faster than the built-in QDLDL solver for most problems, often dramatically so on larger ones, and nothing extra needs to be @@ -48,7 +48,7 @@ Available values: `AUTO`, `QDLDL`, `CPU_INDIRECT`, `MKL`, `ACCELERATE`, `CPU_DENSE`, `GPU_INDIRECT`, `CUDSS`. The pre-built wheels (`pip install scs`) link OpenBLAS on Linux and Windows, -and Apple Accelerate on macOS; the x86-64 Linux wheels also ship the MKL +and Apple Accelerate on macOS; the x86-64 manylinux wheels also ship the MKL Pardiso backend (see Installation). MKL is linked statically rather than bundled as shared libraries, whose dlopen'd CPU dispatch kernels wheel-repair tools cannot see (cvxgrp/scs#423). The MKL backend is also available in @@ -73,7 +73,7 @@ pip install . -Csetup-args=-Duse_spectral_cones=true ``` Notes: -- Linux x86_64 wheels ship a `_scs_mkl` extension with sequential MKL linked statically (CI asserts that no wheel carries a dynamic MKL dependency). Windows wheels do not include the MKL backend; Windows source builds use sequential MKL because Intel's conda `pkg-config` metadata for the threaded variant is still broken. +- x86-64 manylinux wheels ship a `_scs_mkl` extension with sequential MKL linked statically (CI asserts the shipped inventory and that no wheel carries a dynamic MKL dependency). The musllinux, aarch64, macOS and Windows wheels do not include the MKL backend; Windows source builds use sequential MKL because Intel's conda `pkg-config` metadata for the threaded variant is still broken. - `BLAS64` is a general SCS build mode for ILP64 BLAS/LAPACK libraries, not an MKL-only feature. - For the MKL Pardiso backend specifically, `BLAS64` must be paired with 64-bit SCS integers (`DLONG` / `int32=false`), and SCS now fails early if another library in the process has already fixed MKL to an incompatible LP64/ILP64 interface layer. From 635550395c1a242c348ef2ef346d8dca8a51291e Mon Sep 17 00:00:00 2001 From: Brendan O'Donoghue Date: Fri, 4 Sep 2026 17:48:52 +0100 Subject: [PATCH 19/19] Windows wheels: pin conda-forge OpenBLAS to 0.3.33 conda-forge's win-64 libopenblas 0.3.34 (pthreads_h877e47f_1) crashes with an access violation inside dgemm_kernel_HASWELL / dgemm_kernel_ZEN on AMD Zen 4 and Zen 5 CPUs whose hypervisor or firmware exposes AVX-512, on plain dsyrk/dpotrf calls; 0.3.33 and earlier builds are fine on the same machines (conda-forge/openblas-feedstock#196). The wheel CI hit it on the runners that happen to have such CPUs, and every Ryzen 7000/9000 desktop would have. Pin until a fixed build ships. Co-Authored-By: Claude Fable 5.1 --- .github/workflows/build.yml | 5 ++++- README.md | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index b941e1eb..9f98da69 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -253,7 +253,10 @@ jobs: - name: Install openblas from conda on Windows if: runner.os == 'Windows' - run: conda install -y openblas pkgconfig + # 0.3.33 pinned: conda-forge's win-64 0.3.34 build crashes inside its + # DGEMM kernels on AMD Zen 4/5 CPUs that expose AVX-512 + # (conda-forge/openblas-feedstock#196). Unpin once a fixed build ships. + run: conda install -y "openblas=0.3.33" pkgconfig - name: Build wheels uses: pypa/cibuildwheel@v4.2.0 diff --git a/README.md b/README.md index 2f64539a..7346f5b8 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,7 @@ pip install . -Csetup-args=-Duse_spectral_cones=true Notes: - x86-64 manylinux wheels ship a `_scs_mkl` extension with sequential MKL linked statically (CI asserts the shipped inventory and that no wheel carries a dynamic MKL dependency). The musllinux, aarch64, macOS and Windows wheels do not include the MKL backend; Windows source builds use sequential MKL because Intel's conda `pkg-config` metadata for the threaded variant is still broken. +- Windows wheels link conda-forge OpenBLAS pinned to 0.3.33: the win-64 0.3.34 build crashes inside its DGEMM kernels on AMD Zen 4/5 CPUs that expose AVX-512 (conda-forge/openblas-feedstock#196), so Windows source builds should avoid that build too. - `BLAS64` is a general SCS build mode for ILP64 BLAS/LAPACK libraries, not an MKL-only feature. - For the MKL Pardiso backend specifically, `BLAS64` must be paired with 64-bit SCS integers (`DLONG` / `int32=false`), and SCS now fails early if another library in the process has already fixed MKL to an incompatible LP64/ILP64 interface layer.