Build HiGHS's BLAS through libblastrampoline on all platforms#9
Open
nardi wants to merge 9 commits into
Open
Conversation
nardi
force-pushed
the
claude/libblastrampoline-investigation-rat2oa
branch
11 times, most recently
from
July 17, 2026 07:27
70c2e0c to
1999eaa
Compare
nardi
commented
Jul 17, 2026
nardi
force-pushed
the
claude/libblastrampoline-investigation-rat2oa
branch
2 times, most recently
from
July 17, 2026 07:39
f734b1e to
30fa3eb
Compare
Compile HiGHS + HiPO from source everywhere instead of downloading the
prebuilt static-apache archive. On Linux and Windows, the HiPO BLAS
dependency is satisfied through libblastrampoline (lbt) rather than by
linking OpenBLAS directly: lbt is fetched as a prebuilt binary and staged
under the "openblas" name that HiGHS's own find_package(BLAS)/
BLA_VENDOR=OpenBLAS lookup resolves (confirmed by reading HiGHS's actual
cmake/FindHipoDeps.cmake), so no upstream HiGHS change is needed. OpenBLAS
is no longer compiled by this project (that's what caused the AVX2/AVX512
build breakage this project moved away from before): it's fetched prebuilt
from conda-forge and used purely as the default lbt-forwarded backend,
selected at import time in _blas_backend.py rather than at build time.
On macOS, none of this applies. Reading FindHipoDeps.cmake showed HiGHS
unconditionally links Apple's Accelerate framework there regardless of
anything found via find_package/find_library, so lbt is skipped entirely
and cyhighs's macOS wheels use Accelerate directly. musllinux (Alpine)
wheels are the other exception: conda-forge has no musl builds, so the
default OpenBLAS backend there comes from Alpine's own apk package instead,
discovered via find_library the same way this project sourced OpenBLAS
before adopting lbt.
Getting this working across both the wheel builds (cibuildwheel, with an
auditwheel/delvewheel repair step) and the plain editable build that
tests.yml / uv sync use (no repair step) took several real-CI-driven
fixes, all for the same underlying reason -- the editable build and the
wheel repair each need _core to be able to find its bundled libraries, and
nothing was arranging that:
* HiGHS's own CLI executable statically links libhighs_extras.a in full,
including a call to the OpenBLAS-specific (not standard BLAS/LAPACK ABI)
openblas_set_num_threads() that lbt doesn't provide, so it failed to
link. It's excluded from the build (FetchContent EXCLUDE_FROM_ALL, CMake
3.28+); a no-op stub of that symbol is compiled into _core so the same
latent undefined reference can't crash at runtime either.
* The bundled OpenBLAS and libblastrampoline libraries are installed into
the package directory next to _core, and _core is pointed at them: an
$ORIGIN INSTALL_RPATH on Linux (with FOLLOW_SYMLINK_CHAIN so the real
SONAME target libblastrampoline.so.5 and the OpenBLAS/libgfortran
symlink chains are all installed), and a next-to-the-module copy on
Windows, which has no RPATH equivalent. On Windows the MinGW-only lbt
import library is regenerated for MSVC, locating dumpbin/lib.exe next to
the compiler rather than on PATH.
* _blas_backend.py locates the bundled OpenBLAS relative to _core (via its
import spec, without importing it) rather than relative to its own source
file, so LBT_DEFAULT_LIBS is set correctly even in the editable build
where the compiled extension and the .py sources live in different
directories.
This also adds an optional cyhighs[mkl] extra (Linux x86_64 / Windows
x86_64 only, since Intel doesn't publish MKL for macOS or ARM): if
installed, its bundled MKL runtime is used as the BLAS backend instead of
the bundled OpenBLAS, with no cyhighs rebuild required.
Since every wheel now builds the same way, wheels.yml collapses the old
prebuilt-archive/from-source split into a single cibuildwheel job with a
4-way OS matrix.
The libblastrampoline release tag/version and the per-triplet asset
filenames and hashes are confirmed against a live release listing of
JuliaBinaryWrappers/libblastrampoline_jll.jl; the conda-forge
OpenBLAS/libgfortran coordinates, SONAMEs, transitive dependency chains,
and package layouts are all confirmed against real downloaded and
extracted packages.
nardi
force-pushed
the
claude/libblastrampoline-investigation-rat2oa
branch
from
July 17, 2026 09:08
30fa3eb to
d8b8038
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'benchmarks-macos-latest'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: 5f55cb2 | Previous: 8a22b41 | Ratio |
|---|---|---|---|
benchmarks/test_benchmarks.py::test_array_solve[lp-100] |
490.8296818070012 iter/sec (stddev: 0.0010492169459834447) |
1321.1563235840813 iter/sec (stddev: 0.00007036664582816934) |
2.69 |
benchmarks/test_benchmarks.py::test_array_solve[lp-1000] |
28.106348802603026 iter/sec (stddev: 0.008034769869035807) |
41.32705316964579 iter/sec (stddev: 0.0010590375713438864) |
1.47 |
benchmarks/test_benchmarks.py::test_array_solve[mip-100] |
70.90205982258819 iter/sec (stddev: 0.01054781173897715) |
168.92866655968928 iter/sec (stddev: 0.0013315832741551145) |
2.38 |
benchmarks/test_benchmarks.py::test_array_solve[mip-1000] |
0.07532462883617316 iter/sec (stddev: 2.0135731804708383) |
0.10026955485086086 iter/sec (stddev: 1.3656391960338892) |
1.33 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'benchmarks-ubuntu-latest'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: 5f55cb2 | Previous: 8a22b41 | Ratio |
|---|---|---|---|
benchmarks/test_benchmarks.py::test_array_solve[lp-100] |
626.012771286316 iter/sec (stddev: 0.00003158318991405504) |
777.233101456788 iter/sec (stddev: 0.00007331256211535615) |
1.24 |
benchmarks/test_benchmarks.py::test_array_solve[lp-1000] |
26.001115895089754 iter/sec (stddev: 0.0004250498946479398) |
34.82034632086863 iter/sec (stddev: 0.0011065957488331692) |
1.34 |
benchmarks/test_benchmarks.py::test_array_solve[lp-10000] |
0.09557151702676013 iter/sec (stddev: 0.044215852400218324) |
0.13090608970390633 iter/sec (stddev: 0.0732322434031961) |
1.37 |
benchmarks/test_benchmarks.py::test_array_solve[mip-100] |
150.73137349498393 iter/sec (stddev: 0.00008339486869227329) |
192.67692428504841 iter/sec (stddev: 0.00031427658116649925) |
1.28 |
benchmarks/test_benchmarks.py::test_sparse_solve[lp-1000] |
27.272825504486736 iter/sec (stddev: 0.00011666002765166447) |
33.302138332298114 iter/sec (stddev: 0.0021727177507363834) |
1.22 |
benchmarks/test_benchmarks.py::test_sparse_solve[lp-10000] |
0.09675396111284752 iter/sec (stddev: 0.027342285386286004) |
0.1315023074896695 iter/sec (stddev: 0.08971279272085499) |
1.36 |
benchmarks/test_benchmarks.py::test_sparse_solve[mip-100] |
149.8145341014178 iter/sec (stddev: 0.000050711580865967043) |
183.30126233304014 iter/sec (stddev: 0.0000602145582712725) |
1.22 |
benchmarks/test_benchmarks.py::test_linprog_solve[lp-100] |
567.0744442322075 iter/sec (stddev: 0.000021008155132134128) |
766.8958074624117 iter/sec (stddev: 0.0000230580133124934) |
1.35 |
benchmarks/test_benchmarks.py::test_linprog_solve[lp-1000] |
26.489513489022595 iter/sec (stddev: 0.00011072214860643929) |
32.69740873332307 iter/sec (stddev: 0.002101361619337457) |
1.23 |
benchmarks/test_benchmarks.py::test_linprog_solve[lp-10000] |
0.09641045382427667 iter/sec (stddev: 0.01936647660607037) |
0.131812978927733 iter/sec (stddev: 0.09057955846089433) |
1.37 |
benchmarks/test_benchmarks.py::test_linprog_solve[mip-100] |
146.6016935779977 iter/sec (stddev: 0.00008161222234488715) |
198.77732070009353 iter/sec (stddev: 0.000047917507329365796) |
1.36 |
benchmarks/test_benchmarks.py::test_linprog_solve[mip-1000] |
0.07486727009543323 iter/sec (stddev: 0.038557449780266186) |
0.09122154417035709 iter/sec (stddev: 0.17985829651655016) |
1.22 |
This comment was automatically generated by workflow using github-action-benchmark.
Add a section to the bundling guide explaining how to force a specific BLAS backend and how to enable verbose backend-selection diagnostics.
Add a second benchmark run using the cyhighs[mkl] extra and CYHIGHS_LBT_PREFER=mkl, tracked as its own regression series. Skipped on macOS, where MKL is not published and the extra has no effect.
HiGHS calls openblas_get_num_threads/openblas_set_num_threads directly, believing it is linked against real OpenBLAS. Forward both to lbt's own lbt_get_num_threads/lbt_set_num_threads instead of a no-op stub, so HiGHS's thread-count control reaches whichever backend lbt is actually forwarding to, OpenBLAS or MKL.
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'benchmarks-windows-latest'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: 599149d | Previous: 8a22b41 | Ratio |
|---|---|---|---|
benchmarks/test_benchmarks.py::test_linprog_solve[lp-100] |
388.4264456251682 iter/sec (stddev: 0.0004708258452351514) |
470.871866353365 iter/sec (stddev: 0.00009457495323865227) |
1.21 |
This comment was automatically generated by workflow using github-action-benchmark.
Off links the bundled OpenBLAS directly with no libblastrampoline forwarding layer in between, for local A/B testing against the default lbt-forwarded build. Both HiGHS's own BLAS detection and _core link straight to the real backend, so no thread-count stub is needed and _blas_backend.py no-ops (no lbt to configure). Real OpenBLAS needs libgfortran resolvable at link time, not just at runtime, unlike lbt which has no such dependency, so the no-lbt shim directory stages it there too. Verified end to end with a real local build: linked, imported, and solved an LP via IPM correctly with no lbt anywhere in the dependency tree (confirmed via ldd), and the full test suite passes against it.
Rebuild with CYHIGHS_USE_LBT=OFF in a separate build-dir and track it as its own regression series, alongside the existing OpenBLAS and MKL runs. Skipped on macOS, where the flag has no effect.
Build a second wheel variant per Linux/Windows target with CYHIGHS_USE_LBT=OFF, tagged with a PEP 440 +static local version segment so it never collides with or gets picked up in place of the default lbt-forwarded wheel on PyPI. These are attached directly to the GitHub release instead, since PyPI has no way for pip to choose between two wheels of the same name, version, and platform tag. Document the static wheels and the existing CYHIGHS_LBT_PREFER/ CYHIGHS_LBT_DEBUG environment variables in the bundling guide.
Windows runners default run: steps to pwsh, which parses the whole
script block as PowerShell rather than passing it through, and pwsh's
backslash escaping rules differ from bash's. That broke the embedded
Python one-liner's quoting, confirmed by a real CI failure ("The
string is missing the terminator") on Wheels on windows-latest
(static).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compile HiGHS + HiPO from source everywhere instead of downloading the prebuilt static-apache archive. On Linux and Windows, the HiPO BLAS dependency is satisfied through libblastrampoline (lbt) rather than by linking OpenBLAS directly: lbt is fetched as a prebuilt binary and staged under the "openblas" name that HiGHS's own find_package(BLAS)/ BLA_VENDOR=OpenBLAS lookup resolves (confirmed by reading HiGHS's actual cmake/FindHipoDeps.cmake), so no upstream HiGHS change is needed. OpenBLAS is no longer compiled by this project (that's what caused the AVX2/AVX512 build breakage this project moved away from before): it's fetched prebuilt from conda-forge and used purely as the default lbt-forwarded backend, selected at import time in _blas_backend.py rather than at build time.
On macOS, none of this applies. Reading FindHipoDeps.cmake showed HiGHS unconditionally links Apple's Accelerate framework there regardless of anything found via find_package/find_library, so lbt is skipped entirely and cyhighs's macOS wheels use Accelerate directly, as they did before this change. musllinux (Alpine) wheels are the other exception: conda-forge has no musl builds, so the default OpenBLAS backend there comes from Alpine's own apk package instead, discovered via find_library the same way this project sourced OpenBLAS before adopting lbt.
This also adds an optional cyhighs[mkl] extra (Linux x86_64 / Windows x86_64 only, since Intel doesn't publish MKL for macOS or ARM): if installed, its bundled MKL runtime is used as the BLAS backend instead of the bundled OpenBLAS, with no cyhighs rebuild required.
Since every wheel now builds the same way, wheels.yml collapses the old prebuilt-archive/from-source split into a single cibuildwheel job with a 4-way OS matrix, and CMakeLists.txt drops a redundant variable assignment and extracts the Windows import-library generation into its own function.
The libblastrampoline release tag/version and the per-triplet asset filenames and hashes are confirmed against a live release listing of JuliaBinaryWrappers/libblastrampoline_jll.jl, and the MinGW-only import library situation on Windows is confirmed by extracting a real release archive. The conda-forge OpenBLAS/libgfortran coordinates and the Windows/ Linux package layouts are pinned and confirmed against real repodata.json hashes and real extracted packages. The Linux from-source path itself was exercised locally up to the lbt external fetch boundary to validate the surrounding CMake logic.