Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

openblas, mkl: also export unversioned libraries for linux #87134

Merged
merged 2 commits into from May 8, 2020

Conversation

@jonringer
Copy link
Contributor

jonringer commented May 6, 2020

Motivation for this change

While working on #87118 , I noticed that the tests weren't able to find -llapack, even though it was available through openblas. Turns out that it was explicitly searching for just liblapack.so and not liblapack.so.3.

for linux, this will now export both the normal .so and the versioned .so.3

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
@bhipple
bhipple approved these changes May 7, 2020
Copy link
Contributor

bhipple left a comment

Seems like a good change to me!

@drewrisinger
Copy link
Contributor

drewrisinger commented May 7, 2020

cc @matthewbauer, figured he should be looped in b/c he spent a lot of time looking at the LAPACK/BLAS system recently.
EDIT: whoops, missed the pre-existing review request, didn't read closely.

@matthewbauer
Copy link
Member

matthewbauer commented May 7, 2020

So -lblas and -llapack should resolve to blas.so.3 and lapack.so.3 if blas.soandlapack.so` are unavailable. But some other things might expect it to be there.

Copy link
Member

matthewbauer left a comment

Almost forgot that we also need this for mkl as well!

Some build systems look for this specifically.
@bhipple bhipple force-pushed the jonringer:openblas-export-unversioned branch from 386b5ab to 296f5b0 May 8, 2020
@bhipple
Copy link
Contributor

bhipple commented May 8, 2020

Updated with mkl support as well. I also used a let binding to avoid excessively long strings for the shlib file extension lookup.

LGTM now:

$ ls -l $(nix-build -A openblas)/lib/lib*
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libblas.so -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libblas.so.3 -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libcblas.so -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libcblas.so.3 -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/liblapacke.so -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/liblapacke.so.3 -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/liblapack.so -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/liblapack.so.3 -> libopenblasp-r0.3.9.so
-r-xr-xr-x  3 root root 27850664 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libopenblas.so -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libopenblas.so.0 -> libopenblasp-r0.3.9.so

$ ls -l $(nix-build -A mkl)/lib/lib*
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libblas.so -> libmkl_rt.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libblas.so.3 -> libmkl_rt.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libcblas.so -> libmkl_rt.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libcblas.so.3 -> libmkl_rt.so
-r-xr-xr-x  8 root root   697489 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libiomp5_db.so
-r-xr-xr-x  8 root root  2315028 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libiomp5.so
-r-xr-xr-x  8 root root   192373 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libiompstubs5.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/liblapacke.so -> libmkl_rt.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/liblapacke.so.3 -> libmkl_rt.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/liblapack.so -> libmkl_rt.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/liblapack.so.3 -> libmkl_rt.so
-r-xr-xr-x  8 root root 47925310 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_avx2.so
-r-xr-xr-x  8 root root 65553609 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_avx512_mic.so
-r-xr-xr-x  8 root root 60359069 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_avx512.so
-r-xr-xr-x  8 root root 51230378 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_avx.so
-r-xr-xr-x  8 root root   499559 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_blacs_intelmpi_ilp64.so
-r-xr-xr-x  8 root root   302050 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_blacs_intelmpi_lp64.so
-r-xr-xr-x  8 root root   504799 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_blacs_openmpi_ilp64.so
-r-xr-xr-x  8 root root   307258 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_blacs_openmpi_lp64.so
-r-xr-xr-x  8 root root   499880 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_blacs_sgimpt_ilp64.so
-r-xr-xr-x  8 root root   302339 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_blacs_sgimpt_lp64.so
-r-xr-xr-x  8 root root   169344 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_cdft_core.so
-r-xr-xr-x  8 root root 71510820 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_core.so
-r-xr-xr-x  8 root root 40895803 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_def.so
-r-xr-xr-x  8 root root 10507840 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_gf_ilp64.so
-r-xr-xr-x  8 root root 11218247 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_gf_lp64.so
-r-xr-xr-x  8 root root 25398547 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_gnu_thread.so
-r-xr-xr-x  8 root root 10512146 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_intel_ilp64.so
-r-xr-xr-x  8 root root 11222553 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_intel_lp64.so
-r-xr-xr-x  8 root root 40645032 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_intel_thread.so
-r-xr-xr-x  8 root root 48789719 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_mc3.so
-r-xr-xr-x  8 root root 47256466 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_mc.so
-r-xr-xr-x  8 root root  6592837 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_rt.so
-r-xr-xr-x  8 root root  7725985 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_scalapack_ilp64.so
-r-xr-xr-x  8 root root  7735678 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_scalapack_lp64.so
-r-xr-xr-x  8 root root 22190068 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_sequential.so
-r-xr-xr-x  8 root root 12556013 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_avx2.so
-r-xr-xr-x  8 root root 13763906 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_avx512_mic.so
-r-xr-xr-x  8 root root 12386491 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_avx512.so
-r-xr-xr-x  8 root root 12616795 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_avx.so
-r-xr-xr-x  8 root root  6375214 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_cmpt.so
-r-xr-xr-x  8 root root  6748578 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_def.so
-r-xr-xr-x  8 root root 11355391 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_mc2.so
-r-xr-xr-x  8 root root 11433131 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_mc3.so
-r-xr-xr-x  8 root root 11386740 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_mc.so
@bhipple
Copy link
Contributor

bhipple commented May 8, 2020

@GrahamcOfBorg build openblas

@bhipple
bhipple approved these changes May 8, 2020
@bhipple bhipple changed the title openblas: also export unversioned libraries for linux openblas, mkl: also export unversioned libraries for linux May 8, 2020
@ofborg ofborg bot requested a review from bhipple May 8, 2020
@FRidh FRidh requested a review from matthewbauer May 8, 2020
@FRidh FRidh added this to WIP in Staging via automation May 8, 2020
@FRidh FRidh moved this from WIP to Needs review in Staging May 8, 2020
@FRidh FRidh merged commit 7066420 into NixOS:staging May 8, 2020
19 checks passed
19 checks passed
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="296f5b0"; rev="296f5b0b60e57d19afe235a6abc5adc659f2256e"; } ./pkgs/t
Details
grahamcofborg-eval-lib-tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="296f5b0"; rev="296f5b0b60e57d19afe235a6abc5adc659f2256e"; } ./nixos/
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="296f5b0"; rev="296f5b0b60e57d19afe235a6abc5adc659f2256e"; } ./nixos/
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="296f5b0"; rev="296f5b0b60e57d19afe235a6abc5adc659f2256e"; } ./nixos/
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="296f5b0"; rev="296f5b0b60e57d19afe235a6abc5adc659f2256e"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="296f5b0"; rev="296f5b0b60e57d19afe235a6abc5adc659f2256e"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="296f5b0"; rev="296f5b0b60e57d19afe235a6abc5adc659f2256e"; } ./pkgs/t
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
mkl, mkl.passthru.tests, openblas, openblas.passthru.tests on aarch64-linux Success
Details
mkl, mkl.passthru.tests, openblas, openblas.passthru.tests on x86_64-darwin Success
Details
mkl, mkl.passthru.tests, openblas, openblas.passthru.tests on x86_64-linux Success
Details
openblas on aarch64-linux Success
Details
openblas on x86_64-linux Success
Details
Staging automation moved this from Needs review to Done May 8, 2020
@jonringer jonringer deleted the jonringer:openblas-export-unversioned branch May 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Staging
  
Done
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.