Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why and how should blas be added to dependencies #80

Open
maresb opened this issue Jul 3, 2022 · 17 comments
Open

Why and how should blas be added to dependencies #80

maresb opened this issue Jul 3, 2022 · 17 comments
Labels
question Further information is requested

Comments

@maresb
Copy link
Contributor

maresb commented Jul 3, 2022

Comment:

Motivated by this comment I'd like to centralize the info regarding BLAS requirements for Aesara.

In order to solve these sorts of errors, the blas package was added as a host dependency in #48 (see also the links inside).

Since Aesara tries to compile with BLAS at runtime, I believe this should instead be a runtime dependency rather than a host dependency.

Unfortunately I don't currently have a very good understanding of the BLAS libraries. The Conda-Forge BLAS docs seem to be written from the perspective of a program which needs to be built using BLAS, rather than a program that relies on dynamic BLAS compilation at runtime.

@maresb maresb added the question Further information is requested label Jul 3, 2022
@twiecki
Copy link
Contributor

twiecki commented Jul 3, 2022

I think adding it to runtime is pretty obvious then, no?

@maresb
Copy link
Contributor Author

maresb commented Jul 3, 2022

This seems to be a weird edge case: I just added blas to the runtime requirements in 2efe1d4, and now pypy3.7 on Windows fails on the main aesara package with

configparser.NoSectionError: No section: 'blas'
2022-07-03T12:55:12.1607314Z TEST START: D:\bld\win-64\aesara-2.7.4-py37he04ca5b_1.tar.bz2
2022-07-03T12:55:12.2311102Z Multiple meta files found. The meta.yaml file in the base directory (C:\Users\VSSADM~1\AppData\Local\Temp\tmpxgrr8_ot\info\recipe) will be used.
2022-07-03T12:55:12.2313000Z WARNING:conda_build.utils:Multiple meta files found. The meta.yaml file in the base directory (C:\Users\VSSADM~1\AppData\Local\Temp\tmpxgrr8_ot\info\recipe) will be used.
2022-07-03T12:55:13.8897936Z Reloading output folder: D:\bld
2022-07-03T12:55:14.9536511Z 
2022-07-03T12:55:14.9543318Z ## Package Plan ##
2022-07-03T12:55:14.9543997Z 
2022-07-03T12:55:14.9545544Z   environment location: D:\bld\aesara-suite_1656852298621\_test_env
2022-07-03T12:55:14.9546785Z 
2022-07-03T12:55:14.9564047Z 
2022-07-03T12:55:14.9565237Z The following NEW packages will be INSTALLED:
2022-07-03T12:55:14.9566874Z 
2022-07-03T12:55:14.9567811Z     aesara:                         2.7.4-py37he04ca5b_1       local      
2022-07-03T12:55:14.9568728Z     aesara-base:                    2.7.4-py37h4c0cbd9_1       local      
2022-07-03T12:55:14.9569563Z     blas:                           2.115-mkl                  conda-forge
2022-07-03T12:55:14.9570585Z     blas-devel:                     3.9.0-15_win64_mkl         conda-forge
2022-07-03T12:55:14.9571480Z     bzip2:                          1.0.8-h8ffe710_4           conda-forge
2022-07-03T12:55:14.9572449Z     ca-certificates:                2022.6.15-h5b45459_0       conda-forge
2022-07-03T12:55:14.9573350Z     cons:                           0.4.5-pyhd8ed1ab_0         conda-forge
2022-07-03T12:55:14.9574303Z     etuples:                        0.3.5-pyhd8ed1ab_0         conda-forge
2022-07-03T12:55:14.9575170Z     expat:                          2.4.8-h39d44d4_0           conda-forge
2022-07-03T12:55:14.9576077Z     filelock:                       3.7.1-pyhd8ed1ab_0         conda-forge
2022-07-03T12:55:14.9576859Z     intel-openmp:                   2022.1.0-h57928b3_3787     conda-forge
2022-07-03T12:55:14.9577789Z     libblas:                        3.9.0-15_win64_mkl         conda-forge
2022-07-03T12:55:14.9578687Z     libcblas:                       3.9.0-15_win64_mkl         conda-forge
2022-07-03T12:55:14.9579561Z     libffi:                         3.4.2-h8ffe710_5           conda-forge
2022-07-03T12:55:14.9580424Z     libflang:                       5.0.0-h6538335_20180525    conda-forge
2022-07-03T12:55:14.9581823Z     liblapack:                      3.9.0-15_win64_mkl         conda-forge
2022-07-03T12:55:14.9582745Z     liblapacke:                     3.9.0-15_win64_mkl         conda-forge
2022-07-03T12:55:14.9583590Z     libopenblas:                    0.3.20-pthreads_hc469a61_0 conda-forge
2022-07-03T12:55:14.9584456Z     libpython:                      2.2-py37h4c0cbd9_0         conda-forge
2022-07-03T12:55:14.9585277Z     libzlib:                        1.2.12-h8ffe710_1          conda-forge
2022-07-03T12:55:14.9587760Z     llvm-meta:                      5.0.0-0                    conda-forge
2022-07-03T12:55:14.9588713Z     logical-unification:            0.4.5-pyhd8ed1ab_0         conda-forge
2022-07-03T12:55:14.9589623Z     m2w64-binutils:                 2.25.1-5                   conda-forge
2022-07-03T12:55:14.9590616Z     m2w64-bzip2:                    1.0.6-6                    conda-forge
2022-07-03T12:55:14.9591689Z     m2w64-crt-git:                  5.0.0.4636.2595836-2       conda-forge
2022-07-03T12:55:14.9592549Z     m2w64-gcc:                      5.3.0-6                    conda-forge
2022-07-03T12:55:14.9593453Z     m2w64-gcc-ada:                  5.3.0-6                    conda-forge
2022-07-03T12:55:14.9594378Z     m2w64-gcc-fortran:              5.3.0-6                    conda-forge
2022-07-03T12:55:14.9595233Z     m2w64-gcc-libgfortran:          5.3.0-6                    conda-forge
2022-07-03T12:55:14.9596066Z     m2w64-gcc-libs:                 5.3.0-7                    conda-forge
2022-07-03T12:55:14.9596986Z     m2w64-gcc-libs-core:            5.3.0-7                    conda-forge
2022-07-03T12:55:14.9597879Z     m2w64-gcc-objc:                 5.3.0-6                    conda-forge
2022-07-03T12:55:14.9598771Z     m2w64-gmp:                      6.1.0-2                    conda-forge
2022-07-03T12:55:14.9599905Z     m2w64-headers-git:              5.0.0.4636.c0ad18a-2       conda-forge
2022-07-03T12:55:14.9600743Z     m2w64-isl:                      0.16.1-2                   conda-forge
2022-07-03T12:55:14.9601747Z     m2w64-libiconv:                 1.14-6                     conda-forge
2022-07-03T12:55:14.9602669Z     m2w64-libmangle-git:            5.0.0.4509.2e5a9a2-2       conda-forge
2022-07-03T12:55:14.9603545Z     m2w64-libwinpthread-git:        5.0.0.4634.697f757-2       conda-forge
2022-07-03T12:55:14.9604418Z     m2w64-make:                     4.1.2351.a80a8b8-2         conda-forge
2022-07-03T12:55:14.9605269Z     m2w64-mpc:                      1.0.3-3                    conda-forge
2022-07-03T12:55:14.9606710Z     m2w64-mpfr:                     3.1.4-4                    conda-forge
2022-07-03T12:55:14.9607647Z     m2w64-pkg-config:               0.29.1-2                   conda-forge
2022-07-03T12:55:14.9608436Z     m2w64-toolchain:                5.3.0-7                    conda-forge
2022-07-03T12:55:14.9609392Z     m2w64-toolchain_win-64:         2.4.0-0                    conda-forge
2022-07-03T12:55:14.9610356Z     m2w64-tools-git:                5.0.0.4592.90b8472-2       conda-forge
2022-07-03T12:55:14.9611241Z     m2w64-windows-default-manifest: 6.4-3                      conda-forge
2022-07-03T12:55:15.8154662Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/noarch/llvm-meta-5.0.0-0.tar.bz2 HTTP/1.1" 200 2667
2022-07-03T12:55:15.8155972Z     m2w64-winpthreads-git:          5.0.0.4634.697f757-2       conda-forge
2022-07-03T12:55:15.8157370Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/mkl-include-2022.1.0-h6a75c08_874.tar.bz2 HTTP/1.1" 200 778628
2022-07-03T12:55:15.8158172Z     m2w64-zlib:                     1.2.8-10                   conda-forge
2022-07-03T12:55:15.8159238Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-headers-git-5.0.0.4636.c0ad18a-2.tar.bz2 HTTP/1.1" 200 5854125
2022-07-03T12:55:15.8160254Z     minikanren:                     1.0.3-pyhd8ed1ab_0         conda-forge
2022-07-03T12:55:15.8161227Z     mkl:                            2022.1.0-h6a75c08_874      conda-forge
2022-07-03T12:55:15.8162119Z     mkl-devel:                      2022.1.0-h57928b3_875      conda-forge
2022-07-03T12:55:15.8163218Z     mkl-include:                    2022.1.0-h6a75c08_874      conda-forge
2022-07-03T12:55:15.8164180Z     msys2-conda-epoch:              20160418-1                 conda-forge
2022-07-03T12:55:15.8165081Z     multipledispatch:               0.6.0-py_0                 conda-forge
2022-07-03T12:55:15.8165767Z     numpy:                          1.21.5-py37hb10f931_0      conda-forge
2022-07-03T12:55:15.8166721Z     openblas:                       0.3.20-pthreads_h543f93c_0 conda-forge
2022-07-03T12:55:15.8167447Z     openmp:                         5.0.0-vc14_1               conda-forge
2022-07-03T12:55:15.8168144Z     openssl:                        3.0.4-h8ffe710_2           conda-forge
2022-07-03T12:55:15.8168899Z     pypy3.7:                        7.3.7-h4094faa_3           conda-forge
2022-07-03T12:55:15.8169578Z     python:                         3.7.12-0_73_pypy           conda-forge
2022-07-03T12:55:15.8170367Z     python_abi:                     3.7-2_pypy37_pp73          conda-forge
2022-07-03T12:55:15.8171133Z     scipy:                          1.7.1-py37hddcbd54_0       conda-forge
2022-07-03T12:55:15.8171827Z     setuptools:                     61.2.0-py37h4c0cbd9_0      conda-forge
2022-07-03T12:55:15.8172897Z     six:                            1.16.0-pyh6c4a22f_0        conda-forge
2022-07-03T12:55:15.8173711Z     sqlite:                         3.39.0-h8ffe710_0          conda-forge
2022-07-03T12:55:15.8174576Z     tbb:                            2021.5.0-h2d74725_1        conda-forge
2022-07-03T12:55:15.8175421Z     tk:                             8.6.12-h8ffe710_0          conda-forge
2022-07-03T12:55:15.8176439Z     toolz:                          0.11.2-pyhd8ed1ab_0        conda-forge
2022-07-03T12:55:15.8177330Z     typing_extensions:              4.3.0-pyha770c72_0         conda-forge
2022-07-03T12:55:15.8178205Z     ucrt:                           10.0.20348.0-h57928b3_0    conda-forge
2022-07-03T12:55:15.8179010Z     vc:                             14.2-hb210afc_6            conda-forge
2022-07-03T12:55:15.8179885Z     vs2015_runtime:                 14.29.30037-h902a5da_6     conda-forge
2022-07-03T12:55:15.8180787Z     xz:                             5.2.5-h62dcd97_1           conda-forge
2022-07-03T12:55:15.8183507Z     zlib:                           1.2.12-h8ffe710_1          conda-forge
2022-07-03T12:55:15.8184147Z 
2022-07-03T12:55:19.2470447Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-isl-0.16.1-2.tar.bz2 HTTP/1.1" 200 671047
2022-07-03T12:55:19.4614655Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-libiconv-1.14-6.tar.bz2 HTTP/1.1" 200 1540742
2022-07-03T12:55:19.9173143Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-libmangle-git-5.0.0.4509.2e5a9a2-2.tar.bz2 HTTP/1.1" 200 23402
2022-07-03T12:55:19.9783765Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-make-4.1.2351.a80a8b8-2.tar.bz2 HTTP/1.1" 200 119662
2022-07-03T12:55:20.0592439Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-windows-default-manifest-6.4-3.tar.bz2 HTTP/1.1" 200 4756
2022-07-03T12:55:20.1175807Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-crt-git-5.0.0.4636.2595836-2.tar.bz2 HTTP/1.1" 200 3552995
2022-07-03T12:55:23.2930616Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-mpfr-3.1.4-4.tar.bz2 HTTP/1.1" 200 301114
2022-07-03T12:55:23.4131293Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-pkg-config-0.29.1-2.tar.bz2 HTTP/1.1" 200 477972
2022-07-03T12:55:23.5320963Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-mpc-1.0.3-3.tar.bz2 HTTP/1.1" 200 73142
2022-07-03T12:55:23.5992870Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-winpthreads-git-5.0.0.4634.697f757-2.tar.bz2 HTTP/1.1" 200 47760
2022-07-03T12:55:23.6659666Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/openmp-5.0.0-vc14_1.tar.bz2 HTTP/1.1" 200 590466
2022-07-03T12:55:24.1113302Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/libflang-5.0.0-h6538335_20180525.tar.bz2 HTTP/1.1" 200 531143
2022-07-03T12:55:24.2599149Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/libopenblas-0.3.20-pthreads_hc469a61_0.tar.bz2 HTTP/1.1" 200 5820842
2022-07-03T12:55:25.5267408Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-bzip2-1.0.6-6.tar.bz2 HTTP/1.1" 200 103945
2022-07-03T12:55:25.6086441Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-tools-git-5.0.0.4592.90b8472-2.tar.bz2 HTTP/1.1" 200 327531
2022-07-03T12:55:25.7251811Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/mkl-devel-2022.1.0-h57928b3_875.tar.bz2 HTTP/1.1" 200 7461727
2022-07-03T12:55:28.4210439Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-zlib-1.2.8-10.tar.bz2 HTTP/1.1" 200 203678
2022-07-03T12:55:28.5471027Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/openblas-0.3.20-pthreads_h543f93c_0.tar.bz2 HTTP/1.1" 200 207206
2022-07-03T12:55:28.7426186Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/liblapacke-3.9.0-15_win64_mkl.tar.bz2 HTTP/1.1" 200 5889820
2022-07-03T12:55:29.9927220Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-binutils-2.25.1-5.tar.bz2 HTTP/1.1" 200 46459591
2022-07-03T12:55:36.8072550Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/blas-devel-3.9.0-15_win64_mkl.tar.bz2 HTTP/1.1" 200 13409
2022-07-03T12:55:36.8693612Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-gcc-5.3.0-6.tar.bz2 HTTP/1.1" 200 42735401
2022-07-03T12:55:44.9768914Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/blas-2.115-mkl.tar.bz2 HTTP/1.1" 200 13995
2022-07-03T12:55:45.0585103Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-gcc-ada-5.3.0-6.tar.bz2 HTTP/1.1" 200 34889672
2022-07-03T12:55:52.8566396Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-gcc-fortran-5.3.0-6.tar.bz2 HTTP/1.1" 200 10645206
2022-07-03T12:55:54.4324543Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-gcc-objc-5.3.0-6.tar.bz2 HTTP/1.1" 200 15916440
2022-07-03T12:55:56.7659792Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-toolchain-5.3.0-7.tar.bz2 HTTP/1.1" 200 3423
2022-07-03T12:55:56.9238702Z DEBUG:urllib3.connectionpool:https://conda.anaconda.org:443 "GET /conda-forge/win-64/m2w64-toolchain_win-64-2.4.0-0.tar.bz2 HTTP/1.1" 200 4330
2022-07-03T12:55:59.0482771Z Preparing transaction: ...working... done
2022-07-03T12:56:02.7952487Z 
2022-07-03T12:56:02.7954104Z Verifying transaction: ...working... done
2022-07-03T12:56:02.7955253Z ClobberWarning: This transaction has incompatible packages due to a shared path.
2022-07-03T12:56:02.7956356Z   packages: conda-forge/win-64::intel-openmp-2022.1.0-h57928b3_3787, conda-forge/win-64::openmp-5.0.0-vc14_1
2022-07-03T12:56:02.7957306Z   path: 'library/bin/libiomp5md.dll'
2022-07-03T12:56:02.7957680Z 
2022-07-03T12:56:02.7958218Z 
2022-07-03T12:56:02.7959732Z 
2022-07-03T12:56:20.8558964Z 
2022-07-03T12:56:20.8562922Z ClobberWarning: Conda was asked to clobber an existing path.
2022-07-03T12:56:20.8565646Z   source path: C:\Miniforge\pkgs\openmp-5.0.0-vc14_1\Library\bin\libiomp5md.dll
2022-07-03T12:56:20.8567024Z   target path: D:\bld\aesara-suite_1656852298621\_test_env\Library\bin\libiomp5md.dll
2022-07-03T12:56:20.8567625Z 
2022-07-03T12:56:20.8567906Z 
2022-07-03T12:56:20.8568258Z 
2022-07-03T12:57:41.8892832Z Executing transaction: ...working... done
2022-07-03T12:57:41.9534741Z set PREFIX=D:\bld\aesara-suite_1656852298621\_test_env
2022-07-03T12:57:41.9536696Z set SRC_DIR=D:\bld\aesara-suite_1656852298621\test_tmp
2022-07-03T12:57:41.9706180Z 
2022-07-03T12:57:41.9709387Z (base) %SRC_DIR%>call "%SRC_DIR%\conda_test_env_vars.bat" 
2022-07-03T12:57:41.9722502Z 
2022-07-03T12:57:41.9723825Z (base) %SRC_DIR%>set "CONDA_SHLVL="   && 
2022-07-03T12:57:41.9995122Z 
2022-07-03T12:57:41.9996714Z (base) %SRC_DIR%>conda activate "%PREFIX%" 
2022-07-03T12:57:42.7783376Z 
2022-07-03T12:57:42.7785318Z (%PREFIX%) %SRC_DIR%>IF 0 NEQ 0 exit /B 1 
2022-07-03T12:57:42.7786044Z 
2022-07-03T12:57:42.7832339Z (%PREFIX%) %SRC_DIR%>"%PREFIX%\python.exe" -s "%SRC_DIR%\run_test.py" 
2022-07-03T12:57:42.8300026Z import: 'aesara'
2022-07-03T12:57:46.4692056Z Traceback (most recent call last):
2022-07-03T12:57:46.4693871Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib\site-packages\aesara\configparser.py", line 237, in fetch_val_for_key
2022-07-03T12:57:46.4695160Z     return self._aesara_cfg.get(section, option)
2022-07-03T12:57:46.4696427Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib-python\3\configparser.py", line 780, in get
2022-07-03T12:57:46.4698891Z     d = self._unify_values(section, vars)
2022-07-03T12:57:46.4708952Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib-python\3\configparser.py", line 1146, in _unify_values
2022-07-03T12:57:46.4710350Z     raise NoSectionError(section) from None
2022-07-03T12:57:46.4711595Z configparser.NoSectionError: No section: 'blas'
2022-07-03T12:57:46.4712061Z 
2022-07-03T12:57:46.4712788Z During handling of the above exception, another exception occurred:
2022-07-03T12:57:46.4713279Z 
2022-07-03T12:57:46.4713779Z Traceback (most recent call last):
2022-07-03T12:57:46.4714659Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib\site-packages\aesara\configparser.py", line 353, in __get__
2022-07-03T12:57:46.4715684Z     val_str = cls.fetch_val_for_key(self.name, delete_key=delete_key)
2022-07-03T12:57:46.4716699Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib\site-packages\aesara\configparser.py", line 241, in fetch_val_for_key
2022-07-03T12:57:46.4718499Z     raise KeyError(key)
2022-07-03T12:57:46.4724683Z KeyError: 'blas__ldflags'
2022-07-03T12:57:46.4725219Z 
2022-07-03T12:57:46.4725820Z During handling of the above exception, another exception occurred:
2022-07-03T12:57:46.4727956Z 
2022-07-03T12:57:46.4728738Z Traceback (most recent call last):
2022-07-03T12:57:46.4729645Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib\site-packages\aesara\link\c\cmodule.py", line 2670, in check_mkl_openmp
2022-07-03T12:57:46.4730363Z     import mkl
2022-07-03T12:57:46.4731569Z ModuleNotFoundError: No module named 'mkl'
2022-07-03T12:57:46.4732012Z 
2022-07-03T12:57:46.4732580Z During handling of the above exception, another exception occurred:
2022-07-03T12:57:46.4732950Z 
2022-07-03T12:57:46.4733457Z Traceback (most recent call last):
2022-07-03T12:57:46.4734176Z   File "D:\bld\aesara-suite_1656852298621\test_tmp\run_test.py", line 2, in <module>
2022-07-03T12:57:46.4734865Z     import aesara
2022-07-03T12:57:46.4735666Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib\site-packages\aesara\__init__.py", line 126, in <module>
2022-07-03T12:57:46.4736589Z     from aesara import scalar, tensor
2022-07-03T12:57:46.4748506Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib\site-packages\aesara\tensor\__init__.py", line 98, in <module>
2022-07-03T12:57:46.4749606Z     from aesara.tensor import (  # noqa
2022-07-03T12:57:46.4750504Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib\site-packages\aesara\tensor\blas.py", line 164, in <module>
2022-07-03T12:57:46.4751559Z     from aesara.tensor.blas_headers import blas_header_text, blas_header_version
2022-07-03T12:57:46.4752498Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib\site-packages\aesara\tensor\blas_headers.py", line 1015, in <module>
2022-07-03T12:57:46.4753352Z     if not config.blas__ldflags:
2022-07-03T12:57:46.4754027Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib\site-packages\aesara\configparser.py", line 357, in __get__
2022-07-03T12:57:46.4754998Z     val_str = self.default()
2022-07-03T12:57:46.4755775Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib\site-packages\aesara\link\c\cmodule.py", line 2878, in default_blas_ldflags
2022-07-03T12:57:46.4757411Z     check_mkl_openmp()
2022-07-03T12:57:46.4758279Z   File "D:\bld\aesara-suite_1656852298621\_test_env\lib\site-packages\aesara\link\c\cmodule.py", line 2682, in check_mkl_openmp
2022-07-03T12:57:46.4758986Z     """
2022-07-03T12:57:46.4759416Z RuntimeError: 
2022-07-03T12:57:46.4759958Z Could not import 'mkl'.  If you are using conda, update the numpy
2022-07-03T12:57:46.4797148Z packages to the latest build otherwise, set MKL_THREADING_LAYER=GNU in
2022-07-03T12:57:46.4797877Z your environment for MKL 2018.
2022-07-03T12:57:46.4798227Z 
2022-07-03T12:57:46.4798744Z If you have MKL 2017 install and are not in a conda environment you
2022-07-03T12:57:46.4799465Z can set the Aesara flag blas__check_openmp to False.  Be warned that if
2022-07-03T12:57:46.4800126Z you set this flag and don't set the appropriate environment or make
2022-07-03T12:57:46.4800826Z sure you have the right version you *will* get wrong results.
2022-07-03T12:57:46.4802019Z 
2022-07-03T12:57:46.4935588Z 
2022-07-03T12:57:46.4938214Z (%PREFIX%) %SRC_DIR%>IF 1 NEQ 0 exit /B 1 
2022-07-03T12:57:48.0278925Z Tests failed for aesara-2.7.4-py37he04ca5b_1.tar.bz2 - moving package to D:\bld\broken
2022-07-03T12:57:48.0288346Z WARNING:conda_build.build:Tests failed for aesara-2.7.4-py37he04ca5b_1.tar.bz2 - moving package to D:\bld\broken
2022-07-03T12:57:48.1048451Z TESTS FAILED: aesara-2.7.4-py37he04ca5b_1.tar.bz2
2022-07-03T12:58:01.7718531Z ##[error]Cmd.exe exited with code '1'.

@twiecki
Copy link
Contributor

twiecki commented Jul 3, 2022

On the base or main package?

@maresb
Copy link
Contributor Author

maresb commented Jul 3, 2022

@twiecki, on main. Thanks for catching that, I updated my comment above.

@brandonwillard
Copy link
Member

brandonwillard commented Jul 3, 2022

That exact error is due to a failure in import mkl, and I believe that the importable mkl module is provided by the mkl-service package.

@brandonwillard
Copy link
Member

Looking at the codebase, it doesn't seem like we use the mkl-service package for anything other than two mkl.get_version_string() calls. If there's another way to get the version string, or skip the checks that use those calls altogether, then we should definitely get rid of this implicit mkl-service requirement.

@mfansler
Copy link
Member

I think the blas in run makes sense, given the package isn't itself built with -lblas flags. However, the specifying of a specific BLAS implementation (openblas, mkl-service) is questionable. In particular, OpenBLAS was the worst of all possible choices for Apple Silicon in some NumPy testing. Generally, Conda Forge wants to leave choice of BLAS/LAPACK implementation up to the end-user, not the package maintainer, i.e., blas should be sufficient.

@maresb
Copy link
Contributor Author

maresb commented Dec 13, 2023

Thanks so much @mfansler for the feedback!!!

If we were to remove openblas and mkl-service, do you understand how a default BLAS implementation would be chosen?

Also, from the knowledge base and the BLAS feedstock it seems like the blas package is deprecated. Perhaps we should be using libblas instead?

@maresb
Copy link
Contributor Author

maresb commented Dec 13, 2023

Just opened #135 to test

@mfansler
Copy link
Member

mfansler commented Dec 13, 2023

If we were to remove openblas and mkl-service, do you understand how a default BLAS implementation would be chosen?

In my experience it defaults to mkl on win-64 and openblas otherwise. I'm unclear on whether this is configured somewhere in the channel, is an artifact of build order in the libblas recipe, or something even more arbitrary, like what implementation build was last built.

At least in Conda Forge, the pinnings dictate that libblas in a host requirement will always install the netlib version (a reference interface). Packages can safely link against this generically and the other implementations will have a consistent ABI.

Also, from the knowledge base and the BLAS feedstock it seems like the blas package is deprecated. Perhaps we should be using libblas instead?

I believe you are correct that libblas is the more correct specification. blas is a metapackage provided for end-users to control the implementation. For example, from the linked SO post, it shows how it works:

## Note that `conda-forge` channel is prioritized on my system

# MKL
mamba create -n np_mkl python=3.9 numpy 'blas=*=mkl'

# BLIS
mamba create -n np_blis python=3.9 numpy 'blas=*=blis'

# OpenBLAS
mamba create -n np_openblas python=3.9 numpy 'blas=*=openblas'

# Accelerate
mamba create -n np_accelerate python=3.9 numpy 'blas=*=accelerate'

# Netlib
mamba create -n np_netlib python=3.9 numpy 'blas=*=netlib'

However, I'm beginning to grok a bit more the motivating problems involved here. The workaround of including blas was about needing to find include/cblas.h. I did a few checks on include/cblas.h in the different implementations and see that openblas provides it, but blis puts it in include/blis/cblas.h and accelerate doesn't even package one (though maybe you could pick it up from the system SDK?).

Further, mkl-service alone installs mkl, but does not include include/mkl_cblas.h (its version of this) which is instead provided by mkl-include.

A bit of a mess. It seems like

  1. We need some unification done to ensure that all the BLAS implementations of libblas provide some form of cblas.h that is findable.
  2. Work out some robust way for aesera to find this (e.g., via pkg-config?).

@brandonwillard
Copy link
Member

@mfansler, many thanks for these insights!

@mfansler
Copy link
Member

Oh! - I just noticed that mkl-service + blas specification on osx-64 installs MKL, but still resolves to blas=*=openblas, i.e., it installs OpenBLAS which is what ends up providing an include/cblas.h.

@mfansler
Copy link
Member

I created an Issue in the blas feedstock. Please amend it if my explanation there is not correctly characterizing the needs here. My understanding is that this package requires an include/cblas.h and Conda Forge's BLAS implementations lack a consistent way to express this precisely, which is the source of confusion.

conda-forge/blas-feedstock#111

@brandonwillard
Copy link
Member

I created an Issue in the blas feedstock. Please amend it if my explanation there is not correctly characterizing the needs here. My understanding is that this package requires an include/cblas.h and Conda Forge's BLAS implementations lack a consistent way to express this precisely, which is the source of confusion.

conda-forge/blas-feedstock#111

That sounds correct, and thanks again!

@mfansler
Copy link
Member

Last note for the night is that Netlib delivers this with the Netlib implementation of libcblas. Since Netlib is the reference for BLAS, that suggests to me that really libcblas, not libblas, would be the proper dependency, or perhaps even that we need to create a new output package like libcblas-devel as a dependency that would be dedicated to packing the associated header files.

@maresb
Copy link
Contributor Author

maresb commented Dec 14, 2023

What's going on between the blas package's headers and https://github.com/aesara-devs/aesara/blob/main/aesara/tensor/blas_headers.py?

@brandonwillard
Copy link
Member

brandonwillard commented Dec 14, 2023

What's going on between the blas package's headers and https://github.com/aesara-devs/aesara/blob/main/aesara/tensor/blas_headers.py?

If I recall correctly, some code uses the custom headers created there and some other code uses the externally provided headers.

It was/is a sort of mess.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

4 participants