cygwin: Try again to use system BLAS and system R

[mkoeppe]

... they were disabled in spkg-configure for 9.1 in #29538 ...

Previous system BLAS tickets:

• numpy: site.cfg needs a [DEFAULT] section #29025, Repair use of system BLAS broken by #29025: remove numpy site.cfg [DEFAULT/ALL] sections #29051, Accept system (Open)BLAS #29398

Depends on #30157
Depends on #30149
Depends on #29441
Depends on #30127
Depends on #30277

CC: @embray @slel

Component: porting: Cygwin

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/cygwin__try_again_to_use_system_blas_and_system_r @ 7f8ede4

Reviewer: github.com/mkoeppe/sage/actions/runs/207840100, github.com/mkoeppe/sage/actions/runs/207840103

Issue created by migration from https://trac.sagemath.org/ticket/30163

[mkoeppe]

Changed dependencies from #30157, #30149, #29441 to #30157, #30149, #29441, #30127, #30277

[mkoeppe]

Branch: u/mkoeppe/cygwin__try_again_to_use_system_blas_and_system_r

[mkoeppe]

Reviewer: github.com/mkoeppe/sage/actions/runs/207840100, github.com/mkoeppe/sage/actions/runs/207840103

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Last 10 new commits:

b8e366e	build/pkgs/sagelib/spkg-src: chmod +x
0473ef3	Merge branch 't/21559/change-src-bin-installation' into t/29950/build_sagelib_using_installed_sage_setup
7244371	Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup
4344f89	Merge branch 't/21559/change-src-bin-installation' into t/29950/build_sagelib_using_installed_sage_setup
01b96b0	Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup
2818739	Merge branch 't/29950/build_sagelib_using_installed_sage_setup' into t/30277/remove_src_module_list_py
8a19fe2	build/make/Makefile.in (sagelib-clean): Clean the new build location
ccc67b0	src/sage_setup: Update cythonized_dir in doctests
df38027	Merge branch 't/29950/build_sagelib_using_installed_sage_setup' into t/30277/remove_src_module_list_py
7f8ede4	Merge branch 't/30277/remove_src_module_list_py' into t/30163/cygwin__try_again_to_use_system_blas_and_system_r

[mkoeppe]

Commit: 7f8ede4

[mkoeppe]

comment:4

cygwin-standard (https://github.com/mkoeppe/sage/runs/983346777?check_suite_focus=true) maplotlib build fails:

      File "/tmp/pip-req-build-ixb64a6o/setupext.py", line 461, in add_numpy_flags
        import numpy as np
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/__init__.py", line 143, in <module>
        from . import lib
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/lib/__init__.py", line 25, in <module>
        from .index_tricks import *
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/lib/index_tricks.py", line 11, in <module>
        import numpy.matrixlib as matrixlib
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/matrixlib/__init__.py", line 4, in <module>
        from .defmatrix import *
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/matrixlib/defmatrix.py", line 11, in <module>
        from numpy.linalg import matrix_power
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/linalg/__init__.py", line 73, in <module>
        from .linalg import *
      File "/cygdrive/d/a/sage/sage/local/lib/python3.7/site-packages/numpy/linalg/linalg.py", line 33, in <module>
        from numpy.linalg import lapack_lite, _umath_linalg
    ImportError: No such file or directory

also scipy fails with the same error.

r system package is not used because prereq curl is not accepted:

Checking whether SageMath should install SPKG curl...
checking for curl 7.22... /usr/bin/curl
checking for gawk... (cached) gawk
checking for curl-config... /usr/bin/curl-config
checking for the version of libcurl... 7.71.1
checking for libcurl >= version 7.22... yes
checking whether libcurl is usable... no
checking whether programs linking to libcurl can be executed... no
configure: no suitable system package found for SPKG curl

and then building r also fails:

Warning in solve.default(rgb) :
  unable to load shared object '/cygdrive/d/a/sage/sage/local/var/tmp/sage/build/r-3.6.3/src/modules//lapack.dll':
  No such file or directory
Error in solve.default(rgb) : LAPACK routines cannot be loaded
Error: unable to load R code in package 'grDevices'
Execution halted
make[7]: *** [../../../share/make/lazycomp.mk:9: ../../../library/grDevices/R/grDevices.rdb] Error 1
make[7]: Target 'mklazycomp' not remade because of errors.
make[7]: Leaving directory '/cygdrive/d/a/sage/sage/local/var/tmp/sage/build/r-3.6.3/src/src/library/grDevices'
make[6]: *** [Makefile:28: all] Error 2
make[6]: Leaving directory '/cygdrive/d/a/sage/sage/local/var/tmp/sage/build/r-3.6.3/src/src/library/grDevices'
make[5]: *** [Makefile:39: R] Error 1
make[5]: Leaving directory '/cygdrive/d/a/sage/sage/local/var/tmp/sage/build/r-3.6.3/src/src/library'

[mkoeppe]

comment:5

Logs: https://github.com/mkoeppe/sage/suites/1048727679/artifacts/14102351

[mkoeppe]

comment:6

From the numpy build log:

blas_opt_info:
blas_mkl_info:
Disabled blas_mkl_info: (MKLROOT is None)
Disabled blas_mkl_info: (MKLROOT is None)
customize UnixCCompiler
  libraries mkl_rt not found in []
  NOT AVAILABLE

blis_info:
  libraries blis not found in ['/cygdrive/d/a/sage/sage/local/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

openblas_info:
Disabled openblas_info: (OPENBLAS is None)
  libraries openblas not found in []
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Disabled atlas_3_10_blas_threads_info: (PTATLAS is None)
  libraries tatlas not found in []
  NOT AVAILABLE

atlas_3_10_blas_info:
Disabled atlas_3_10_blas_info: (ATLAS is None)
  libraries satlas not found in []
  NOT AVAILABLE

atlas_blas_threads_info:
Disabled atlas_blas_threads_info: (PTATLAS is None)
  libraries ptf77blas,ptcblas,atlas not found in []
  NOT AVAILABLE

atlas_blas_info:
Disabled atlas_blas_info: (ATLAS is None)
  libraries f77blas,cblas,atlas not found in []
  NOT AVAILABLE

accelerate_info:
  NOT AVAILABLE

blas_info:
C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -fdebug-prefix-map=/cygdrive/d/cyg_pub/devel/python/python37/python37-3.7.7-1.x86_64/build=/usr/src/debug/python37-3.7.7-1 -fdebug-prefix-map=/cygdrive/d/cyg_pub/devel/python/python37/python37-3.7.7-1.x86_64/src/Python-3.7.7=/usr/src/debug/python37-3.7.7-1 -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -fdebug-prefix-map=/cygdrive/d/cyg_pub/devel/python/python37/python37-3.7.7-1.x86_64/build=/usr/src/debug/python37-3.7.7-1 -fdebug-prefix-map=/cygdrive/d/cyg_pub/devel/python/python37/python37-3.7.7-1.x86_64/src/Python-3.7.7=/usr/src/debug/python37-3.7.7-1

creating /tmp/tmp7np5_kr4/tmp
creating /tmp/tmp7np5_kr4/tmp/tmp7np5_kr4
compile options: '-I/usr/include -I/usr/include/suitesparse -I/cygdrive/d/a/sage/sage/local/include -c'
gcc: /tmp/tmp7np5_kr4/source.c
gcc /tmp/tmp7np5_kr4/tmp/tmp7np5_kr4/source.o -L/usr/lib -lblas -llapack -o /tmp/tmp7np5_kr4/a.out.exe
  FOUND:
    libraries = ['blas', 'lapack', 'blas']
    library_dirs = ['/usr/lib']
    include_dirs = ['/usr/include', '/usr/include/suitesparse', '/cygdrive/d/a/sage/sage/local/include']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

  FOUND:
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    libraries = ['blas', 'lapack', 'blas']
    library_dirs = ['/usr/lib']
    include_dirs = ['/usr/include', '/usr/include/suitesparse', '/cygdrive/d/a/sage/sage/local/include']
    language = c

lapack_opt_info:
lapack_mkl_info:
Disabled lapack_mkl_info: (MKLROOT is None)
Disabled lapack_mkl_info: (MKLROOT is None)
  libraries mkl_rt not found in []
  NOT AVAILABLE

openblas_lapack_info:
Disabled openblas_lapack_info: (OPENBLAS is None)
  libraries openblas not found in []
  NOT AVAILABLE

openblas_clapack_info:
Disabled openblas_clapack_info: (OPENBLAS is None)
  libraries openblas,lapack not found in []
  NOT AVAILABLE

flame_info:
  libraries flame not found in ['/cygdrive/d/a/sage/sage/local/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

atlas_3_10_threads_info:
Disabled atlas_3_10_threads_info: (PTATLAS is None)
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
  NOT AVAILABLE

atlas_3_10_info:
Disabled atlas_3_10_info: (ATLAS is None)
<class 'numpy.distutils.system_info.atlas_3_10_info'>
  NOT AVAILABLE

atlas_threads_info:
Disabled atlas_threads_info: (PTATLAS is None)
<class 'numpy.distutils.system_info.atlas_threads_info'>
  NOT AVAILABLE

atlas_info:
Disabled atlas_info: (ATLAS is None)
<class 'numpy.distutils.system_info.atlas_info'>
  NOT AVAILABLE

lapack_info:
  FOUND:
    libraries = ['blas', 'lapack', 'lapack']
    library_dirs = ['/usr/lib']
    language = f77

  FOUND:
    libraries = ['blas', 'lapack', 'lapack', 'blas', 'lapack', 'blas']
    library_dirs = ['/usr/lib']
    language = c
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    include_dirs = ['/usr/include', '/usr/include/suitesparse', '/cygdrive/d/a/sage/sage/local/include']

.....





/cygdrive/d/a/sage/sage/local/var/tmp/sage/build/numpy-1.19.1/src/numpy/distutils/system_info.py:1914: UserWarning: 
    Optimized (vendor) Blas libraries are not found.
    Falls back to netlib Blas library which has worse performance.
    A better performance should be easily gained by switching
    Blas library.
  if self._calc_info(blas):

[mkoeppe]

comment:9

Hi Erik, do you have time to look into this ticket?

[embray]

comment:10

This looks like a good start so far and your analysis of how to detect openblas makes sense. I'll look into it when I get back to my Windows machine.

[mkoeppe]

comment:11

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[embray]

comment:12

It looks like one of the first things that needs to be fixed is properly detecting the system curl on Cygwin. I remember having trouble with this before. I thought I looked into it at one point, and I think even fixed it somehow, but now I don't remember for sure, or why the fix hasn't made it into Sage if indeed I did fix it...

[mkoeppe]

comment:13

No, libcurl is detected correctly, see https://github.com/mkoeppe/sage/runs/2070658581

[embray]

comment:14

Ok, great. I just mentioned it since one of your earlier comments on this thread said as much. But that was also several months ago. I could have sworn it was working now.