Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backport 3 PRs for CBLAS compat #233

Merged
merged 1 commit into from
May 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion recipe/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,14 @@ source:
- patches/0001-BLD-use-a-relative-path-to-numpy-pybind11-pythran-in.patch
# backport https://github.com/scipy/scipy/pull/18034
- patches/0002-BLD-avoid-running-run_command-py3-.-for-better-cross.patch
# backport https://github.com/scipy/scipy/pull/18263
- patches/0003-BUG-some-tweaks-to-PROPACK-f2py-wrapper-and-build-fl.patch # [not win]
# backport https://github.com/scipy/scipy/pull/18264 (3 commits)
- patches/0004-MAINT-remove-from-numpy.math-cimport-usages.patch # [not win]
- patches/0005-MAINT-sync-changes-to-npy_blas.h-and-npy_blas_base.h.patch # [not win]
- patches/0006-BUG-include-npy_common.h-in-npy_cblas.h.patch # [not win]
# backport https://github.com/scipy/scipy/pull/18426
- patches/0007-BLD-fix-two-regressions-for-Duse-g77-abi.patch # [not win]
# https://github.com/scipy/scipy/tree/v{{ version }}/scipy/_lib
- git_url: https://github.com/scipy/boost-headers-only.git
git_rev: 3af99e6d566043072e95bc882d32c9c26f37e0ba
Expand All @@ -35,7 +43,7 @@ source:
folder: scipy/sparse/linalg/_propack/PROPACK

build:
number: 0
number: 1

requirements:
build:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 66ccdf8520ec803cfd00371833170008f3b0f666 Mon Sep 17 00:00:00 2001
From: Ralf Gommers <ralf.gommers@gmail.com>
Date: Sun, 19 Feb 2023 22:13:39 +0000
Subject: [PATCH 1/2] BLD: use a relative path to numpy/pybind11/pythran
Subject: [PATCH 1/7] BLD: use a relative path to numpy/pybind11/pythran
include and lib dirs

This should make things work with in-tree virtual envs, as well as some
Expand Down Expand Up @@ -1175,6 +1175,3 @@ index 0fca32054..828ccbe5c 100644
link_args: version_link_args,
install: true,
subdir: 'scipy/stats'
--
2.38.1.windows.1

Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 60c26eccbd97de83d0cba143146365527667d0f8 Mon Sep 17 00:00:00 2001
From: Ralf Gommers <ralf.gommers@gmail.com>
Date: Thu, 23 Feb 2023 14:43:52 +0000
Subject: [PATCH 2/2] BLD: avoid running `run_command(py3, ...)`, for better
Subject: [PATCH 2/7] BLD: avoid running `run_command(py3, ...)`, for better
cross-compiling

Adding the ability to specify the paths to the numpy, pybind11 and
Expand Down Expand Up @@ -222,6 +222,3 @@ index a7a9ba7ae..32f909e47 100644
conf_data.set('PYTHRAN_INCDIR', incdir_pythran)
endif

--
2.38.1.windows.1

Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
From c1ed688ccb326fe8b95a33d55f8c4a3851343236 Mon Sep 17 00:00:00 2001
From: Ralf Gommers <ralf.gommers@gmail.com>
Date: Sat, 8 Apr 2023 20:11:36 +0100
Subject: [PATCH 3/7] BUG: some tweaks to PROPACK f2py wrapper and build flags

Still far from fixed, but it should be closer - and prepares
for removal of `setup.py` and re-introduction of Accelerate.
---
scipy/meson.build | 4 +++-
scipy/sparse/linalg/_propack/meson.build | 13 ++++++++++---
scipy/sparse/linalg/_propack/zpropack.pyf | 2 +-
3 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/scipy/meson.build b/scipy/meson.build
index 32f909e47..6de5aea26 100644
--- a/scipy/meson.build
+++ b/scipy/meson.build
@@ -199,7 +199,9 @@ dependency_map = {
# requesting `mkl-dynamic-lp64-seq` here. And then there's work needed
# in general to enable the ILP64 interface (also for OpenBLAS).
uses_mkl = blas.name().to_lower().startswith('mkl') or lapack.name().to_lower().startswith('mkl')
-if uses_mkl or get_option('use-g77-abi')
+uses_accelerate = blas.name().to_lower().startswith('accelerate') or lapack.name().to_lower().startswith('accelerate')
+use_g77_abi = uses_mkl or uses_accelerate or get_option('use-g77-abi')
+if use_g77_abi
g77_abi_wrappers = files([
'_build_utils/src/wrap_g77_abi_f.f',
'_build_utils/src/wrap_g77_abi_c.c'
diff --git a/scipy/sparse/linalg/_propack/meson.build b/scipy/sparse/linalg/_propack/meson.build
index fb8d99c66..70e098241 100644
--- a/scipy/sparse/linalg/_propack/meson.build
+++ b/scipy/sparse/linalg/_propack/meson.build
@@ -85,12 +85,19 @@ elements = [
['_zpropack', z_src, 'zpropack.pyf']
]

+cargs_propack = ['-D_OPENMP'] # FIXME: _OPENMP is needed now, but not good!
+if use_g77_abi
+ # This define needs to be removed from PROPACK code, in favor of using
+ # `wcdotc` unconditionally, like is done in ARPACK.
+ cargs_propack += ['-DSCIPY_USE_G77_CDOTC_WRAP=1']
+endif
foreach ele: elements
- # FIXME: this doesn't match `setup.py` for g77 ABI issue. That is pretty much
- # broken anyway, see for example gh-15108.
+ # PROPACK integration is pretty much broken, see for example gh-15108.
+ # Note that in setup.py, adding `g77_abi_wrappers` is skipped for 32-bit
+ # architectures (stated reason: "it blows up").
propack_lib = static_library('lib_' + ele[0],
[ele[1], g77_abi_wrappers],
- c_args: ['-D_OPENMP'], # FIXME: this is needed now, but not good!
+ c_args: cargs_propack,
fortran_args: [
fortran_ignore_warnings,
_fflag_Wno_intrinsic_shadow,
diff --git a/scipy/sparse/linalg/_propack/zpropack.pyf b/scipy/sparse/linalg/_propack/zpropack.pyf
index 929098b3e..81f418099 100644
--- a/scipy/sparse/linalg/_propack/zpropack.pyf
+++ b/scipy/sparse/linalg/_propack/zpropack.pyf
@@ -10,7 +10,7 @@ python module __user__routines
complex*16 intent(in,out),dimension((transa[0] == 'n' ? m : n)) :: y
integer dimension(*) :: iparm
complex*16 dimension(*) :: zparm
- end function caprod
+ end function zaprod
end interface
end python module __user__routines

Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
From ad526ed010333ffcac566a5a6bc43589fd389e82 Mon Sep 17 00:00:00 2001
From: Ralf Gommers <ralf.gommers@gmail.com>
Date: Sat, 8 Apr 2023 22:45:37 +0100
Subject: [PATCH 4/7] MAINT: remove `from numpy.math cimport` usages

We'd like to get rid of `libnpymath`, and the (unrelated)
`numpy.math` in the Python API was also just deprecated.
So best to not use `numpy.math` from Cython either.

[skip circle]
---
scipy/sparse/csgraph/_matching.pyx | 2 +-
scipy/sparse/csgraph/_shortest_path.pyx | 2 +-
scipy/spatial/_ckdtree.pyx | 3 +--
scipy/spatial/transform/_rotation.pyx | 3 +--
scipy/special/_ellip_harm.pxd | 3 +--
scipy/special/_hyp0f1.pxd | 3 +--
scipy/stats/_stats.pyx | 5 ++---
7 files changed, 8 insertions(+), 13 deletions(-)

diff --git a/scipy/sparse/csgraph/_matching.pyx b/scipy/sparse/csgraph/_matching.pyx
index a92e8a91e..e1ed15d77 100644
--- a/scipy/sparse/csgraph/_matching.pyx
+++ b/scipy/sparse/csgraph/_matching.pyx
@@ -3,7 +3,7 @@ import warnings
cimport cython
import numpy as np
cimport numpy as np
-from numpy.math cimport INFINITY
+from libc.math cimport INFINITY


from scipy.sparse import isspmatrix_coo, isspmatrix_csc, isspmatrix_csr
diff --git a/scipy/sparse/csgraph/_shortest_path.pyx b/scipy/sparse/csgraph/_shortest_path.pyx
index 459672ca2..680b4d4a7 100644
--- a/scipy/sparse/csgraph/_shortest_path.pyx
+++ b/scipy/sparse/csgraph/_shortest_path.pyx
@@ -20,7 +20,7 @@ from scipy.sparse.csgraph._validation import validate_graph
cimport cython

from libc.stdlib cimport malloc, free
-from numpy.math cimport INFINITY
+from libc.math cimport INFINITY

np.import_array()

diff --git a/scipy/spatial/_ckdtree.pyx b/scipy/spatial/_ckdtree.pyx
index aaa641eee..eadfdef6f 100644
--- a/scipy/spatial/_ckdtree.pyx
+++ b/scipy/spatial/_ckdtree.pyx
@@ -10,12 +10,11 @@ import numpy as np
import scipy.sparse

cimport numpy as np
-from numpy.math cimport INFINITY

from cpython.mem cimport PyMem_Malloc, PyMem_Free
from libcpp.vector cimport vector
from libcpp cimport bool
-from libc.math cimport isinf
+from libc.math cimport isinf, INFINITY

cimport cython
import os
diff --git a/scipy/spatial/transform/_rotation.pyx b/scipy/spatial/transform/_rotation.pyx
index d97a9b0cd..86ae595e8 100644
--- a/scipy/spatial/transform/_rotation.pyx
+++ b/scipy/spatial/transform/_rotation.pyx
@@ -7,8 +7,7 @@ from ._rotation_groups import create_group
cimport numpy as np
cimport cython
from cython.view cimport array
-from libc.math cimport sqrt, sin, cos, atan2, acos
-from numpy.math cimport PI as pi, NAN, isnan # avoid MSVC error
+from libc.math cimport sqrt, sin, cos, atan2, acos, hypot, isnan, NAN, pi

np.import_array()

diff --git a/scipy/special/_ellip_harm.pxd b/scipy/special/_ellip_harm.pxd
index 4f83d05c3..431c6a017 100644
--- a/scipy/special/_ellip_harm.pxd
+++ b/scipy/special/_ellip_harm.pxd
@@ -32,9 +32,8 @@ import cython

from . cimport sf_error

-from libc.math cimport sqrt, fabs, pow
+from libc.math cimport sqrt, fabs, pow, NAN
from libc.stdlib cimport malloc, free
-from numpy.math cimport NAN, PI

cdef extern from "lapack_defs.h":
ctypedef int CBLAS_INT # actual type defined in the header
diff --git a/scipy/special/_hyp0f1.pxd b/scipy/special/_hyp0f1.pxd
index baacca8cf..54d7374e5 100644
--- a/scipy/special/_hyp0f1.pxd
+++ b/scipy/special/_hyp0f1.pxd
@@ -1,5 +1,4 @@
-from libc.math cimport pow, sqrt, floor, log, log1p, exp, M_PI, fabs
-from numpy.math cimport NAN, isinf
+from libc.math cimport pow, sqrt, floor, log, log1p, exp, M_PI, NAN, fabs, isinf
cimport numpy as np

from ._cephes cimport iv, jv, Gamma, lgam, gammasgn
diff --git a/scipy/stats/_stats.pyx b/scipy/stats/_stats.pyx
index b8496b1f3..94c5f060d 100644
--- a/scipy/stats/_stats.pyx
+++ b/scipy/stats/_stats.pyx
@@ -1,11 +1,10 @@
from cpython cimport bool
from libc cimport math
+from libc.math cimport NAN, INFINITY, M_PI as PI
cimport cython
cimport numpy as np
-from numpy.math cimport PI
-from numpy.math cimport INFINITY
-from numpy.math cimport NAN
from numpy cimport ndarray, int64_t, float64_t, intp_t
+
import warnings
import numpy as np
import scipy.stats, scipy.special
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
From e16518a0a04db6b04be7550d2d243d76969edd6d Mon Sep 17 00:00:00 2001
From: Ralf Gommers <ralf.gommers@gmail.com>
Date: Sun, 9 Apr 2023 12:46:04 +0100
Subject: [PATCH 5/7] MAINT: sync changes to `npy_blas.h` and `npy_blas_base.h`
from numpy

---
scipy/_build_utils/src/npy_cblas.h | 41 +++++++++++++++++++++++--
scipy/_build_utils/src/npy_cblas_base.h | 5 +++
2 files changed, 43 insertions(+), 3 deletions(-)

diff --git a/scipy/_build_utils/src/npy_cblas.h b/scipy/_build_utils/src/npy_cblas.h
index 97308238a..751854b6e 100644
--- a/scipy/_build_utils/src/npy_cblas.h
+++ b/scipy/_build_utils/src/npy_cblas.h
@@ -3,8 +3,8 @@
* because not all providers of cblas provide cblas.h. For instance, MKL provides
* mkl_cblas.h and also typedefs the CBLAS_XXX enums.
*/
-#ifndef _NPY_CBLAS_H_
-#define _NPY_CBLAS_H_
+#ifndef NUMPY_CORE_SRC_COMMON_NPY_CBLAS_H_
+#define NUMPY_CORE_SRC_COMMON_NPY_CBLAS_H_

#include <stddef.h>

@@ -47,8 +47,10 @@ enum CBLAS_SIDE {CblasLeft=141, CblasRight=142};

#ifdef HAVE_BLAS_ILP64
#define CBLAS_INT npy_int64
+#define CBLAS_INT_MAX NPY_MAX_INT64
#else
#define CBLAS_INT int
+#define CBLAS_INT_MAX INT_MAX
#endif

#define BLASNAME(name) CBLAS_FUNC(name)
@@ -59,8 +61,41 @@ enum CBLAS_SIDE {CblasLeft=141, CblasRight=142};
#undef BLASINT
#undef BLASNAME

-#ifdef __cplusplus
+
+/*
+ * Convert NumPy stride to BLAS stride. Returns 0 if conversion cannot be done
+ * (BLAS won't handle negative or zero strides the way we want).
+ */
+static inline CBLAS_INT
+blas_stride(npy_intp stride, unsigned itemsize)
+{
+ /*
+ * Should probably check pointer alignment also, but this may cause
+ * problems if we require complex to be 16 byte aligned.
+ */
+ if (stride > 0 && (stride % itemsize) == 0) {
+ stride /= itemsize;
+ if (stride <= CBLAS_INT_MAX) {
+ return stride;
+ }
+ }
+ return 0;
}
+
+/*
+ * Define a chunksize for CBLAS.
+ *
+ * The chunksize is the greatest power of two less than CBLAS_INT_MAX.
+ */
+#if NPY_MAX_INTP > CBLAS_INT_MAX
+# define NPY_CBLAS_CHUNK (CBLAS_INT_MAX / 2 + 1)
+#else
+# define NPY_CBLAS_CHUNK NPY_MAX_INTP
#endif

+
+#ifdef __cplusplus
+}
#endif
+
+#endif /* NUMPY_CORE_SRC_COMMON_NPY_CBLAS_H_ */
diff --git a/scipy/_build_utils/src/npy_cblas_base.h b/scipy/_build_utils/src/npy_cblas_base.h
index 792b6f09e..12dfb2e78 100644
--- a/scipy/_build_utils/src/npy_cblas_base.h
+++ b/scipy/_build_utils/src/npy_cblas_base.h
@@ -9,6 +9,9 @@
* Prototypes for level 1 BLAS functions (complex are recast as routines)
* ===========================================================================
*/
+#ifndef NUMPY_CORE_SRC_COMMON_NPY_CBLAS_BASE_H_
+#define NUMPY_CORE_SRC_COMMON_NPY_CBLAS_BASE_H_
+
float BLASNAME(cblas_sdsdot)(const BLASINT N, const float alpha, const float *X,
const BLASINT incX, const float *Y, const BLASINT incY);
double BLASNAME(cblas_dsdot)(const BLASINT N, const float *X, const BLASINT incX, const float *Y,
@@ -555,3 +558,5 @@ void BLASNAME(cblas_zher2k)(const enum CBLAS_ORDER Order, const enum CBLAS_UPLO
void *C, const BLASINT ldc);

void BLASNAME(cblas_xerbla)(BLASINT p, const char *rout, const char *form, ...);
+
+#endif /* NUMPY_CORE_SRC_COMMON_NPY_CBLAS_BASE_H_ */
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
From 578b57b0f40da9d4aa07f21445aa9e19d00a4f77 Mon Sep 17 00:00:00 2001
From: Ralf Gommers <ralf.gommers@gmail.com>
Date: Sun, 9 Apr 2023 13:00:51 +0100
Subject: [PATCH 6/7] BUG: include `npy_common.h` in `npy_cblas.h`

This is necessary, because it uses `npy_int64` & co. The implicit
assumption previously was that this was made available in the
file that included `npy_cblas.h`, but this wasn't guaranteed in
any way. This is why the build failed after removing a
`from numpy.math cimport` line in `_ellip_harm.pxd`.
---
scipy/_build_utils/src/npy_cblas.h | 1 +
1 file changed, 1 insertion(+)

diff --git a/scipy/_build_utils/src/npy_cblas.h b/scipy/_build_utils/src/npy_cblas.h
index 751854b6e..02961d430 100644
--- a/scipy/_build_utils/src/npy_cblas.h
+++ b/scipy/_build_utils/src/npy_cblas.h
@@ -6,6 +6,7 @@
#ifndef NUMPY_CORE_SRC_COMMON_NPY_CBLAS_H_
#define NUMPY_CORE_SRC_COMMON_NPY_CBLAS_H_

+#include <numpy/npy_common.h>
#include <stddef.h>

/* Allow the use in C++ code. */
Loading