[EXTENDED] PR 26 Parallelized multiple right-hand side support, fully typed tools
#27
Conversation
… test that test more different cases
…proved import chain; improved code readability
… cover the edge cases
… f-string; fixed typo
…hon level; fixed typos
…utput and warning behaviour; tested it altogether
…tate development without compromising on build
MothNik
changed the title
toolstools
MothNik
changed the title
toolstools
… fucking comment; augmented tests to cover this error
|
The last commit currently does not change the internal logic at all. The idea was taken from LAPACK. It returns an |
…ers I and II: - now, they share one common processing logic that - calls the solvers it needs depending on which solver was called Besides, a shape check was introduced and the `info`-variable was extended. All of this is now reflected in `core`, where the solvers and errors are not structured a bit better.
… auxiliary functions in `core`; added NumPy-solver error case test
|
To be able to cope with #28 and #29 in the future without compromising anything on the solvers side, I rearranged the Cython core (not the interface) as follows:
This is probably a bit too abstract (I barely know how phrase it), so please let me provide the following As of now, this PR basically implements the |
|
@MuellerSeb Sorry for all these iterations. It tooks me some time to get the right strategy for Cython, the GIL, Memory Allocation, the Parallelization, and a testing framework (they all pass still ✅). |
…trix Reason: it is the main diagonal of the matrix L for A = LU just like ps is the main diagonal of U for PTRANS-II. So now, the main diagonals of the non unit triangular factors are always in the central column which makes the most sense.
|
|
||
| - fully parallelized the Cython implementation of PTRANS-I and PTRANS-II for single and multiple right-hand sides support that can now be enabled via the new ``workers`` parameter in ``pentapy.solve`` (default: 1) | ||
| - fully typed the ``pentapy.tools`` module | ||
| - updated the **Cython low level interfaces** to PTRANS-I and PTRANS-II to **only accept C-contiguous arrays** (not backwards compatible) |
There was a problem hiding this comment.
This is the only reason for the major version jump, right? Do you want an intermediate 1.4 release? If not, I would vote for merging the changes of 1.4 and 2.0 here.
| @@ -107,7 +107,8 @@ Have a look at the script: [``examples/03_perform_simple.py``](https://github.co | |||
|
|
|||
| ## Requirements: | |||
|
|
|||
| - [NumPy >= 1.14.5](https://www.numpy.org) | |||
| - [NumPy >= 1.20.0](https://www.numpy.org) | |||
| - [psutil >= 5.8.0](https://psutil.readthedocs.io/en/latest/) (for parallelisation) | |||
| @@ -94,7 +94,8 @@ The performance plot was created with ``perfplot`` (`link <https://github.com/ns | |||
| Requirements | |||
| ============ | |||
|
|
|||
| - `Numpy >= 1.14.5 <http://www.numpy.org>`_ | |||
| - `Numpy >= 1.20.0 <http://www.numpy.org>`_ | |||
| - `psutil >= 5.8.0 <https://psutil.readthedocs.io/en/latest/>`_ | |||
| @@ -0,0 +1,2 @@ | |||
| [pytest] | |||
| addopts = --doctest-modules | |||
There was a problem hiding this comment.
Can't this file content be part of pyproject.toml?
| [project.optional-dependencies] | ||
| scipy = ["scipy"] | ||
| umfpack = ["scikit-umfpack"] | ||
| all = [ | ||
| "scipy", | ||
| "scikit-umfpack", | ||
| ] | ||
| doc = [ | ||
| "m2r2>=0.2.8", | ||
| "scipy>=1.1.0", | ||
| "matplotlib>=3", | ||
| "perfplot<0.9", | ||
| "numpydoc>=1.1", | ||
| "sphinx>=7", | ||
| "sphinx-gallery>=0.8", | ||
| "sphinx-rtd-theme>=2", | ||
| ] | ||
| test = ["pytest-cov>=3"] | ||
| check = [ | ||
| "black>=24,<25", | ||
| "isort[colors]", | ||
| "pylint", | ||
| "cython-lint", | ||
| ] | ||
| [tool.setuptools.dynamic] | ||
| dependencies = {file = ["requirements/base.txt"]} | ||
| optional-dependencies = {scipy = {file = ["requirements/scipy.txt"]}, umfpack = {file = ["requirements/umfpack.txt"]}, all = {file = ["requirements/all.txt"]}, doc = {file = ["requirements/doc.txt"]}, test = {file = ["requirements/test.txt"]}, check = {file = ["requirements/check.txt"]}} | ||
|
|
There was a problem hiding this comment.
I have to admit, I am not a fan of these req files. I would like to use my veto here to have the deps in the pyproject.toml file.
| if sys.platform.startswith("win"): | ||
| openmp_arg = "/openmp" | ||
| else: | ||
| openmp_arg = "-fopenmp" | ||
|
|
||
| Cython.Compiler.Options.annotate = True |
There was a problem hiding this comment.
Please use https://github.com/astropy/extension-helpers/ here an only activate it with a env-var. OpenMP shouldn't be used by default. See here for example: https://discuss.python.org/t/conflicting-binary-extensions-in-different-packages/25332
See here for reference: https://github.com/GeoStat-Framework/GSTools/blob/v1.6.0/setup.py
Update June 10, 2024
I temporarily converted this to a draft because the decisions for #28 heavily affect this PR because this might involve another change in the Cython-level API.
Changes
This pull request is an improved version of #26, so please refer to the basic changes in there. On top of these changes, this pull request
tools-modulesolver.pyxdso that the solvers only accept C-contiguous Arrays. This will not be backwards compatible because it could already be that users pass Fortran-contiguous Arrays (maybe without even knowing).setup.pyto include OpenMPworker-argument to the Python interfaceSince all that is a breaking change and the Cython interface is not backwards-compatible, I suggest that this is a major version jump from 1 to 2 (maybe as an$\alpha$ -version of it?).
Again, I tried to update the changelog, but it might be that you need to still look over it.
Tests
The installation and parallelisation were tested on Windows 11 and Ubuntu (WSL). ✅
Tests now cover serial and parallelel solves, but they have to run with
pytest --cov=pentapy .\tests --cov-report html -xnow to prevent interfering with the multithreading. ✅
⚠️ Note that the ⚠️
-xcancels the process for the first failure which is mandatory for this delicate topic where we deal with pointersDepending on the OS, it might be that the doctests of
tests/util_funcsfail because the Array output includes thedtypeon Windows while it does not on Linux. ❌Given that the C-implementation is already very fast in serial, the parallelization does not cause a quantum leap. With 8 threads on a relatively old laptop (there it is, the grain of salt 🧂), I observe a threefold speedup for huge systems (1,000 x 1,000 with 10,000 right-hand sides):
On massively parallel systems, this will give better speedups.
However, it does not negatively affect the serial solves when
workers=1: