Enable building DIPY with Meson #2715

skoudoro · 2023-01-18T19:27:36Z

PEP632: In Python 3.12, distutils will no longer be installed by make install or any of the first-party distribution. Third-party redistributors should no longer include distutils in their bundles or repositories. This PEP makes no specification on migrating the parts of the CPython build process that currently use distutils.

This is the main motivation of this PR. Python 3.12 should be out in September/October this year.

Our current build system is using distutils 😨 . Since all major packages/depencencies are moving to meson (scipy, numpy, scikit-image, scikit-learn, etc..), we follow the trend 😅 . Futhermore, Meson seems easy to use from my small and short experience. The maintenance should be easier.

The build should be complete. The full test suite passes. It is slower for now. I suppose, it is just missing some important compilation flags. Tested only on my macOS M1 (arm64 and rosetta - x86_64).

For now, This contains:

A set of meson.build files.
pyproject.toml

To test it

pip install -r requirements/build.txt
python -m dev.py build
python -m dev.py test -- -svv

Need to be done:

Some References

Quansight Labs blog: Moving SciPy to the Meson build system
SciPy docs: How to build SciPy with Meson
SciPy docs: advanced Meson build topics
Scipy migration discussion

close #2514

codecov · 2023-01-18T23:53:14Z

Codecov Report

Merging #2715 (5d009be) into master (bc3adcc) will decrease coverage by 0.10%.
The diff coverage is 54.16%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2715      +/-   ##
==========================================
- Coverage   81.81%   81.72%   -0.10%     
==========================================
  Files         147      147              
  Lines       20554    20587      +33     
  Branches     3289     3288       -1     
==========================================
+ Hits        16816    16824       +8     
- Misses       2905     2933      +28     
+ Partials      833      830       -3

Files	Coverage Δ
dipy/__init__.py	`80.00% <100.00%> (+5.00%)`	⬆️
dipy/conftest.py	`88.88% <100.00%> (ø)`
dipy/workflows/cli.py	`51.47% <51.47%> (ø)`

... and 1 file with indirect coverage changes

meson.build

eli-schwartz · 2023-01-30T02:04:48Z

meson.build

+# Hide symbols when building on Linux with GCC. For Python extension modules,
+# we only need `PyInit_*` to be public, anything else may cause problems. So we
+# use a linker script to avoid exporting those symbols (this is in addition to
+# Meson using `-fvisibility=hidden` for C and `-fvisibility-inlines-hidden` for
+# C++ code. See gh-15996 for details.
+_linker_script = meson.project_source_root() / 'skimage/_build_utils/link-version-pyinit.map'
+version_link_args = ['-Wl,--version-script=' + _linker_script]
+# Note that FreeBSD only accepts version scripts when -shared is passed,
+# hence we need to pass that to `cc.links` explicitly (flag is already
+# present for `extension_module` invocations).
+if not cc.links('', name: '-Wl,--version-script', args: ['-shared', version_link_args])
+  version_link_args = []
+endif


I wonder if you strictly need this? In SciPy's case it's complicated because they have a variety of code sources, and for example, Fortran code doesn't support visibility, so hiding Fortran symbols requires the linker script. And for vendored code embedded as static libraries, it can be too complicated to modify that external code to make it hide itself.

You may be able to get away with not bothering with this. :)

MANIFEST.in

skoudoro · 2023-01-30T20:32:44Z

Thank you for your feedback @eli-schwartz! I will deeper into all this later this week. As you notice, there is still a lot of cleaning to do. I will ping you back when I feel this PR ready.

jarrodmillman · 2023-03-16T22:04:20Z

We renamed devpy to spin due to PyPI availability:

skoudoro · 2023-08-31T16:03:51Z

ok, some progress. Quick note to myself (or anyone here)

I do not really know how the other project are handling the *.pxd file
Compilation fails on Windows (mingw-64) with this error fatal error: '-fuse-linker-plugin', but liblto_plugin.dll not found. Not sure why. I need to test all of that in a windows machine
Need to test on windows without mingw-64
Check openmp flag for macOSx, we might need to install openmp in our wheelbuilder to activate this flag on macOS
We do not use cpp, should we keep just in case or just remove.... To decide

-some cleaning still needed.

stefanv · 2023-09-14T21:23:40Z

Out of curiosity, what is the status of this PR, and are there any blockers?

skoudoro · 2023-09-15T02:05:46Z

Hi @stefanv,

Thank you for asking. Currently the main blocker is that the compilation fails on Windows (mingw-64) with this error fatal error: '-fuse-linker-plugin', but liblto_plugin.dll not found. Not sure why. Also, we need to test on windows without using mingw-64.

To be correct, they failed with the github action. I did not try yet on a personal window machine (working with macOs)

If you have any idea how to fix this, it will be more than welcomed!

Otherwise, some optimization flags to check (make sure they are correct in all OS) and make sure that we are using correctly the *.pxd file. But those tasks are not a blocker

stefanv · 2023-09-15T16:29:46Z

Would compiling with clang be an option? That's the only compiler we were able to get working for skimage: https://github.com/scikit-image/scikit-image/blob/main/.github/workflows/wheels_recipe.yml#L168

skoudoro · 2023-09-15T17:49:28Z

Interesting, I will get back to this next week and try with clang.

Thanks @stefanv !

skoudoro · 2023-09-19T13:02:41Z

ok, some progress, it seems to pass on Windows now with mingw-w64 from rtools.

Few tests are failing and it might be due to some compilation flag.

Also, not sure why pytest do not ignore benchmark. the config file is ignored for some reason

tons of commit need to be squash.

jarrodmillman · 2023-09-26T14:14:53Z

You may want to try spin 0.7: https://pypi.org/project/spin/0.7/

Here are the release notes: https://github.com/scientific-python/spin/blob/main/CHANGELOG.md

stefanv · 2023-09-27T06:03:35Z

.spin/cmds.py

+
+@click.command()
+@click.argument("asv_args", nargs=-1)
+def asv(asv_args):


You may also want to look at what we've written for NumPy:

https://github.com/numpy/numpy/blob/main/.spin/cmds.py#L331

Yours is much more straightforward, which I like, but perhaps there are useful features there.

that's cool! thanks @stefanv.

for sure, I will have a closer look later on a new PR. For now, I keep it simple and focus on meson + failing CI's.

this PR is close to be done. I will tackle / update the documentation on a new PR.

@jarrodmillman and @stefanv, thank you for spin tools, really useful, looking forward to add new commands on a new PR.

ps: Do you know why on Windows set_num_threads does not work. it seems that OpenMP use all the threads each times. Did you encounter this issue?

Which set_num_threads are you referring to?

Are you looking for something like https://pypi.org/project/threadpoolctl/ perhaps?

Sorry, I was referring to our own codebase:

dipy/dipy/utils/omp.pyx

Line 82 in 79c3939

cdef void set_num_threads(num_threads):

but more generally, do you have issue with openmp with scikit-image or other package ?

We haven't had any reports, but I'm not sure anyone is watching that closely on Windows. I think the most likely place where these issues would have been hashed out is in threadpoolctl.

pep8speaks · 2023-10-05T20:26:26Z

Hello @skoudoro, Thank you for updating !

In the file .spin/cmds.py:

Line 196:13: E117 over-indented
Line 199:35: W292 no newline at end of file

In the file benchmarks/asv_pip_nopep517.py:

Line 31:35: W292 no newline at end of file

In the file benchmarks/benchmarks/bench_reconst.py:

Line 67:5: E303 too many blank lines (2)

In the file benchmarks/benchmarks/bench_tracking.py:

Line 60:54: W292 no newline at end of file

In the file doc/tools/build_modref_templates.py:

Line 50:81: E501 line too long (81 > 80 characters)
Line 57:81: E501 line too long (81 > 80 characters)

In the file doc/tools/docgen_cmd.py:

Line 96:5: E303 too many blank lines (2)
Line 103:81: E501 line too long (81 > 80 characters)

Comment last updated at 2023-11-17 02:34:56 UTC

skoudoro · 2023-10-05T20:33:07Z

For now, this PR is done.

Can you someone test it and have a look @arokem? @Garyfallidis? this PR superesed #2787.

Next week, I will create new PR:

to handle our dipy_* command-line
Update our documentation
~~Update asv management for benchmarking~~

provides benchmarking mvp

skoudoro · 2023-11-03T05:17:49Z

@jhlegarreta, Can you look why it failed on python3.8? You can see the error here: https://github.com/dipy/dipy/actions/runs/6740916808/job/18324698874?pr=2715#step:7:150

maybe there is another way to get includes files from numpy. thank you.

stefanv · 2023-11-03T05:19:53Z

https://github.com/dipy/dipy/actions/runs/6740916808/job/18324698874?pr=2715#step:9:6644

type float has no attribute sqrt

jhlegarreta · 2023-11-03T12:57:36Z

Python 3.8 failure:
https://github.com/dipy/dipy/actions/runs/6741509003/job/18326222539?pr=2715#step:7:150

 ../dipy/meson.build:88:15: ERROR: Command `/Users/runner/work/dipy/dipy/venv/bin/python -c 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'` failed with status 1.

At fist sight having to do with mason and maybe the NumPy version in Python 3.8.

jhlegarreta · 2023-11-06T19:56:18Z

@skoudoro concerning #2715 (comment) I am not sure if NumPy is available at the time this call is performed (?)
https://github.com/dipy/dipy/actions/runs/6741509003/job/18326222539?pr=2715#step:6:156
https://github.com/dipy/dipy/actions/runs/6741509003/job/18326222539?pr=2715#step:7:9

No trace of messages saying that numpy has been installed after the latter line.

skoudoro · 2023-11-06T20:49:00Z

@skoudoro concerning #2715 (comment) I am not sure if NumPy is available at the time this call is performed (?)

Numpy is installed before as you can see here : https://github.com/dipy/dipy/actions/runs/6741509003/job/18326222539?pr=2715#step:6:66

jhlegarreta · 2023-11-06T20:58:54Z

Numpy is installed before as you can see here : https://github.com/dipy/dipy/actions/runs/6741509003/job/18326222539?pr=2715#step:6:66

OK, missed that line then. Thanks.

rgommers · 2023-11-07T11:22:25Z

Getting the NumPy include directory is a bit cumbersome right now I'm afraid. I suggest to copy exactly what SciPy does: https://github.com/scipy/scipy/blob/6ce4aaf4bc5222e3d1355cd8a392e88f45b4f959/scipy/meson.build#L30-L82.

The different cases are needed to deal with things like in-package-tree virtualenvs. I aim to make this as simple as

np_dep = dependency('numpy')

but that won't land before NumPy 2.0 early next year.

skoudoro · 2023-11-07T15:45:53Z

Getting the NumPy include directory is a bit cumbersome right now I'm afraid. I suggest to copy exactly what SciPy does:

Thank you @rgommers, looking forward for the new way to handle this!

skoudoro · 2023-11-07T19:08:32Z

Do you have an idea @rgommers why numpy include dir could fail on python3.8.

see here: https://github.com/dipy/dipy/actions/runs/6786880602/job/18448417652?pr=2715#step:7:159

all our python 3.8 build failed with this same error:

  ../dipy/meson.build:106:17: ERROR: Command `/home/runner/work/dipy/dipy/venv/bin/python -c '
  import numpy as np
  try:
    incdir = os.path.relpath(np.get_include())
  except Exception:
    incdir = np.get_include()
  print(incdir)
    '` failed with status 1.

rgommers · 2023-11-07T19:15:56Z

all our python 3.8 build failed with this same error:

You have no entry for numpy with python 3.8 in your build dependencies (requires = [) in pyproject.toml, so it doesn't get installed for 3.8.

Also, your pyproject.toml contains requires-python = ">=3.9". It seems like you should either drop the Python 3.8 CI jobs (which is certainly fine, NumPy and SciPy have already done that) or add in 3.8-specific metadata.

skoudoro · 2023-11-07T19:22:42Z

Arf, how I missed that... Good catch! thank you @rgommers

ok, I will just drop the python 3.8 CI's. Sorry @jhlegarreta (but no worries, wheels will be created for python 3.8 for this release a last time).

add python 3.12 CIs

jhlegarreta · 2023-11-07T19:27:43Z

ok, I will just drop the python 3.8 CI's. Sorry @jhlegarreta (but no worries, wheels will be created for python 3.8 for this release a last time).

If you drop CI's do you have a means to know and let people openly know that wheels for Python 3.8 will be working correctly? If the changes Ralf has mentioned are not hard, I would keep 3.8 until the release has been done.

skoudoro · 2023-11-07T19:33:04Z

wheels will not be created if CI's fails for python 3.8. I will let you look at https://github.com/MacPython/dipy-wheels to go deeper in the wheels creation mechanism for DIPY.

Another thing that we need to document 😅 and clarify.

jhlegarreta · 2023-11-07T19:37:47Z

wheels will not be created if CI's fails for python 3.8.

Then it seems counter-intuitive to remove the CIs for Python 3.8: if CI's pass, you have a better guarantee that you will not run into trouble creating the wheels. I still do not see why we should drop the CIs it at this stage; it would be 2 lines to modify in pyproject.toml.

skoudoro · 2023-11-17T14:26:15Z

ok, I see no additional comment, all is fixed. I am going ahead and merge this PR to move forward for the release.

Feel free to create an issue if there are any concerns!

Thanks everyone for the helps / advise / reviews

arokem · 2023-11-17T15:45:59Z

Monumental work on this serge! Kudos and big thanks to everyone who helped!

jhlegarreta · 2023-12-02T22:33:52Z

@skoudoro I have noticed that adapting some of the CI scripts to this change was inadvertently left behind, e.g.
https://github.com/dipy/dipy/blob/master/tools/ci/install.sh#L31

We no longer have a setup.py file.

skoudoro added type:Maintenance state: work-in-progress labels Jan 18, 2023

skoudoro force-pushed the meson branch from 3a3b17c to e5bcb2c Compare January 23, 2023 21:50

eli-schwartz suggested changes Jan 30, 2023

View reviewed changes

This was referenced Apr 21, 2023

Add an installation from a specific fork/commit. tee-ar-ex/trx-python#57

Merged

Trx integration to Dipy (part 2) tee-ar-ex/trx-python#59

Merged

skoudoro force-pushed the meson branch from e5bcb2c to ede5321 Compare August 30, 2023 04:29

skoudoro force-pushed the meson branch from ad6cbe9 to 15ba15a Compare September 15, 2023 02:56

skoudoro force-pushed the meson branch from 15ba15a to 7796843 Compare September 18, 2023 16:29

skoudoro force-pushed the meson branch from eee663a to 6ab1450 Compare September 22, 2023 03:45

skoudoro mentioned this pull request Sep 22, 2023

Question: Show compilation flag scientific-python/spin#116

Closed

matthew-brett mentioned this pull request Sep 22, 2023

No longer available in Python 3.12: numpy.distutils nipy/nipy#526

Closed

skoudoro mentioned this pull request Sep 26, 2023

[UPCOMING] Release 1.8.0 #2911

Merged

41 tasks

stefanv reviewed Sep 27, 2023

View reviewed changes

skoudoro force-pushed the meson branch 2 times, most recently from 3c73ed9 to 7fd33fd Compare October 5, 2023 01:39

skoudoro mentioned this pull request Oct 5, 2023

TRX integration #2759

Merged

skoudoro changed the title ~~[WIP] Enable building DIPY with Meson~~ Enable building DIPY with Meson Oct 5, 2023

skoudoro added 2 commits November 2, 2023 23:23

[NF/OPT] update/fix benchmarking

fc96060

provides benchmarking mvp

[TST] revert dtype for length_python

adfa4fa

skoudoro added 3 commits November 7, 2023 10:17

[RF] update cli management

a608aa2

[BF] improve numpy include management

1f57ec5

[RF] update docs generation

9cf0131

[MTN] remove python 3.8 CIs

b6b04f3

add python 3.12 CIs

skoudoro added 5 commits November 8, 2023 12:29

add clean command

fb9ad58

[DOC] update instructions

24191a6

[MTN] back to origin

821ade3

[MTN] update minimal CI

a35e01e

Merge branch 'master' into meson

5d009be

skoudoro merged commit cf4641a into dipy:master Nov 17, 2023
32 of 34 checks passed

skoudoro deleted the meson branch November 17, 2023 14:26

Enable building DIPY with Meson #2715

Enable building DIPY with Meson #2715

Conversation

skoudoro commented Jan 18, 2023 • edited

To test it

Need to be done:

Some References

codecov bot commented Jan 18, 2023 • edited

Codecov Report

eli-schwartz Jan 30, 2023

Choose a reason for hiding this comment

skoudoro commented Jan 30, 2023

jarrodmillman commented Mar 16, 2023

skoudoro commented Aug 31, 2023

stefanv commented Sep 14, 2023

skoudoro commented Sep 15, 2023 • edited

stefanv commented Sep 15, 2023 • edited

skoudoro commented Sep 15, 2023

skoudoro commented Sep 19, 2023 • edited

jarrodmillman commented Sep 26, 2023

stefanv Sep 27, 2023

Choose a reason for hiding this comment

skoudoro Sep 27, 2023

Choose a reason for hiding this comment

stefanv Sep 27, 2023

Choose a reason for hiding this comment

skoudoro Sep 27, 2023

Choose a reason for hiding this comment

stefanv Sep 27, 2023

Choose a reason for hiding this comment

pep8speaks commented Oct 5, 2023 • edited

Comment last updated at 2023-11-17 02:34:56 UTC

skoudoro commented Oct 5, 2023 • edited

skoudoro commented Nov 3, 2023

stefanv commented Nov 3, 2023

jhlegarreta commented Nov 3, 2023

jhlegarreta commented Nov 6, 2023

skoudoro commented Nov 6, 2023

jhlegarreta commented Nov 6, 2023

rgommers commented Nov 7, 2023

skoudoro commented Nov 7, 2023

skoudoro commented Nov 7, 2023

rgommers commented Nov 7, 2023

skoudoro commented Nov 7, 2023

jhlegarreta commented Nov 7, 2023

skoudoro commented Nov 7, 2023

jhlegarreta commented Nov 7, 2023

skoudoro commented Nov 17, 2023

arokem commented Nov 17, 2023

jhlegarreta commented Dec 2, 2023

skoudoro commented Jan 18, 2023 •

edited

codecov bot commented Jan 18, 2023 •

edited

skoudoro commented Sep 15, 2023 •

edited

stefanv commented Sep 15, 2023 •

edited

skoudoro commented Sep 19, 2023 •

edited

pep8speaks commented Oct 5, 2023 •

edited

skoudoro commented Oct 5, 2023 •

edited