Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 1 - some packages without OptionalExtensions)

[mkoeppe]

This simplifies src/module_list.py a lot. This is preparation for splitting sagelib into separate distutils packages.

Follow-up tickets take care of the remaining packages; in the end, in #29701, src/module_list.py will no longer be used, and in another follow-up ticket, it will disappear completely:

• Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 2 - OptionalExtensions) #29720: Move Extension options ... (part 2 - OptionalExtensions)
• Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 3: Get rid of uname_specific) #29785: Move Extension options ... (part 3: Get rid of uname_specific)
• Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 4: sage.rings) #29786: Move Extension options ... (part 4: sage.rings)
• Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 5: sage.graphs) #29790: Move Extension options ... (part 5: sage.graphs)
• Move Extension options from src/module_list.py to "distutils:" directives in the individual files (part 6: last) #29791: Move Extension options ... (part 6: last)
• coxeter3: Add spkg-configure.m4, use coxeter/ prefix for includes, move OptionalExtension options to .pyx file directives #29721: takes care of sage.libs.coxeter3
• Replace use of module_list and OptionalExtension by extending find_python_sources #29701: takes care of sage.graphs.graph_decompositions

CC: @kiwifb @dimpase @jhpalmieri @tscrim @kliem @roed314 @videlec @vbraun

Component: refactoring

Keywords: sd109

Author: Matthias Koeppe

Branch/Commit: 1bfe30d

Reviewer: Jonathan Kliem, John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/29706

[mkoeppe]

Branch: u/mkoeppe/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files

[sagetrac-git]

Commit: f31b6a0

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

f31b6a0

src/sage/geometry: Move Extension options from src/module_list.py to distutils directives

[sagetrac-git]

Changed commit from f31b6a0 to 346bd88

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

0c0ef43	src/sage/libs/ntl: Move Extension options from src/module_list.py to distutils directives
a4d04d3	src/sage/modular: Move Extension options from src/module_list.py to distutils directives
17ca1d7	src/sage/env.py (cython_aliases): Add aliases for libraries use in module_list.py
346bd88	src/sage/modules: Move Extension options from src/module_list.py to distutils directives

[mkoeppe]

Author: Matthias Koeppe

[sagetrac-git]

Changed commit from 346bd88 to 70901ea

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

70901ea

src/sage/env.py (cython_aliases): Update doctest

[dimpase]

comment:10

Aren't all these extra_compile_args=['-std=c++11'] (and the corresponding cython #-directives obsolete (as we now set this globally for the C++ compiler)?

[mkoeppe]

comment:11

I would guess so but would prefer to do that on a different ticket

[sagetrac-git]

Changed commit from 70901ea to 10f7542

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

eed920e	src/sage/tests: Move Extension options from src/module_list.py to distutils directives
0d25d1d	src/sage/structure: Move Extension options from src/module_list.py to distutils directives
808f46a	src/sage/stats: Move Extension options from src/module_list.py to distutils directives
10f7542	src/sage/schemes: Move Extension options from src/module_list.py to distutils directives

[kliem]

comment:14

Isn't src/sage/geometry/triangulation/base.pyx itself redundant here?

-# distutils: sources = sage/geometry/triangulation/base.pyx sage/geometry/triangulation/functions.cc sage/geometry/triangulation/data.cc sage/geometry/triangulation/triangulations.cc
+# distutils: sources = sage/geometry/triangulation/functions.cc sage/geometry/triangulation/data.cc sage/geometry/triangulation/triangulations.cc

Same thing in

• /src/sage/schemes/hyperelliptic_curves/hypellfrob.pyx
• sage/stats/distributions/discrete_gaussian_integer.pyx)

I think you missed this one. At least I don't find the corresponding file in the diffs.

-    Extension('sage.modular.pollack_stevens.dist',
-              sources = ['sage/modular/pollack_stevens/dist.pyx'],
-              libraries = ["gmp", "zn_poly"],
-              extra_compile_args = ["-D_XPG6"]),

I don't understand (this is probably reasonable, but I don't have any knowledge about that)

    # TODO: Remove Cygwin hack by installing a suitable cblas.pc
    if os.path.exists('/usr/lib/libblas.dll.a'):
        aliases["CBLAS_LIBS"] = ['gslcblas']

Everything else looks great to me. However, I didn't test anything yet.

[kliem]

Changed keywords from none to sd109

[mkoeppe]

comment:16

Replying to @kliem:

Isn't src/sage/geometry/triangulation/base.pyx itself redundant here?

-# distutils: sources = sage/geometry/triangulation/base.pyx sage/geometry/triangulation/functions.cc sage/geometry/triangulation/data.cc sage/geometry/triangulation/triangulations.cc
+# distutils: sources = sage/geometry/triangulation/functions.cc sage/geometry/triangulation/data.cc sage/geometry/triangulation/triangulations.cc

Same thing in

• /src/sage/schemes/hyperelliptic_curves/hypellfrob.pyx
• sage/stats/distributions/discrete_gaussian_integer.pyx)

I think you missed this one. At least I don't find the corresponding file in the diffs.

-    Extension('sage.modular.pollack_stevens.dist',
-              sources = ['sage/modular/pollack_stevens/dist.pyx'],
-              libraries = ["gmp", "zn_poly"],
-              extra_compile_args = ["-D_XPG6"]),

Quite possible on both of these. If you have time to test and make these fixes, that would be great!

I don't understand (this is probably reasonable, but I don't have any knowledge about that)

    # TODO: Remove Cygwin hack by installing a suitable cblas.pc
    if os.path.exists('/usr/lib/libblas.dll.a'):
        aliases["CBLAS_LIBS"] = ['gslcblas']

I don't understand it either, I only it moved it there from its origin. It's quite likely that it is outdated - this should be addressed on a separate ticket.

[kliem]

comment:17

Ok. I'm testing it now both locally and with github actions.

[kliem]

comment:18

It all worked on my end (sage -t --all --long)

Testing it on github reavealed some strange error with debian buster standard.

https://github.com/kliem/sage-test-27122/runs/722743115

  [dochtml]     File "/sage/local/lib/python3.7/site-packages/sage/matrix/__init__.py", line 2, in <module>
  [dochtml]       import sage.matrix.args
  [dochtml]     File "sage/matrix/args.pyx", line 23, in init sage.matrix.args (build/cythonized/sage/matrix/args.c:21224)
  [dochtml]     File "/sage/local/lib/python3.7/site-packages/sage/matrix/matrix_space.py", line 44, in <module>
  [dochtml]       from . import matrix_modn_sparse
  [dochtml]     File "sage/matrix/matrix_integer_sparse.pxd", line 5, in init sage.matrix.matrix_modn_sparse (build/cythonized/sage/matrix/matrix_modn_sparse.cpp:16301)
  [dochtml]   ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /sage/local/lib/python3.7/site-packages/sage/matrix/matrix_integer_sparse.cpython-

How come numerical, rings etc. wheren't touched?

[dimpase]

comment:19

I also see this kind of strange error on GH Actions on buster. Perhaps the buster image used is faulty?

[kliem]

comment:20

Also happens on ubuntu bionic and linuxmint 19.

Overall the testing experience is awful at the moment and it appears to have nothing to do with this ticket: https://github.com/kliem/sage-test-27122/actions/runs/119764262

Except for ubuntu focal and eoan and debian bullseye and sid (all standard not minimal) nothing passes within 6 hours and many failures.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

1bfe30d

sage.env.cython_aliases: Another fix for macOS without zlib pc

[sagetrac-git]

Changed commit from b1b3787 to 1bfe30d

[mkoeppe]

comment:48

Sorry, here's another fix.

[jhpalmieri]

comment:49

Any suggestions for how to reproduce the error? I've tried using the previous version of the branch, and either building with lots of system packages or with very few, but I haven't hit this build error.

[mkoeppe]

comment:50

Basically, one needs a macOS without homebrew (which installs .pc files for system libraries). Or do the following to simulate that:

  (cd /usr/local/Homebrew/Library/Homebrew/os/mac/ && mv pkgconfig pkgconfig-moved)

[mkoeppe]

comment:51

I created #29929 (tox.ini: Add a macos environment without homebrew, conda) so we can test such a barebones macOS environment on GH Actions

[kliem]

comment:52

Have you tried the current branch with such a system? I am completely clueless, whether the last commit resolves the problem.

[mkoeppe]

comment:53

Replying to @kliem:

Have you tried the current branch with such a system? I am completely clueless, whether the last commit resolves the problem.

Yes, a build from scratch with the manipulation mentioned in comment 50 just finished successfully on my machine.

[jhpalmieri]

comment:54

I have not yet been able to reproduce the error, so I can't tell if the most recent change fixes anything. Renaming /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig didn't cause breakage, nor did renaming /usr/local/lib/pkgconfig. I'm trying on a different machine by just removing homebrew altogether.

[jhpalmieri]

comment:55

I can confirm on a machine without Homebrew, I hit the failure that Volker mentioned, and the most recent change fixes it. I'm going to switch to a positive review.

[mkoeppe]

Changed reviewer from Jonathan Kliem to Jonathan Kliem, John Palmieri

[mkoeppe]

comment:56

Thank you!

[vbraun]

Changed branch from u/mkoeppe/c536daa32dcc69adbc7b37d9a170e22fd733bca8 to 1bfe30d