Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420)

[mkoeppe]

This ticket prepares the modularization of sagelib into separate distributions (pip-installable packages) by adding support for native namespace packages (PEP 420) to sagelib's build system.

Ordinary packages (directories of modules) can be turned into namespace packages by removing the empty __init__.py file.
Then several distribution package can share the same namespace.

Because our source discovery mechanism in sagelib's build system (setup.py + sage_setup) distinguishes package directories from other directories containing data files by the presence of the __init__.py file, the present ticket makes several adjustments to the build system - in particular to the functions find_python_sources and find_extra_files.

Per convention in the Sage library, namespace packages are recognized by the presence of all.py or all__*.py files (#33033).

In this ticket, we apply it to turn sagemath-objects and sagemath-categories (from #29865) into namespace packages,
in which __init__.py are removed from the MANIFEST.in files.

Because the actual files are not removed from the source tree, the normal build of the Sage distribution is not affected.

To test these two distributions:

./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-objects && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'

./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-categories && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'

Follow-up tickets:

• Remove __init__.py files for subpackages designated to be namespace packages #33011 Remove the __init__.py files for packages designated to be namespace packages
• Wishlist item: Warn or give errors if the sage_setup: distribution directives, namespace and nonamespace files indicate an inconsistent structure such as when it was forgotten to remove __init__.py.
• sage_setup: Modify clean_stale_files to support out-of-tree namespace packages #30152 will make further changes to the cleaner that will allow out-of-tree namespace packages

See also:

• Meta-ticket: Modularize sagelib into separate distributions (pip-installable packages) sagemath-... for Sage 10.x #29705: Meta-ticket: Modularize sagelib into separate distutils packages
• Initial discussion of namespace packages for Sage in Move optional sage optimization backends (COIN, CPLEX, Gurobi) to separate Cython packages to remove OptionalExtension problems #28175 and Use native namespace package sage-numerical-backends-coin#4
• src/setup.py: Disentangle cleaning of stale installed files in build directory and in install directory #21654: src/setup.py: Disentangle cleaning of stale installed files in build directory and in install directory

Depends on #33803

CC: @isuruf @embray @jdemeyer @dimpase @dcoudert @videlec @vbraun @tobiasdiez @kiwifb

Component: build

Keywords: sd111

Author: Matthias Koeppe

Branch/Commit: f9df1ae

Reviewer: Kwankyu Lee, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/28925

[mkoeppe]

comment:1

pushing these forward to 9.2

[mkoeppe]

Branch: u/mkoeppe/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[mkoeppe]

Commit: 8bc5662

[mkoeppe]

New commits:

8e50bbe	sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)
dff6e62	Mark src/sage/ext_data as directories that are not namespace packages
8bc5662	sage_setup.find.find_python_sources: Handle native namespace packages and 'nonamespace' files

[mkoeppe]

Author: Matthias Koeppe

[sagetrac-git]

Changed commit from 8bc5662 to 606a509

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

606a509

sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)

[mkoeppe]

comment:18

Next steps: adjust the cleaner, find Cython sources

[mkoeppe]

comment:19

The Cython source finder could perhaps also parse a new directive that controls OptionalExtension behavior (until we get rid of OptionalExtension altogether in #29701) so that we can get rid of module_list.py completely (see #29706)

[sagetrac-git]

Changed commit from 606a509 to a7b2287

[kwankyu]

comment:164

Replying to @dimpase:

hmm, do you have the current directory, i.e. ., in your PATH? IMHO this might mess things up.

Yes. But even after removing it from the path, it's the same.

./configure shows python3-3.10.3: using system package; SPKG will not be installed, and

$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages']

$ pkgs/sagemath-categories/.tox/python/bin/python 
Python 3.10.4 (main, Apr 26 2022, 19:42:59) [Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/local/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python310.zip', '/usr/local/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10', '/usr/local/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload', '/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages']

[mkoeppe]

comment:165

Replying to @dimpase:

Replying to @kwankyu:

Is this

./sage -sh -c '(cd pkgs/sagemath-categories && tox -v -v -v)'  

broken? It doesn't work with (and without) this ticket.

works for me (after make build from a clean state done)

sagemath-categories (in contrast to sagemath-standard) does not need make build because all of its build and runtime dependencies can be found on PyPI.

[mkoeppe]

comment:166

Replying to @kwankyu:

This is the initial part that goes wrong:

sage -t --random-seed=251639358214975434990568781743714015030 --environment=sage.all__sagemath_categories /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/structure/coerce_actions.pyx
**********************************************************************
File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/structure/coerce_actions.pyx", line 58, in sage.structure.coerce_actions.GenericAction.__init__
Failed example:
    M = MatrixSpace(ZZ,2)
Exception raised:
    Traceback (most recent call last):
      File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.structure.coerce_actions.GenericAction.__init__[0]>", line 1, in <module>
        M = MatrixSpace(ZZ,Integer(2))
    NameError: name 'MatrixSpace' is not defined

sagemath-categories as of this ticket does not contain src/sage/matrix/matrix_space.py (see pkgs/sagemath-categories/MANIFEST.in), so MatrixSpace is not available in sage.all__sagemath_categories.

Note that the tox run ends with the message "lots of doctest failures are expected". This is one of them.

[mkoeppe]

comment:167

Replying to @kwankyu:

The real problem is:

$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.categories.category import Category
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages/sage/categories/category.py", line 107, in <module>
    from sage.misc.c3_controlled import _cmp_key, _cmp_key_named, C3_sorted_merge
  File "sage/misc/c3_controlled.pyx", line 364, in init sage.misc.c3_controlled (build/cythonized/sage/misc/c3_controlled.c:11819)
  File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages/sage/structure/__init__.py", line 2, in <module>
    import sage.structure.element
  File "sage/structure/element.pyx", line 1, in init sage.structure.element (build/cythonized/sage/structure/element.c:36204)
  File "sage/structure/category_object.pyx", line 60, in init sage.structure.category_object (build/cythonized/sage/structure/category_object.c:9982)
ImportError: cannot import name Category

Strangely this works

Hera:sage-dev$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.structure.sage_object import SageObject
>>> from sage.categories.category import Category
>>> 

The Sage library contains many cyclic imports. If one starts at the wrong place in an import cycle, one can run into problems with partially initialized modules. See #33580 (Meta-ticket: make all modules importable without sage.all)

[mkoeppe]

comment:168

Thanks both for the review!

[kwankyu]

comment:169

Replying to @mkoeppe:

Note that the tox run ends with the message "lots of doctest failures are expected".

Yes, it's there. It would perhaps better be colored red or green...

[vbraun]

comment:170

Merge failure on top of:

843eb03 Updated SageMath version to 9.7.beta4

merge was not clean: conflicts in pkgs/sagemath-categories/tox.ini

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

f9df1ae

Merge tag '9.7.beta4' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

[sagetrac-git]

Changed commit from 9ab3200 to f9df1ae

[vbraun]

Changed branch from u/mkoeppe/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_ to f9df1ae