New option "./configure --enable-wheels" #32874

mkoeppe · 2021-11-15T02:28:03Z

When ./configure --disable-editable is in use (the default before #32406), the Sage library is installed using its custom incremental implementation of setup.py install -- which installs its files in site-packages and then removes the files that it does not know about (the "cleaner").

Problem 1: The cleaner is incompatible with namespace packages and trying to fix it would lead to complicated and awkward code (#32927). (Besides, setup.py install is deprecated since https://setuptools.pypa.io/en/latest/history.html#v58-3-0)

Problem 2: Because of the direct installation, we do not have wheels for sagemath-standard available (even if make wheels is used explicitly). Such wheels can be useful for making separate venvs.

In this ticket:

We create a new option ./configure --enable-wheels (which can be used both with ./configure --enable-editable and ./configure --disable-editable).
If enabled, installation of sagelib will go through wheel building and installation - like we do with all other Python packages. This enables modularized builds (Refactor sagemath-standard through sagemath-{categories,environment,repl} #34587).
If disabled, there is no change.

A trivial ./sage -b is much slower with ./configure --disable-editable --enable-wheels compared to ./configure --disable-editable (from 28s as per timings in #32406 comment:21 to 70s).
This ticket brings a number of speed improvements to mitigate this.

CC: @jhpalmieri

Component: build

Author: Matthias Koeppe

Branch: 25898ba

Reviewer: Kwankyu Lee

Issue created by migration from https://trac.sagemath.org/ticket/32874

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-11-15T02:45:07Z

Branch: u/mkoeppe/remove_use_of__setup_py_install__for_sagelib

mkoeppe · 2021-11-15T03:12:20Z

New commits:

`b9b1dba`	`Enable editable mode also for other sage packages`
`a257621`	`Add error handling`
`89bc3ee`	`Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace_ext`
`7bd6ce4`	`Partly revert "Enable editable mode also for other sage packages"`
`3ef9c1d`	`Merge #32713`
`50dd26c`	`build/pkgs/sagelib/spkg-install: Use helper function sdh_pip_editable_install`
`b5c40a8`	`build/pkgs/sagelib/spkg-install: In non-editable mode, Use sdh_pip_install instead of 'setup.py install'`

mkoeppe · 2021-11-15T03:12:20Z

Author: Matthias Koeppe

mkoeppe · 2021-11-15T03:12:20Z

Commit: b5c40a8

mkoeppe · 2022-05-12T02:10:33Z

Changed dependencies from #32713 to #32406

sagetrac-git · 2022-07-22T19:03:13Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`8619b29`	`configure.ac: Make --enable-editable the default`
`6513b6b`	`README.md: Explain configure --disable-editable`
`ff8710e`	`docker/Dockerfile: Use configure --disable-editable`
`7221a76`	`src/setup.py: Do not run find_namespace_packages for 'setup.py dist_info'`
`a911e0f`	`src/MANIFEST.in: prune sage_docbuild, doc`
`edfa3b4`	`build/pkgs/sagelib/spkg-install: Use helper function sdh_pip_editable_install`
`f7ea602`	`build/pkgs/sagelib/spkg-install: In non-editable mode, Use sdh_pip_install instead of 'setup.py install'`

sagetrac-git · 2022-07-22T19:03:13Z

Changed commit from b5c40a8 to f7ea602

mkoeppe · 2022-07-22T19:35:12Z

comment:10

[sagelib-9.7.beta5] sage-flock: error: argument LOCK: can't create '/doesnotexist/var/lock': [Errno 30] Read-only file system: '/doesnotexist'

sagetrac-git · 2022-07-22T21:03:13Z

Branch pushed to git repo; I updated commit sha1. New commits:

`df1e7eb`	`build/bin/sage-pip-install: Put lock file in SAGE_VENV, not SAGE_LOCAL`
`f358962`	`build/pkgs/sagelib/spkg-install: No need to set env var PYTHON`
`c761788`	`build/bin/sage-pip-uninstall: Put lock file in SAGE_VENV, not SAGE_LOCAL`
`99d4129`	`build/pkgs/sagelib/spkg-install [SAGE_EDITABLE=no]: Use --no-build-isolation`

sagetrac-git · 2022-07-22T21:03:13Z

Changed commit from f7ea602 to 99d4129

sagetrac-git · 2022-09-26T05:01:26Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`07f5cde`	`build/pkgs/sagelib/spkg-install: Use helper function sdh_pip_editable_install`
`3cdf86e`	`build/pkgs/sagelib/spkg-install: In non-editable mode, Use sdh_pip_install instead of 'setup.py install'`
`459ee2f`	`build/bin/sage-pip-install: Put lock file in SAGE_VENV, not SAGE_LOCAL`
`3de2608`	`build/pkgs/sagelib/spkg-install: No need to set env var PYTHON`
`e6eae68`	`build/bin/sage-pip-uninstall: Put lock file in SAGE_VENV, not SAGE_LOCAL`
`e829b08`	`build/pkgs/sagelib/spkg-install [SAGE_EDITABLE=no]: Use --no-build-isolation`

sagetrac-git · 2022-09-26T05:01:26Z

Changed commit from 99d4129 to e829b08

sagetrac-git · 2022-10-04T18:15:49Z

Changed commit from 11507a5 to 974c7fd

sagetrac-git · 2022-10-04T18:15:49Z

Branch pushed to git repo; I updated commit sha1. New commits:

`974c7fd`	`src/MANIFEST.in: Remove redundant entries for MANIFEST.in, pyproject.toml`

mkoeppe · 2022-10-04T18:16:37Z

comment:74

Replying to Kwankyu Lee:

Questions on changes in src/MANIFEST.in:

Is this line include MANIFEST.in necessary? Perhaps because src/MANIFEST.in is a symlink?

No, not necessary any more; it probably was necessary with prehistoric versions of setuptools.

sagetrac-git · 2022-10-04T18:26:27Z

Branch pushed to git repo; I updated commit sha1. New commits:

`25898ba`	`src/MANIFEST.in: Exclude generated file farey_symbol.h; add comments`

sagetrac-git · 2022-10-04T18:26:27Z

Changed commit from 974c7fd to 25898ba

mkoeppe · 2022-10-04T18:27:11Z

comment:76

Replying to Kwankyu Lee:

How was this list made ?

I've added a comment (and fixed a mistake that I had made along the way).

kwankyu · 2022-10-04T23:27:31Z

comment:77

Thanks.

It works well and looks good to me.

mkoeppe · 2022-10-04T23:58:30Z

comment:78

Thank you!

vbraun · 2022-10-16T22:16:00Z

Changed branch from u/mkoeppe/remove_use_of__setup_py_install__for_sagelib to 25898ba

antonio-rojas · 2022-10-17T20:14:15Z

Changed commit from 25898ba to none

antonio-rojas · 2022-10-17T20:14:15Z

comment:80

This makes tests depend on sage_setup, so one gets failures when testing an installed sage

sage -t --long --random-seed=334405888414880658562662015982432628863 /usr/lib/python3.10/site-packages/sage/misc/package_dir.py
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/misc/package_dir.py", line 108, in sage.misc.package_dir.read_distribution
Failed example:
    from sage_setup.find import read_distribution
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package_dir.read_distribution[1]>", line 1, in <module>
        from sage_setup.find import read_distribution
    ModuleNotFoundError: No module named 'sage_setup'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/misc/package_dir.py", line 109, in sage.misc.package_dir.read_distribution
Failed example:
    read_distribution(os.path.join(SAGE_SRC, 'sage', 'graphs', 'graph_decompositions', 'tdlib.pyx'))
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package_dir.read_distribution[2]>", line 1, in <module>
        read_distribution(os.path.join(SAGE_SRC, 'sage', 'graphs', 'graph_decompositions', 'tdlib.pyx'))
    NameError: name 'read_distribution' is not defined
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/misc/package_dir.py", line 111, in sage.misc.package_dir.read_distribution
Failed example:
    read_distribution(os.path.join(SAGE_SRC, 'sage', 'graphs', 'graph_decompositions', 'modular_decomposition.py'))
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package_dir.read_distribution[3]>", line 1, in <module>
        read_distribution(os.path.join(SAGE_SRC, 'sage', 'graphs', 'graph_decompositions', 'modular_decomposition.py'))
    NameError: name 'read_distribution' is not defined
**********************************************************************

mkoeppe · 2022-10-17T20:54:33Z

comment:81

I've opened #34674 for this

kwankyu · 2022-10-17T22:17:53Z

comment:82

A question to educate myself: In what environment, sage_setup module is not available? Is this a linux package?

mkoeppe · 2022-10-17T22:21:06Z

comment:83

In distribution packaging (Antonio does the packaging for Arch Linux), sage_setup is typically not available at runtime.

kwankyu · 2022-10-17T22:27:19Z

comment:84

Okay. Thanks.

mkoeppe added this to the sage-9.5 milestone Nov 15, 2021

mkoeppe added c: build labels Nov 15, 2021

This comment has been minimized.

Sign in to view

mkoeppe modified the milestones: sage-9.5, sage-9.6 Nov 15, 2021

This comment has been minimized.

Sign in to view

mkoeppe modified the milestones: sage-9.6, sage-9.7 Mar 5, 2022

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Remove use of "setup.py install" for sagelib~~ Remove use of "setup.py install" for sagelib when ./configure --disable-editable is in use Jul 22, 2022

mkoeppe added s: needs review and removed s: needs review labels Jul 22, 2022

This comment has been minimized.

Sign in to view

mkoeppe modified the milestones: sage-9.7, sage-9.8 Jul 23, 2022

kwankyu added s: positive review and removed s: needs review labels Oct 4, 2022

vbraun removed the s: positive review label Oct 16, 2022

vbraun closed this as completed in a9abd3b Oct 16, 2022

isuruf mentioned this issue Jan 7, 2023

New modularized distributions sagemath-{sirocco,bliss,meataxe,mcqd,coxeter3,tdlib} #34346

Closed

This was referenced Jan 7, 2023

Refactor sagemath-standard through sagemath-{categories,environment,repl} #34587

Open

sage_setup: Output depfiles #34631

Open

Use src/setup.py for the installation mode --disable-editable --disable-wheels #34633

Open

vbraun mentioned this issue Oct 11, 2022

fix more W605 in rings #34645

Closed

mkoeppe mentioned this issue Feb 11, 2023

Switch modularized sagelib's build systems from setuptools to meson-python / ninja #34630

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New option "./configure --enable-wheels" #32874

New option "./configure --enable-wheels" #32874

mkoeppe commented Nov 15, 2021

mkoeppe commented Nov 15, 2021

This comment has been minimized.

mkoeppe commented Nov 15, 2021

mkoeppe commented Nov 15, 2021

mkoeppe commented Nov 15, 2021

This comment has been minimized.

mkoeppe commented May 12, 2022

sagetrac-git mannequin commented Jul 22, 2022

sagetrac-git mannequin commented Jul 22, 2022

This comment has been minimized.

mkoeppe commented Jul 22, 2022

sagetrac-git mannequin commented Jul 22, 2022

sagetrac-git mannequin commented Jul 22, 2022

This comment has been minimized.

sagetrac-git mannequin commented Sep 26, 2022

sagetrac-git mannequin commented Sep 26, 2022

sagetrac-git mannequin commented Oct 4, 2022

sagetrac-git mannequin commented Oct 4, 2022

mkoeppe commented Oct 4, 2022

sagetrac-git mannequin commented Oct 4, 2022

sagetrac-git mannequin commented Oct 4, 2022

mkoeppe commented Oct 4, 2022

kwankyu commented Oct 4, 2022

mkoeppe commented Oct 4, 2022

vbraun commented Oct 16, 2022

antonio-rojas commented Oct 17, 2022

antonio-rojas commented Oct 17, 2022

mkoeppe commented Oct 17, 2022

kwankyu commented Oct 17, 2022

mkoeppe commented Oct 17, 2022

kwankyu commented Oct 17, 2022

New option "./configure --enable-wheels" #32874

New option "./configure --enable-wheels" #32874

Comments

mkoeppe commented Nov 15, 2021

mkoeppe commented Nov 15, 2021

This comment has been minimized.

mkoeppe commented Nov 15, 2021

mkoeppe commented Nov 15, 2021

mkoeppe commented Nov 15, 2021

This comment has been minimized.

mkoeppe commented May 12, 2022

sagetrac-git mannequin commented Jul 22, 2022

sagetrac-git mannequin commented Jul 22, 2022

This comment has been minimized.

mkoeppe commented Jul 22, 2022

sagetrac-git mannequin commented Jul 22, 2022

sagetrac-git mannequin commented Jul 22, 2022

This comment has been minimized.

sagetrac-git mannequin commented Sep 26, 2022

sagetrac-git mannequin commented Sep 26, 2022

sagetrac-git mannequin commented Oct 4, 2022

sagetrac-git mannequin commented Oct 4, 2022

mkoeppe commented Oct 4, 2022

sagetrac-git mannequin commented Oct 4, 2022

sagetrac-git mannequin commented Oct 4, 2022

mkoeppe commented Oct 4, 2022

kwankyu commented Oct 4, 2022

mkoeppe commented Oct 4, 2022

vbraun commented Oct 16, 2022

antonio-rojas commented Oct 17, 2022

antonio-rojas commented Oct 17, 2022

mkoeppe commented Oct 17, 2022

kwankyu commented Oct 17, 2022

mkoeppe commented Oct 17, 2022

kwankyu commented Oct 17, 2022