New make targets "sagelib-tox-...", "sage_docbuild-tox-...", ... #31535

mkoeppe · 2021-03-22T03:00:14Z

For all script packages SPKG that have an embedded source tree -- currently sagelib, sage_docbuild, sage_sws2rst -- we add a new target:

SPKG-tox-% (which calls tox -e % in the package's source tree)

These targets are particularly useful for invoking them within the portability tests of the top-level tox:

$ TARGETS_PRE=config.status tox -e docker-ubuntu-focal-standard -- sage_docbuild-tox invokes the tox testing of the sage_docbuild distribution
$ EXTRA_CONFIGURE_ARGS="--disable-notebook" tox -e docker-ubuntu-focal-standard -- build sagelib-tox-python-sagewheels-nopypi builds (in a Docker container) the Sage distribution (without Jupyter notebook) and then invokes the tox testing of sagelib (added in sagelib: Generate pyproject.toml and setup.cfg [install_requires], requirements.txt, Pipfile, and src/Pipfile #30913) in a venv populated using the wheels
$ tox -e local-homebrew-macos-maximal -- build-local cysignals networkx sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi (added in sagelib: Generate pyproject.toml and setup.cfg [install_requires], requirements.txt, Pipfile, and src/Pipfile #30913) builds the non-Python parts of the Sage distribution on top of Homebrew, using a virtual environment that uses homebrew's Python packages
$ tox -e local-conda-forge-macos-src-environment -- sage_conf sagelib-tox-python-sagewheels-sitepackages-norequirements-nobuildisolation-nopypi (needs sagelib: Generate pyproject.toml and setup.cfg [install_requires], requirements.txt, Pipfile, and src/Pipfile #30913, cython_aliases: Do not fail if one of the listed libraries is not known to pkgconfig #31384) uses the src/environment.yml file to set up a complete environment from conda and then tests sagelib in a venv using tox

Depends on #32899

CC: @jhpalmieri @dimpase @kliem @orlitzky @tobiasdiez

Component: porting

Author: Matthias Koeppe

Branch/Commit: ad12d25

Reviewer: Michael Orlitzky

Issue created by migration from https://trac.sagemath.org/ticket/31535

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-03-22T03:02:47Z

Branch: u/mkoeppe/new_make_targets__sagelib_sdist____sage_docbuild_sdist_______and__sagelib_tox__________

mkoeppe · 2021-03-22T03:03:30Z

Author: Matthias Koeppe

mkoeppe · 2021-03-22T03:03:30Z

New commits:

`1eef6d6`	`build/make/Makefile.in: Add targets SPKG-sdist, SPKG-tox, SPKG-tox-% for script packages`

mkoeppe · 2021-03-22T03:03:30Z

Commit: 1eef6d6

sagetrac-git · 2021-03-22T04:01:18Z

Changed commit from 1eef6d6 to 6964933

sagetrac-git · 2021-03-22T04:01:18Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6964933`	`build/make/Makefile.in: fixup`

sagetrac-git · 2021-03-22T19:54:12Z

Changed commit from 6964933 to 188bf2e

sagetrac-git · 2021-03-22T19:54:12Z

Branch pushed to git repo; I updated commit sha1. New commits:

`9993348`	`build/make/Makefile.in: Run targets SPKG-tox, SPKG-tox-% outside of sage-env, using SAGE_ORIG_PATH - so that tox from conda works`
`4533018`	`tox.ini: Add local-conda-environment-src`
`188bf2e`	`tox.ini, src/doc/bootstrap: Get tox into standard builds`

mkoeppe · 2021-04-02T20:21:54Z

comment:11

Moving this ticket to 9.4, as it seems unlikely that it will be merged in 9.3, which is in the release candidate stage

mkoeppe · 2021-07-19T00:44:56Z

comment:12

Setting a new milestone for this ticket based on a cursory review.

mkoeppe · 2021-10-16T20:38:57Z

Dependencies: #32442, #32698

sagetrac-git · 2021-10-16T20:44:34Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

`8af6041`	`Merge tag '9.5.beta2' into t/32442/__configure__make___with_sage_venv_the_default`
`b3f7d34`	`.gitignore: Add /venv`
`fc4b641`	`configure.ac: Remove conveniene symlinks before (re)creating them`
`8bcbf1d`	`Makefile (distclean): Remove convenience symlinks prefix, venv`
`dcb4a08`	`configure.ac: Indicate default in help string for --with-sage-venv`
`e33bc89`	`Merge tag '9.5.beta3' into t/32442/__configure__make___with_sage_venv_the_default`
`2101b8c`	`build/pkgs/python3/spkg-build.in: Make sure that python finds sqlite3 when determining which extension modules to build`
`3bbc5d8`	`build/pkgs/python3/spkg-build.in: Set rpath`
`124b605`	`Merge #32698`
`bca2141`	`pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels`

sagetrac-git · 2021-10-16T20:44:34Z

Changed commit from 188bf2e to bca2141

sagetrac-git · 2021-10-16T20:57:09Z

Branch pushed to git repo; I updated commit sha1. New commits:

`bdbab3a`	`build/make/Makefile.in: Remove SPKG-tox, SPKG-sdist targets`

sagetrac-git · 2021-10-16T20:57:09Z

Changed commit from bca2141 to bdbab3a

mkoeppe · 2021-10-16T20:58:10Z

comment:17

Reduced the scope of the ticket

orlitzky · 2021-12-01T12:12:07Z

comment:18

I've tried to test these locally but I always get some complaint about sage_setup.py. For example,

$ EXTRA_CONFIGURE_ARGS="--without-system-python3" tox -e local-direct -- build sagelib-tox-python-sagewheels

This proceeded to build everything inside of .tox/local-direct, but then...

cd '/home/mjo/src/sage.git/build/pkgs/sagelib/src' && export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/12/bin:/home/mjo/bin:/home/mjo/bin" && SAGE_SPKG_WHEELS=/home/mjo/src/sage.git/.tox/local-direct/local/var/lib/sage/wheels tox -v -v -v -e python-sagewheels
using tox.ini: /home/mjo/src/sage.git/pkgs/sagemath-standard/tox.ini (pid 12133)
  removing /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/log
using tox-3.24.4 from /usr/lib/python3.9/site-packages/tox/__init__.py (pid 12133)
GLOB start: packaging 
GLOB sdist-make: /home/mjo/src/sage.git/pkgs/sagemath-standard/setup.py
[12140] /home/mjo/src/sage.git/pkgs/sagemath-standard$ /usr/bin/python3.9 setup.py sdist --formats=zip --dist-dir /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/dist >.tox/log/GLOB-0.log
ERROR: invocation failed (exit code 1), logfile: /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/log/GLOB-0.log
================================== log start ===================================
Traceback (most recent call last):
  File "/home/mjo/src/sage.git/pkgs/sagemath-standard/setup.py", line 31, in <module>
    from sage_setup.excepthook import excepthook
ModuleNotFoundError: No module named 'sage_setup'

=================================== log end ====================================
ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python3.9 setup.py sdist --formats=zip --dist-dir /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/dist', 1)
make[1]: *** [Makefile:2798: sagelib-tox-python-sagewheels] Error 2
make[1]: Leaving directory '/home/mjo/src/sage.git/build/make'

real	0m4.988s
user	0m1.223s
sys	0m0.145s
***************************************************************
Error building Sage.

The following package(s) may have failed to build (not necessarily
during this run of 'make sagelib-tox-python-sagewheels'):

It is safe to delete any log files and build directories, but they
contain information that is helpful for debugging build problems.
WARNING: If you now run 'make' again, the build directory of the
same version of the package will, by default, be deleted. Set the
environment variable SAGE_KEEP_BUILT_SPKGS=yes to prevent this.

make: *** [Makefile:39: sagelib-tox-python-sagewheels] Error 1
ERROR: InvocationError for command /bin/bash -c 'export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/12/bin:/home/mjo/bin:/home/mjo/bin && : &&  case "" in 1|y*|Y*);; *) ./bootstrap ;; esac &&  : &&  case "" in 1|y*|Y*);; *) ./configure --prefix=/home/mjo/src/sage.git/.tox/local-direct/local --enable-experimental-packages --enable-download-from-upstream-url  --with-system-python3=yes  --without-system-python3 ;; esac &&  case "build sagelib-tox-python-sagewheels" in  bash)    bash -i; exit ;;  config*) ;;  *)       make -k V=0 base-toolchain ;;  esac &&  make -k V=0 SAGE_CHECK=warn SAGE_CHECK_PACKAGES="!cython,!r,!python3,!gap,!cysignals,!linbox,!git,!ppl,!cmake,!rpy2,!sage_sws2rst"  build sagelib-tox-python-sagewheels && ( [ -z "" ] || make -k V=0 SAGE_CHECK=warn SAGE_CHECK_PACKAGES="!cython,!r,!python3,!gap,!cysignals,!linbox,!git,!ppl,!cmake,!rpy2,!sage_sws2rst"  || echo "(error ignored)" ) ' (exited with code 2)
__________________________________________________________________________________ summary __________________________________________________________________________________
ERROR:   local-direct: commands failed

Is that expected (did I construct my example wrong?)

mkoeppe · 2021-12-01T17:34:13Z

Changed dependencies from #32442, #32698 to #32899

sagetrac-git · 2021-12-01T17:35:01Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`e5ad518`	`pkgs/sagemath-standard/setup.py: Do not depend on sage_setup for sdist or egg_info`
`821d16e`	`src/doc/en/developer/packaging_sage_library.rst: Add bootstrap to testing instructions to avoid the trap of #32868`
`77c957f`	`src/doc/en/developer/packaging_sage_library.rst: Put hierarchy section one level higher`
`9375ef8`	`pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels`
`c6fc111`	`Prettier diagram`
`7d6a44c`	`Use :mod: as markup for packages/modules`
`f83d424`	`Improve ABC example`
`817a8d0`	`src/doc/en/developer/packaging_sage_library.rst: More :mod: and :class: markup`
`b555735`	`src/doc/en/developer/packaging_sage_library.rst: Link to pypi.org and to documentation of packaging metadata`
`ad12d25`	`Merge #32899`

sagetrac-git · 2021-12-01T17:35:01Z

Changed commit from bdbab3a to ad12d25

mkoeppe · 2021-12-01T17:36:43Z

comment:21

Should be better with this branch

mkoeppe · 2021-12-01T17:36:55Z

comment:22

(needs ./bootstrap)

orlitzky · 2021-12-03T21:30:54Z

comment:23

I just noticed these as well, probably from an empty $CC --version in sage_setup. No idea if they're new, I probably just happened to be looking in the right place at exactly the right time:

  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Running command /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/python-s\
agewheels-nopypi/bin/python /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/\
python-sagewheels-nopypi/lib/python3.9/site-packages/pip/_vendor/pep517/in_proc\
ess/_in_process.py prepare_metadata_for_build_wheel /tmp/tmphl6_9kin
    /bin/sh: 1: --version: not found

  Building wheel for sagemath-standard (PEP 517): started
  Running command /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/python-sag\
ewheels-nopypi/bin/python /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/py\
thon-sagewheels-nopypi/lib/python3.9/site-packages/pip/_vendor/pep517/in_proces\
s/_in_process.py build_wheel /tmp/tmpqovldekw
  /bin/sh: 1: --version: not found

mkoeppe · 2021-12-04T00:08:06Z

comment:24

Yes, this is #30876

orlitzky · 2021-12-04T01:29:16Z

Reviewer: Michael Orlitzky

orlitzky · 2021-12-04T01:29:16Z

comment:25

Looking better now; regardless, the Makefile targets themselves are working as intended.

mkoeppe · 2021-12-04T01:39:04Z

comment:26

Thanks!

vbraun · 2021-12-23T20:26:10Z

Changed branch from u/mkoeppe/new_make_targets__sagelib_sdist____sage_docbuild_sdist_______and__sagelib_tox__________ to ad12d25

mkoeppe added this to the sage-9.3 milestone Mar 22, 2021

mkoeppe added c: porting labels Mar 22, 2021

mkoeppe added the s: needs review label Mar 22, 2021

This comment has been minimized.

Sign in to view

mkoeppe added s: needs work and removed s: needs review labels Mar 27, 2021

mkoeppe modified the milestones: sage-9.3, sage-9.4 Apr 2, 2021

mkoeppe modified the milestones: sage-9.4, sage-9.5 Jul 19, 2021

mkoeppe removed the s: needs work label Oct 16, 2021

mkoeppe added the s: needs review label Oct 16, 2021

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~New make targets "sagelib-sdist", "sage_docbuild-sdist", ... and "sagelib-tox-...", ...~~ New make targets "sagelib-tox-...", "sage_docbuild-tox-...", ... Oct 16, 2021

orlitzky added s: positive review and removed s: needs review labels Dec 4, 2021

vbraun removed the s: positive review label Dec 23, 2021

vbraun closed this as completed in 0320f4f Dec 23, 2021

mkoeppe mentioned this issue Jan 29, 2023

Meta-ticket: Refactor and improve system package related scripts, tox.ini, build/bin/write_dockerfile.sh #29146

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New make targets "sagelib-tox-...", "sage_docbuild-tox-...", ... #31535

New make targets "sagelib-tox-...", "sage_docbuild-tox-...", ... #31535

mkoeppe commented Mar 22, 2021

mkoeppe commented Mar 22, 2021

mkoeppe commented Mar 22, 2021

mkoeppe commented Mar 22, 2021

mkoeppe commented Mar 22, 2021

sagetrac-git mannequin commented Mar 22, 2021

sagetrac-git mannequin commented Mar 22, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sagetrac-git mannequin commented Mar 22, 2021

sagetrac-git mannequin commented Mar 22, 2021

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Apr 2, 2021

mkoeppe commented Jul 19, 2021

mkoeppe commented Oct 16, 2021

sagetrac-git mannequin commented Oct 16, 2021

sagetrac-git mannequin commented Oct 16, 2021

sagetrac-git mannequin commented Oct 16, 2021

sagetrac-git mannequin commented Oct 16, 2021

mkoeppe commented Oct 16, 2021

This comment has been minimized.

orlitzky commented Dec 1, 2021

mkoeppe commented Dec 1, 2021

sagetrac-git mannequin commented Dec 1, 2021

sagetrac-git mannequin commented Dec 1, 2021

mkoeppe commented Dec 1, 2021

mkoeppe commented Dec 1, 2021

orlitzky commented Dec 3, 2021

mkoeppe commented Dec 4, 2021

orlitzky commented Dec 4, 2021

orlitzky commented Dec 4, 2021

mkoeppe commented Dec 4, 2021

vbraun commented Dec 23, 2021

New make targets "sagelib-tox-...", "sage_docbuild-tox-...", ... #31535

New make targets "sagelib-tox-...", "sage_docbuild-tox-...", ... #31535

Comments

mkoeppe commented Mar 22, 2021

mkoeppe commented Mar 22, 2021

mkoeppe commented Mar 22, 2021

mkoeppe commented Mar 22, 2021

mkoeppe commented Mar 22, 2021

sagetrac-git mannequin commented Mar 22, 2021

sagetrac-git mannequin commented Mar 22, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sagetrac-git mannequin commented Mar 22, 2021

sagetrac-git mannequin commented Mar 22, 2021

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Apr 2, 2021

mkoeppe commented Jul 19, 2021

mkoeppe commented Oct 16, 2021

sagetrac-git mannequin commented Oct 16, 2021

sagetrac-git mannequin commented Oct 16, 2021

sagetrac-git mannequin commented Oct 16, 2021

sagetrac-git mannequin commented Oct 16, 2021

mkoeppe commented Oct 16, 2021

This comment has been minimized.

orlitzky commented Dec 1, 2021

mkoeppe commented Dec 1, 2021

sagetrac-git mannequin commented Dec 1, 2021

sagetrac-git mannequin commented Dec 1, 2021

mkoeppe commented Dec 1, 2021

mkoeppe commented Dec 1, 2021

orlitzky commented Dec 3, 2021

mkoeppe commented Dec 4, 2021

orlitzky commented Dec 4, 2021

orlitzky commented Dec 4, 2021

mkoeppe commented Dec 4, 2021

vbraun commented Dec 23, 2021