Meta-ticket: Refactor and improve system package related scripts, tox.ini, build/bin/write_dockerfile.sh

[mkoeppe]

Improvements to low-level system packages scripts:

• Add more homebrew system package information, including packages without spkg-configure.m4 #30861, sage_bootstrap: Update/extend "sage -package", fix "make download" #30865, src/doc/bootstrap: Simplify by using new options of "sage -package list" #30947, tox.ini, build/bin/write-dockerfile.sh: Simplify by using new options of "sage -package list" #30951, bootstrap: Simplify by using new options of "sage -package list" #30968: Refactor system package code in tox.ini, build/bin/write_dockerfile.sh by extending and using sage_bootstrap
• Refactor system package scripts #33873: Extend build/bin/sage-print-system-package-command so that it can provide all distribution-specific commands such as apt-get install for build/bin/write_dockerfile.sh
• sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}" #29041 sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}"
• sage-package: Add commands "dependencies", "trees" #33860 sage-package: Add commands "dependencies", "trees"

Improvements to the system package database:

• Add script packages build/pkgs/_prereq, build/pkgs/_bootstrap #29124 Add script packages build/pkgs/_prereq, build/pkgs/_toolchain, build/pkgs/_bootstrap
• System package quoting fixes #36910 define shell quoting in files like debian.txt
• handle options correctly when installing packages one by one, fix up passing installation options when IGNORE_MISSING_PACKAGES=yes

Improvements to tox docker:

• Make docker images from tox/GitHub CI workflow, regular Sage Docker images, gitpod, CoCalc interoperable #29536 Make docker images from GitHub CI workflow and regular Sage Docker images interoperable
• optionally create a script for docker run instead of a Dockerfile (this gives more flexibility for multiarch - see tox.ini update: Fix breakage caused by src/ext move, add 32-bit linux environments and other architectures supported by docker #29143)
• cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
• move log extraction code from tox.yml to tox.ini (Make Docker images generated by GitHub CI workflows available; minor CI fixes #29530 did some preparation already)
• system package installation should retry a few times when installation errors because of network errors occur. In particular for docker because an incomplete step will be cached
• Make obtaining image from container of failing builds more robust: Use LABEL commands in dockerfiles, use docker ps --filter label=.... to look up the container (https://docs.docker.com/engine/reference/commandline/ps/)
• setenv/passenv SAGE_SERVER
• Remove TARGETS_PRE/TARGETS distinction: Instead leave a stamp file and exit from make with an error to trigger the ADD of src/; for example, add sagelib dependency on $(SAGE_SRC)/sage, and add the rule to Makefile:

$(SAGE_SRC)/sage:
	touch .need_src; exit 1

• use environment variable MAKE for docker builds as well, instead of asking users to pass the USE_MAKEFLAGS variable
• rewrite Dockerfile generation via dev container templates (https://github.com/devcontainers/cli)

Improvements to tox local:

• tox.ini (local): Add environment variables to skip system package installs and other steps, add mechanism for a local interactive shell #31216: tox.ini (local): Add environment variables to skip system package installs, mechanism for a local interactive shell
• ci-cygwin*.yml: delegate to tox, add more stages, use more specific SAGE_LOCAL #31064: ci-cygwin*.yml: delegate to tox, add more stages, use more specific SAGE_LOCAL
• Enable VPATH builds (several independent build trees connected to one source tree) #21469: Enable VPATH builds (several independent build trees connected to one source tree)
• tox.ini: Add local-copy #29416: tox.ini: Add local-copy
• the complicated rule for the commands of local consists entirely of workarounds for things that should really be fixed/improved in our Makefiles:
  • build/make/Makefile.in: base-toolchain should be a dependency of every non-base, non-toolchain package #30721: build/make/Makefile.in: base-toolchain should be a dependency of every non-base, non-toolchain package
  • Clean up settings for SAGE_CHECK_PACKAGES_DEFAULT_yes, etc. #32785: tox.ini and build/bin/write-dockerfile.sh use better defaults for SAGE_CHECK_PACKAGES; this should be moved to build/make/Makefile.in (SAGE_CHECK_PACKAGES_DEFAULT_yes).
  • New make targets "sagelib-tox-...", "sage_docbuild-tox-...", ... #31535 New make targets "sagelib-sdist", "sage_docbuild-sdist", ... and "sagelib-tox-...", ...
    Other improvements to tox.ini:
• SAGE_ROOT/tox.ini: Make tox -p auto and tox --listenvs useful again #32764: Make tox -p auto and tox --listenvs useful again (it does not understand the nested braces)
• tox.ini, build/bin/write-dockerfile.sh: Do not use "make -k" by default #31574: tox.ini, build/bin/write-dockerfile.sh: Do not use make -k by default

Improvements to auto-generated parts of the manuals:

• Autogenerated parts of installation guide #29558 Autogenerated parts of installation guide
• Add script package _recommended and generate "recommended system packages" info #29557 Add script package _recommended and generate "recommended system packages" info

Improvements for messages at the end of a ./configure run:

• sage-guess-package-system: Ignore conda if no environment is activated #29560 sage-guess-package-system: Ignore conda if no environment is activated
• Perhaps sage-guess-package-system should report several package systems such as conda ubuntu or homebrew cpan when that makes sense
• At the end of configure, indicate which optional/experimental packages are configured to be installed #29363 - At the end of configure, indicate which optional/experimental packages are configured to be installed
• Improve wording and formatting of configure's recommendation message #30624: Improve wording and formatting of configure's recommendation message
• At the end of configure, show installation hints for non-enabled optional packages separately #29372 - At the end of configure, show installation hints for disabled optional packages separately
• Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
• configure: Change defaults to --with-system-gcc=force, defer error exits after system package info is shown #32060 configure: Change defaults to --with-system-gcc=force, --with-system-python3=force
• Improve configure's recommendations #29586: Improve configure's recommendations
• refine definition of variable SAGE_NEED_SYSTEM_PACKAGES in m4/sage_spkg_collect.m4 #30863: refine definition of variable SAGE_NEED_SYSTEM_PACKAGES in m4/sage_spkg_collect.m4

Tickets regarding sage_bootstrap:

• sage_bootstrap: Remove purported Python 2.6 support #20023 sage_bootstrap: Remove Python 2.6 support
• sage --package: Add commands "list :standard:", "update-latest" (for packages from PyPI), "upload" #20104 sage --package: Add commands list :standard:, update-latest (for packages from PyPI), upload
• Restore "huge" package type #29919 Restore "huge" package type

See also:

• Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages #29060 Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Depends on #29417

CC: @embray @vbraun @dimpase @jhpalmieri @sheerluck @slel @tobiasdiez @seblabbe

Component: build

Branch/Commit: u/mkoeppe/meta_ticket__refactor_and_improve_system_package_related_scripts__tox_ini__build_bin_write_dockerfile_sh @ fd9b928

Issue created by migration from https://trac.sagemath.org/ticket/29146

[mkoeppe]

Dependencies: #29417