Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modularization of sagelib: Distributions sagemath-{brial,combinat,eclib,flint,gap,giac,glpk,graphs,groups,homfly,lcalc,libbraiding,libecm,linbox,modules,mpmath,ntl,pari,plot,polyhedra,schemes,singular,standard-no-symbolics,symbolics} #35095

Draft
wants to merge 4,002 commits into
base: develop
Choose a base branch
from

Conversation

mkoeppe
Copy link
Member

@mkoeppe mkoeppe commented Feb 12, 2023

📚 Description

Screenshot 2023-09-09 at 11 57 02 PM

📝 Checklist

  • I have made sure that the title is self-explanatory and the description concisely explains the PR.
  • I have linked an issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation accordingly.

⌛ Dependencies

This is part of:

@mkoeppe mkoeppe changed the title Modularization of sagelib: Break out a separate package **sagemath-polyhedra** Modularization of sagelib: Break out a separate package sagemath-polyhedra Feb 12, 2023
@mkoeppe mkoeppe added this to the sage-9.9 milestone Feb 12, 2023
@codecov-commenter
Copy link

codecov-commenter commented Feb 14, 2023

Codecov Report

Patch coverage: 63.80% and project coverage change: -1.22 ⚠️

Comparison is base (c00e6c2) 88.62% compared to head (ef82aaa) 87.40%.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop   #35095      +/-   ##
===========================================
- Coverage    88.62%   87.40%   -1.22%     
===========================================
  Files         2148     2180      +32     
  Lines       398855   399144     +289     
===========================================
- Hits        353480   348878    -4602     
- Misses       45375    50266    +4891     
Impacted Files Coverage Δ
src/sage/all__sagemath_categories.py 0.00% <0.00%> (ø)
src/sage/all__sagemath_polyhedra.py 0.00% <0.00%> (ø)
src/sage/arith/all.py 100.00% <ø> (ø)
src/sage/arith/misc.py 90.49% <ø> (-0.50%) ⬇️
src/sage/calculus/test_sympy.py 0.00% <ø> (ø)
src/sage/categories/additive_magmas.py 97.77% <ø> (ø)
src/sage/categories/additive_semigroups.py 100.00% <ø> (ø)
src/sage/categories/affine_weyl_groups.py 100.00% <ø> (ø)
src/sage/categories/algebra_functor.py 80.00% <ø> (-20.00%) ⬇️
src/sage/categories/algebras.py 100.00% <ø> (ø)
... and 232 more

... and 283 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@mkoeppe mkoeppe force-pushed the t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra branch 2 times, most recently from ee6eb49 to d368b6e Compare February 17, 2023 19:49
@mkoeppe mkoeppe self-assigned this Feb 18, 2023
@kwankyu
Copy link
Collaborator

kwankyu commented Feb 22, 2023

I ran

./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-polyhedra && SAGE_NUM_THREADS=16 tox -v -v -v)'

after full rebuild on macOS monterey, and got

            [37/54] gcc -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I./sage/cpython -I/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-build-env-_on11jf5/overlay/lib/python3.11/site-packages/gmpy2 -I./sage/rings -I/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/pip-build-env-_on11jf5/overlay/lib/python3.11/site-packages -I/usr/local/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -Ibuild/cythonized -I/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-polyhedra/.tox/.pkg/include -I/usr/local/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c build/cythonized/sage/structure/coerce.c -o build/temp.macosx-12-x86_64-cpython-311/build/cythonized/sage/structure/coerce.o -fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1
            build/cythonized/sage/structure/coerce.c:779:10: fatal error: 'gmp.h' file not found
            #include "gmp.h"
                     ^~~~~~~
            1 error generated.

@mkoeppe
Copy link
Member Author

mkoeppe commented Feb 22, 2023

I've updated the instructions for testing.

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 23, 2023

sagemath-polyhedra fails to build for me. It says gmp.h is missing.

@mkoeppe
Copy link
Member Author

mkoeppe commented Feb 23, 2023

Is this on macOS with homebrew? DId you run .homebrew-build-env?

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 23, 2023

Yes. My bash profile does source /Users/kwankyu/GitHub/sage-dev/.homebrew-build-env.

I manually pasted the line and tried again. But I got the same.

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 23, 2023

I am trying again after deleting pkgs/sagemath-polyhedra/.tox directory.

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 23, 2023

I got the same failure. Part of the log

.pkg: 89422 W install_requires> python -I -m pip install 'cysignals>=1.10.2' 'cython<1.0,>=0.29.21' 'gmpy2>=2.1.0' memory_allocator 'pkgconfig>=1.5.1' 'sage-setup~=10.0b1' 'sagemath-categories~=10.0b1' 'sagemath-environment~=10.0b1' 'setuptools>=49.6.0' 'wheel>=0.36.2' [tox/tox_env/api.py:428]
Collecting cysignals>=1.10.2
  Using cached cysignals-1.11.2-cp311-cp311-macosx_12_0_x86_64.whl
Collecting cython<1.0,>=0.29.21
  Using cached Cython-0.29.33-py2.py3-none-any.whl (987 kB)
Collecting gmpy2>=2.1.0
  Using cached gmpy2-2.1.5-cp311-cp311-macosx_10_9_x86_64.whl (806 kB)
Collecting memory_allocator
  Using cached memory_allocator-0.1.3.tar.gz (19 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pkgconfig>=1.5.1
  Using cached pkgconfig-1.5.5-py3-none-any.whl (6.7 kB)
Collecting sage-setup~=10.0b1
  Using cached sage_setup-10.0b1-py3-none-any.whl
Collecting sagemath-categories~=10.0b1
  Using cached sagemath-categories-10.0b1.tar.gz (504 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [946 lines of output]

@mkoeppe
Copy link
Member Author

mkoeppe commented Feb 23, 2023

Have you tried the command make wheels?

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 23, 2023

No. Shall I?

@mkoeppe
Copy link
Member Author

mkoeppe commented Feb 23, 2023

Yes, that's what I would recommend as the best way of testing

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 23, 2023

I ran make wheels. It ended with

[sagemath_objects-10.0.beta1] * Creating venv isolated environment...
[sagemath_objects-10.0.beta1] * Installing packages in isolated environment... (cysignals >=1.10.2, cython >=0.29.21, <1.0, gmpy2 >=2.1.0, sage-setup ~= 10.0b1, sagemath-environment ~= 10.0b1, setuptools >=49.6.0, wheel >=0.36.2)
[sagemath_objects-10.0.beta1] Looking in links: file:///Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels
[sagemath_objects-10.0.beta1] Processing /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/sagemath_environment-10.0b1-py3-none-any.whl
[sagemath_objects-10.0.beta1] Processing /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/setuptools-65.6.3-py3-none-any.whl
[sagemath_objects-10.0.beta1] Processing /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/Cython-0.29.32-cp311-cp311-macosx_12_0_x86_64.whl
[sagemath_objects-10.0.beta1] Processing /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/gmpy2-2.1.2-cp311-cp311-macosx_12_0_x86_64.whl
[sagemath_objects-10.0.beta1] ERROR: Could not find a version that satisfies the requirement sage-setup~=10.0b1 (from versions: none)
[sagemath_objects-10.0.beta1] ERROR: No matching distribution found for sage-setup~=10.0b1
[sagemath_objects-10.0.beta1] 
[sagemath_objects-10.0.beta1] Traceback (most recent call last):
[sagemath_objects-10.0.beta1]   File "/Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages/build/__main__.py", line 375, in main
[sagemath_objects-10.0.beta1]     built = build_call(
[sagemath_objects-10.0.beta1]             ^^^^^^^^^^^
[sagemath_objects-10.0.beta1]   File "/Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages/build/__main__.py", line 235, in build_package_via_sdist
[sagemath_objects-10.0.beta1]     sdist = _build(isolation, builder, outdir, 'sdist', config_settings, skip_dependency_check)
[sagemath_objects-10.0.beta1]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[sagemath_objects-10.0.beta1]   File "/Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages/build/__main__.py", line 145, in _build
[sagemath_objects-10.0.beta1]     return _build_in_isolated_env(builder, outdir, distribution, config_settings)
[sagemath_objects-10.0.beta1]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[sagemath_objects-10.0.beta1]   File "/Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages/build/__main__.py", line 113, in _build_in_isolated_env
[sagemath_objects-10.0.beta1]     env.install(builder.build_system_requires)
[sagemath_objects-10.0.beta1]   File "/Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages/build/env.py", line 214, in install
[sagemath_objects-10.0.beta1]     _subprocess(cmd)
[sagemath_objects-10.0.beta1]   File "/Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages/build/env.py", line 79, in _subprocess
[sagemath_objects-10.0.beta1]     raise e
[sagemath_objects-10.0.beta1]   File "/Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages/build/env.py", line 76, in _subprocess
[sagemath_objects-10.0.beta1]     subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
[sagemath_objects-10.0.beta1]   File "/usr/local/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
[sagemath_objects-10.0.beta1]     raise CalledProcessError(retcode, process.args,
[sagemath_objects-10.0.beta1] subprocess.CalledProcessError: Command '['/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/build-env-y415lely/bin/python', '-Im', 'pip', 'install', '--use-pep517', '--no-warn-script-location', '-r', '/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/build-reqs-t6wqxz1a.txt']' returned non-zero exit status 1.
[sagemath_objects-10.0.beta1] 
[sagemath_objects-10.0.beta1] ERROR Command '['/private/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/build-env-y415lely/bin/python', '-Im', 'pip', 'install', '--use-pep517', '--no-warn-script-location', '-r', '/var/folders/td/fw1q9ljs311ggyph77rs53_40000gn/T/build-reqs-t6wqxz1a.txt']' returned non-zero exit status 1.
[sagemath_objects-10.0.beta1] ********************************************************************************
[sagemath_objects-10.0.beta1] Failure building sdist and wheel
[sagemath_objects-10.0.beta1] ********************************************************************************
make[3]: *** [sagemath_objects-SAGE_VENV-no-deps] Error 1
make[2]: *** [/Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/var/lib/sage/installed/sagemath_objects-10.0.beta1] Error 2
make[1]: *** [wheels] Error 2

real	0m39.310s
user	0m29.301s
sys	0m8.784s
***************************************************************
Error building Sage.

The following package(s) may have failed to build (not necessarily
during this run of 'make wheels'):

It is safe to delete any log files and build directories, but they
contain information that is helpful for debugging build problems.
WARNING: If you now run 'make' again, the build directory of the
same version of the package will, by default, be deleted. Set the
environment variable SAGE_KEEP_BUILT_SPKGS=yes to prevent this.

make: *** [wheels] Error 1

@mkoeppe
Copy link
Member Author

mkoeppe commented Feb 23, 2023

Could you try if ./bootstrap fixes this problem?

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 23, 2023

I ran ./bootstrap && make wheels. I got the same error.

@mkoeppe
Copy link
Member Author

mkoeppe commented Feb 23, 2023

Could you check the contents of venv/var/lib/sage/wheels/sage*.whl please?

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 23, 2023

$ ls venv/var/lib/sage/wheels/sage*.whl
venv/var/lib/sage/wheels/sage_sws2rst-10.0b1-py3-none-any.whl
venv/var/lib/sage/wheels/sagemath_environment-10.0b1-py3-none-any.whl
venv/var/lib/sage/wheels/sagenb_export-3.3-py3-none-any.whl
venv/var/lib/sage/wheels/sagetex-3.6.1-py3-none-any.whl

@mkoeppe
Copy link
Member Author

mkoeppe commented Feb 23, 2023

The sage_setup wheel is missing, which should have been one of the first wheels that make wheels makes.

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 23, 2023

Part of the log:

make --no-print-directory SAGE_EDITABLE=no SAGE_WHEELS=yes sage_sws2rst sage_setup sagemath_environment sagemath_objects sagemath_repl sagemath_categories sagemath_polyhedra sage_conf sagelib sage_docbuild
make[2]: Nothing to be done for `sage_sws2rst'.
make[2]: Nothing to be done for `sage_setup'.
make --no-print-directory python_build-no-deps
sage-logger -p 'sage --pip install -r "/Users/kwankyu/GitHub/sage-dev/build/pkgs/python_build/requirements.txt"' '/Users/kwankyu/GitHub/sage-dev/logs/pkgs/python_build.log'
[python_build] Requirement already satisfied: build in /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages (from -r /Users/kwankyu/GitHub/sage-dev/build/pkgs/python_build/requirements.txt (line 1)) (0.10.0)
[python_build] Requirement already satisfied: packaging>=19.0 in /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages (from build->-r /Users/kwankyu/GitHub/sage-dev/build/pkgs/python_build/requirements.txt (line 1)) (21.3)
[python_build] Requirement already satisfied: pyproject_hooks in /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages (from build->-r /Users/kwankyu/GitHub/sage-dev/build/pkgs/python_build/requirements.txt (line 1)) (1.0.0)
[python_build] Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/lib/python3.11/site-packages (from packaging>=19.0->build->-r /Users/kwankyu/GitHub/sage-dev/build/pkgs/python_build/requirements.txt (line 1)) (3.0.9)
make --no-print-directory sagemath_objects-SAGE_VENV-no-deps
[sagemath_objects-10.0.beta1] * Creating venv isolated environment...
[sagemath_objects-10.0.beta1] * Installing packages in isolated environment... (cysignals >=1.10.2, cython >=0.29.21, <1.0, gmpy2 >=2.1.0, sage-setup ~= 10.0b1, sagemath-environment ~= 10.0b1, setuptools >=49.6.0, wheel >=0.36.2)
[sagemath_objects-10.0.beta1] Looking in links: file:///Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels
[sagemath_objects-10.0.beta1] Processing /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/wheel-0.38.4-py3-none-any.whl
[sagemath_objects-10.0.beta1] Processing /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/sagemath_environment-10.0b1-py3-none-any.whl
[sagemath_objects-10.0.beta1] Processing /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/gmpy2-2.1.2-cp311-cp311-macosx_12_0_x86_64.whl
[sagemath_objects-10.0.beta1] Processing /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/cysignals-1.11.2-cp311-cp311-macosx_12_0_x86_64.whl
[sagemath_objects-10.0.beta1] Processing /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/setuptools-65.6.3-py3-none-any.whl
[sagemath_objects-10.0.beta1] ERROR: Could not find a version that satisfies the requirement sage-setup~=10.0b1 (from versions: none)
[sagemath_objects-10.0.beta1] ERROR: No matching distribution found for sage-setup~=10.0b1
[sagemath_objects-10.0.beta1] 

@kwankyu
Copy link
Collaborator

kwankyu commented Feb 23, 2023

It says Nothing to be done for sage_setup!

vbraun pushed a commit to vbraun/sage that referenced this pull request Apr 20, 2024
…arate jobs for pyright, build, modularized tests, long tests

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

<!-- Why is this change required? What problem does it solve? -->
Running the containers explicitly, instead of using the declarative
`container:` feature of GH Actions gives us more control:
- we can create more space on the host if necessary; we just scraped by
an out of space condition in sagemath#36473 /
sagemath#36469 (comment)
- we can run some operations outside of the container but in the same
job; this will make the separate "Get CI fixes" jobs unnecessary,
addressing the cosmetic concerns from
sagemath#36338 (comment),
sagemath#36349
- it enables caching between the various workflows (as first discussed
in sagemath#36446).

We split out static type checking with pyright into its separate
workflow:
- **pyright.yml**: As static checking does not need  a build of the Sage
library, for PRs that do not make any changes to packages, there's
nothing to build, and the workflow gives a fast turnaround just after 10
minutes. For applying the CI fixes from blocker tickets, this workflow
uses the technique favored in sagemath#36349.

The workflow **build.yml** first launches a job:
- **test-new:** It builds incrementally (using a tox-generated
`Dockerfile` and https://github.com/marketplace/actions/build-and-push-
docker-images) and does the quick incremental test. This completes
within 10 to 20 minutes when there's no change.

After this is completed, more jobs are launched:
- **test-mod:** It again builds incrementally and tests a modularized
distribution. Later (with more from sagemath#35095), more jobs will be added to
this matrix job for other distributions.
- **test-long:** It again builds incrementally and runs the long test.

The workflows **doc-build.yml** and **doc-build-pdf.yml** again build
incrementally and then build the documentation. The diffing code for the
HTML documentation now runs in the host, not the container, which
simplifies things. (To show that diffing still works, we include a small
change to the Sage library.)

Splitting the workflow jobs implements @kwankyu's  suggestion from:
- sagemath#35652 (comment)
(Fixes sagemath#35814)

The steps "again builds incrementally" actually use the GH Actions
cache, https://docs.docker.com/build/ci/github-actions/cache/#cache-
backend-api. When nothing is cached and the 3 workflows are launched in
parallel, they may each build the same thing. But when there's
congestion that leads to the workflows to be run serially, the 2nd and
3rd workflow will be able to use the cache from the 1st workflow. This
elasticity may be helpful in reducing congestion.

There is a rather small per-project limit of 10 GB for this cache, so
we'll have to see how effectively caching works in practice. See
https://github.com/sagemath/sage/actions/caches


<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
- Depends on sagemath#36938 (merged here)
- Depends on sagemath#36951 (merged here)
- Depends on sagemath#37351 (merged here)
- Depends on sagemath#37750 (merged here)

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36498
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee, Matthias Köppe
vbraun pushed a commit to vbraun/sage that referenced this pull request Apr 20, 2024
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Cherry-picked from sagemath#35095.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37714
Reported by: Matthias Köppe
Reviewer(s): Matthias Köppe, Sebastian A. Spindler
vbraun pushed a commit to vbraun/sage that referenced this pull request Apr 20, 2024
… after sagemath#37022

    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

- Cherry-picked updates of testing infrastructure from sagemath#35095.
- Add upper version bounds to avoid a regression in 4.14.1
- Fixes sagemath#37734.
- Also removes some unnecessary runs of the "CI Linux Incremental"
workflow.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37750
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee
vbraun pushed a commit to vbraun/sage that referenced this pull request Apr 25, 2024
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Cherry-picked from sagemath#35095.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37714
Reported by: Matthias Köppe
Reviewer(s): Matthias Köppe, Sebastian A. Spindler
mkoeppe added a commit to mkoeppe/sage that referenced this pull request Apr 27, 2024
mkoeppe added a commit to mkoeppe/sage that referenced this pull request Apr 27, 2024
vbraun pushed a commit to vbraun/sage that referenced this pull request Apr 28, 2024
… after sagemath#37022

    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

- Cherry-picked updates of testing infrastructure from sagemath#35095.
- Add upper version bounds to avoid a regression in 4.14.1
- Fixes sagemath#37734.
- Also removes some unnecessary runs of the "CI Linux Incremental"
workflow.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37750
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee
vbraun pushed a commit to vbraun/sage that referenced this pull request May 2, 2024
…arate jobs for pyright, build, modularized tests, long tests

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

<!-- Why is this change required? What problem does it solve? -->
Running the containers explicitly, instead of using the declarative
`container:` feature of GH Actions gives us more control:
- we can create more space on the host if necessary; we just scraped by
an out of space condition in sagemath#36473 /
sagemath#36469 (comment)
- we can run some operations outside of the container but in the same
job; this will make the separate "Get CI fixes" jobs unnecessary,
addressing the cosmetic concerns from
sagemath#36338 (comment),
sagemath#36349
- it enables caching between the various workflows (as first discussed
in sagemath#36446).

We split out static type checking with pyright into its separate
workflow:
- **pyright.yml**: As static checking does not need  a build of the Sage
library, for PRs that do not make any changes to packages, there's
nothing to build, and the workflow gives a fast turnaround just after 10
minutes. For applying the CI fixes from blocker tickets, this workflow
uses the technique favored in sagemath#36349.

The workflow **build.yml** first launches a job:
- **test-new:** It builds incrementally (using a tox-generated
`Dockerfile` and https://github.com/marketplace/actions/build-and-push-
docker-images) and does the quick incremental test. This completes
within 10 to 20 minutes when there's no change.

After this is completed, more jobs are launched:
- **test-mod:** It again builds incrementally and tests a modularized
distribution. Later (with more from sagemath#35095), more jobs will be added to
this matrix job for other distributions.
- **test-long:** It again builds incrementally and runs the long test.

The workflows **doc-build.yml** and **doc-build-pdf.yml** again build
incrementally and then build the documentation. The diffing code for the
HTML documentation now runs in the host, not the container, which
simplifies things. (To show that diffing still works, we include a small
change to the Sage library.)

Splitting the workflow jobs implements @kwankyu's  suggestion from:
- sagemath#35652 (comment)
(Fixes sagemath#35814)

The steps "again builds incrementally" actually use the GH Actions
cache, https://docs.docker.com/build/ci/github-actions/cache/#cache-
backend-api. When nothing is cached and the 3 workflows are launched in
parallel, they may each build the same thing. But when there's
congestion that leads to the workflows to be run serially, the 2nd and
3rd workflow will be able to use the cache from the 1st workflow. This
elasticity may be helpful in reducing congestion.

There is a rather small per-project limit of 10 GB for this cache, so
we'll have to see how effectively caching works in practice. See
https://github.com/sagemath/sage/actions/caches


<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
- Depends on sagemath#36938 (merged here)
- Depends on sagemath#36951 (merged here)
- Depends on sagemath#37351 (merged here)
- Depends on sagemath#37750 (merged here)
- Depends on sagemath#37277 (merged here)

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36498
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee, Matthias Köppe
vbraun pushed a commit to vbraun/sage that referenced this pull request May 2, 2024
…string/doctest cosmetics, `# needs`

    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Cherry-picked from sagemath#35095.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37715
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee, Matthias Köppe
vbraun pushed a commit to vbraun/sage that referenced this pull request May 2, 2024
… after sagemath#37022

    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

- Cherry-picked updates of testing infrastructure from sagemath#35095.
- Add upper version bounds to avoid a regression in 4.14.1
- Fixes sagemath#37734.
- Also removes some unnecessary runs of the "CI Linux Incremental"
workflow.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37750
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee
vbraun pushed a commit to vbraun/sage that referenced this pull request May 4, 2024
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Cherry-picked from sagemath#35095.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37716
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee
vbraun pushed a commit to vbraun/sage that referenced this pull request May 9, 2024
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Cherry-picked from sagemath#35095.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37713
Reported by: Matthias Köppe
Reviewer(s): Martin Rubey
vbraun pushed a commit to vbraun/sage that referenced this pull request May 9, 2024
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Cherry-picked from sagemath#35095.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37716
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee
@mantepse mantepse mentioned this pull request May 10, 2024
vbraun pushed a commit to vbraun/sage that referenced this pull request May 11, 2024
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Cherry-picked from sagemath#35095.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37713
Reported by: Matthias Köppe
Reviewer(s): Martin Rubey
vbraun pushed a commit to vbraun/sage that referenced this pull request May 11, 2024
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Cherry-picked from sagemath#35095.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37716
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee
vbraun pushed a commit to vbraun/sage that referenced this pull request May 12, 2024
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Cherry-picked from sagemath#35095.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37713
Reported by: Matthias Köppe
Reviewer(s): Martin Rubey
vbraun pushed a commit to vbraun/sage that referenced this pull request May 12, 2024
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Cherry-picked from sagemath#35095.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37716
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee
mkoeppe added a commit to mkoeppe/sage that referenced this pull request May 12, 2024
mkoeppe added a commit to mkoeppe/sage that referenced this pull request May 12, 2024
mkoeppe added a commit to mkoeppe/sage that referenced this pull request May 12, 2024
mkoeppe added a commit to mkoeppe/sage that referenced this pull request May 12, 2024
vbraun pushed a commit to vbraun/sage that referenced this pull request May 12, 2024
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Cherry-picked from sagemath#35095.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37713
Reported by: Matthias Köppe
Reviewer(s): Martin Rubey
vbraun pushed a commit to vbraun/sage that referenced this pull request May 12, 2024
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Cherry-picked from sagemath#35095.

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#37716
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment