CI: Merge workflows into build.yml #36446

mkoeppe · 2023-10-11T02:05:57Z

Simplifying our workflows running on every ticket, as suggested in https://groups.google.com/g/sage-devel/c/Cisdhf3nSbM

Unchanged as of this PR: build, doc-build, doc-build-pdf jobs all build the image. This can be improved by building a container first.

📝 Checklist

The title is concise, informative, and self-explanatory.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

Depends on CI / tox: Update and speed up pyright #36443
Depends on CI: Refactor build*.yml through a new script .ci/retrofit-worktree.sh #36442
Depends on Retrieve upstream tarballs from GitHub release assets #36435
Depends on Fix conda install instructions after setuptools update #36411
Depends on CI docbuild: Do not rebuild sagelib from scratch #36473

…nor versions (environment-3.9.yml etc.) This is convenient for conda-lock. Partial cherry-pick from "Use conda-lock for reproducible conda env"

….. files

…nt-... files

….. files

…: Use configure --enable-system-site-packages

…irror to mirror_list

…g to first use

…ces in SAGE_ROOT/.upstream.d

…h-sage-releases

… now

…ild*.yml

…node installation

… 'sage -tox'

mkoeppe · 2023-10-11T04:41:23Z

@kwankyu Here's an attempt to reorganize the CI workflows. Checks tab: https://github.com/sagemath/sage/pull/36446/checks

This is not the final implementation; the merge of the separate workflows also enables sharing the build phases of build and docbuild workflows via Docker.

mkoeppe · 2023-10-11T05:04:45Z

Also ci-conda could be merged here, but I'm not touching it for the moment because other PRs are working on it

kwankyu · 2023-10-11T06:02:47Z

We may have separate "Build & Test" and "Build & Test using Conda". They are well distinguished. As far as I understand, Building on conda is still not as stable as on the vanilla system. Hence merging them would make "Build & Test" unstable and too big. "Build & Test" is our core workflow, and better to be kept rock-reliable.

…message

…remental build more incremental via git, via git, .ci/retrofit-worktree.sh

…-check`   We add `make` targets for testing wheel-building script packages that have already been installed. For example: ``` $ make SAGE_WHEELS=yes sagemath_categories # builds and stores (but does not install) a wheel $ make sagemath_categories-check # tests the wheel using tox ``` The target `make pypi-wheels-check` calls all of them. We restructure the Build & Test CI so that all tests are run after build has been completed. Later, as a follow up of this and sagemath#36446, we can store the result of Build as a container image and then parallelize the various tests on top of it. This is implemented on top of an improvement of how we record information on our wheels. ``` $ cat venv/var/lib/sage/scripts/sagemath_categories/spkg- requirements.txt sagemath_categories @ file:///Users/mkoeppe/s/sage/sage- rebasing/worktree-algebraic-2018-spring/local/var/lib/sage/venv-python3. 11/var/lib/sage/wheels/sagemath_categories-10.2b6-cp311-cp311- macosx_13_0_x86_64.whl ``` Previously, this information was only known during the execution of the `spkg-install` script. In a follow-up, we can address sagemath#30956 by delaying the wheel installation to spkg-postinst.    ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: sagemath#36452 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

tobiasdiez · 2023-10-18T00:37:08Z

How do you plan to handle the fact that some of the workflows currently have different triggers? In particular, the Linux work flow only being run when there is a label? Similarly, I think it will be hard to trigger the publish doc workflow directly after the docbuild has been finished.

…-check`   We add `make` targets for testing wheel-building script packages that have already been installed. For example: ``` $ make SAGE_WHEELS=yes sagemath_categories # builds and stores (but does not install) a wheel $ make sagemath_categories-check # tests the wheel using tox ``` The target `make pypi-wheels-check` calls all of them. We restructure the Build & Test CI so that all tests are run after build has been completed. Later, as a follow up of this and sagemath#36446, we can store the result of Build as a container image and then parallelize the various tests on top of it. This is implemented on top of an improvement of how we record information on our wheels. ``` $ cat venv/var/lib/sage/scripts/sagemath_categories/spkg- requirements.txt sagemath_categories @ file:///Users/mkoeppe/s/sage/sage- rebasing/worktree-algebraic-2018-spring/local/var/lib/sage/venv-python3. 11/var/lib/sage/wheels/sagemath_categories-10.2b6-cp311-cp311- macosx_13_0_x86_64.whl ``` Previously, this information was only known during the execution of the `spkg-install` script. In a follow-up, we can address sagemath#30956 by delaying the wheel installation to spkg-postinst.    ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: sagemath#36452 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

mkoeppe · 2023-10-19T02:39:16Z

Thanks, these are relevant points to consider.

Current status (mainly for future reference):

Pushing the image to a registry works great for pushes but, as we know, cannot work for PRs because there aren't (and likely should not be) permissions.
Saving the Docker image as an artifact is too slow, much slower than pushing the image to a registry (in which case existing layers are reused). Pushing a multi-gigabyte image via upload-artifact takes very long, so not a suitable approach.
One can of course easily run a local registry (e.g. via the services keyword), but it will only be accessible by a job, not by all jobs in a workflow, so nothing gained.
Saving the changes as an incremental tarfile (https://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html), which I was using successfully in the multi-stage Cygwin workflow, unfortunately also does not work well because of the type of file system used within Docker containers.
Approaches such as https://github.com/aidanhs/dlgrab are possible, but I won't pursue this direction.

Next approach that I'll try:

Docker layer caching via https://docs.docker.com/build/cache/backends/gha/ -- after building the image incrementally, the jobs for docbuild, ptest, etc. will just again rebuild the image incrementally from the same source image using the same steps; when the cache can be retrieved, this will be very fast; otherwise, it just falls back to building the image incrementally (= current status).

mkoeppe · 2023-10-21T22:27:56Z

I'm closing this and will do the docker layer caching in a separate PR; which will keep the workflows separate.

Edit: But first #36498 / #36508.

@kwankyu

…arate jobs for pyright, build, modularized tests, long tests    Running the containers explicitly, instead of using the declarative `container:` feature of GH Actions gives us more control: - we can create more space on the host if necessary; we just scraped by an out of space condition in sagemath#36473 / sagemath#36469 (comment) - we can run some operations outside of the container but in the same job; this will make the separate "Get CI fixes" jobs unnecessary, addressing the cosmetic concerns from sagemath#36338 (comment), sagemath#36349 - it enables caching between the various workflows (as first discussed in sagemath#36446). We split out static type checking with pyright into its separate workflow: - **pyright.yml**: As static checking does not need a build of the Sage library, for PRs that do not make any changes to packages, there's nothing to build, and the workflow gives a fast turnaround just after 10 minutes. For applying the CI fixes from blocker tickets, this workflow uses the technique favored in sagemath#36349. The workflow **build.yml** first launches a job: - **test-new:** It builds incrementally (using a tox-generated `Dockerfile` and https://github.com/marketplace/actions/build-and-push- docker-images) and does the quick incremental test. This completes within 10 to 20 minutes when there's no change. After this is completed, more jobs are launched: - **test-mod:** It again builds incrementally and tests a modularized distribution. Later (with more from sagemath#35095), more jobs will be added to this matrix job for other distributions. - **test-long:** It again builds incrementally and runs the long test. The workflows **doc-build.yml** and **doc-build-pdf.yml** again build incrementally and then build the documentation. The diffing code for the HTML documentation now runs in the host, not the container, which simplifies things. (To show that diffing still works, we include a small change to the Sage library.) Splitting the workflow jobs implements @kwankyu's suggestion from: - sagemath#35652 (comment) (Fixes sagemath#35814) The steps "again builds incrementally" actually use the GH Actions cache, https://docs.docker.com/build/ci/github-actions/cache/#cache- backend-api. When nothing is cached and the 3 workflows are launched in parallel, they may each build the same thing. But when there's congestion that leads to the workflows to be run serially, the 2nd and 3rd workflow will be able to use the cache from the 1st workflow. This elasticity may be helpful in reducing congestion. There is a rather small per-project limit of 10 GB for this cache, so we'll have to see how effectively caching works in practice. See https://github.com/sagemath/sage/actions/caches   ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36938 (merged here) - Depends on sagemath#36951 (merged here) - Depends on sagemath#37351 (merged here) - Depends on sagemath#37750 (merged here)  URL: sagemath#36498 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

@kwankyu

…arate jobs for pyright, build, modularized tests, long tests    Running the containers explicitly, instead of using the declarative `container:` feature of GH Actions gives us more control: - we can create more space on the host if necessary; we just scraped by an out of space condition in sagemath#36473 / sagemath#36469 (comment) - we can run some operations outside of the container but in the same job; this will make the separate "Get CI fixes" jobs unnecessary, addressing the cosmetic concerns from sagemath#36338 (comment), sagemath#36349 - it enables caching between the various workflows (as first discussed in sagemath#36446). We split out static type checking with pyright into its separate workflow: - **pyright.yml**: As static checking does not need a build of the Sage library, for PRs that do not make any changes to packages, there's nothing to build, and the workflow gives a fast turnaround just after 10 minutes. For applying the CI fixes from blocker tickets, this workflow uses the technique favored in sagemath#36349. The workflow **build.yml** first launches a job: - **test-new:** It builds incrementally (using a tox-generated `Dockerfile` and https://github.com/marketplace/actions/build-and-push- docker-images) and does the quick incremental test. This completes within 10 to 20 minutes when there's no change. After this is completed, more jobs are launched: - **test-mod:** It again builds incrementally and tests a modularized distribution. Later (with more from sagemath#35095), more jobs will be added to this matrix job for other distributions. - **test-long:** It again builds incrementally and runs the long test. The workflows **doc-build.yml** and **doc-build-pdf.yml** again build incrementally and then build the documentation. The diffing code for the HTML documentation now runs in the host, not the container, which simplifies things. (To show that diffing still works, we include a small change to the Sage library.) Splitting the workflow jobs implements @kwankyu's suggestion from: - sagemath#35652 (comment) (Fixes sagemath#35814) The steps "again builds incrementally" actually use the GH Actions cache, https://docs.docker.com/build/ci/github-actions/cache/#cache- backend-api. When nothing is cached and the 3 workflows are launched in parallel, they may each build the same thing. But when there's congestion that leads to the workflows to be run serially, the 2nd and 3rd workflow will be able to use the cache from the 1st workflow. This elasticity may be helpful in reducing congestion. There is a rather small per-project limit of 10 GB for this cache, so we'll have to see how effectively caching works in practice. See https://github.com/sagemath/sage/actions/caches   ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36938 (merged here) - Depends on sagemath#36951 (merged here) - Depends on sagemath#37351 (merged here) - Depends on sagemath#37750 (merged here)  URL: sagemath#36498 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

@kwankyu

…arate jobs for pyright, build, modularized tests, long tests    Running the containers explicitly, instead of using the declarative `container:` feature of GH Actions gives us more control: - we can create more space on the host if necessary; we just scraped by an out of space condition in sagemath#36473 / sagemath#36469 (comment) - we can run some operations outside of the container but in the same job; this will make the separate "Get CI fixes" jobs unnecessary, addressing the cosmetic concerns from sagemath#36338 (comment), sagemath#36349 - it enables caching between the various workflows (as first discussed in sagemath#36446). We split out static type checking with pyright into its separate workflow: - **pyright.yml**: As static checking does not need a build of the Sage library, for PRs that do not make any changes to packages, there's nothing to build, and the workflow gives a fast turnaround just after 10 minutes. For applying the CI fixes from blocker tickets, this workflow uses the technique favored in sagemath#36349. The workflow **build.yml** first launches a job: - **test-new:** It builds incrementally (using a tox-generated `Dockerfile` and https://github.com/marketplace/actions/build-and-push- docker-images) and does the quick incremental test. This completes within 10 to 20 minutes when there's no change. After this is completed, more jobs are launched: - **test-mod:** It again builds incrementally and tests a modularized distribution. Later (with more from sagemath#35095), more jobs will be added to this matrix job for other distributions. - **test-long:** It again builds incrementally and runs the long test. The workflows **doc-build.yml** and **doc-build-pdf.yml** again build incrementally and then build the documentation. The diffing code for the HTML documentation now runs in the host, not the container, which simplifies things. (To show that diffing still works, we include a small change to the Sage library.) Splitting the workflow jobs implements @kwankyu's suggestion from: - sagemath#35652 (comment) (Fixes sagemath#35814) The steps "again builds incrementally" actually use the GH Actions cache, https://docs.docker.com/build/ci/github-actions/cache/#cache- backend-api. When nothing is cached and the 3 workflows are launched in parallel, they may each build the same thing. But when there's congestion that leads to the workflows to be run serially, the 2nd and 3rd workflow will be able to use the cache from the 1st workflow. This elasticity may be helpful in reducing congestion. There is a rather small per-project limit of 10 GB for this cache, so we'll have to see how effectively caching works in practice. See https://github.com/sagemath/sage/actions/caches   ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36938 (merged here) - Depends on sagemath#36951 (merged here) - Depends on sagemath#37351 (merged here) - Depends on sagemath#37750 (merged here) - Depends on sagemath#37277 (merged here)  URL: sagemath#36498 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

mkoeppe added 21 commits October 8, 2023 13:07

bootstrap-conda: Generate environment files with hard-coded Python mi…

6127016

…nor versions (environment-3.9.yml etc.) This is convenient for conda-lock. Partial cherry-pick from "Use conda-lock for reproducible conda env"

src/doc/en/installation/conda.rst: Update for versioned environment-.…

03a2acf

….. files

bootstrap, Makefile (bootstrap-clean): Update for versioned environme…

5c0bac8

…nt-... files

tox.ini (local-conda-environment): Update for versioned environment-.…

c2be172

….. files

.github/workflows/ci-conda.yml: Update for versioned environment*.yml

b3ba591

.github/workflows/ci-conda.yml: Fix environment name

018de2b

tox.ini (conda-environment-{src,dev}), .github/workflows/ci-conda.yml…

fc9bab5

…: Use configure --enable-system-site-packages

bootstrap-conda: Refactor using sage-get-system-packages

b02ef8e

bootstrap-conda: Only one 'python' line + speed up

235ce91

build/sage_bootstrap/download/mirror_list.py: Remove sagepad.org

3332a89

sage_bootstrap.{tarball, download.mirror_list}: Lower layout of the m…

dd939ff

…irror to mirror_list

sage_bootstrap.download.mirror_list: Delay downloading/reading/rankin…

2dfe081

…g to first use

sage_bootstrap.download.mirror_list: Use master list of download sour…

a37f4fd

…ces in SAGE_ROOT/.upstream.d

src/bin/sage-update-version: Update .upstream.d/20-github.com-sagemat…

2323480

…h-sage-releases

sage_bootstrap.download.mirror_list: Skip empty lines earlier

7e65260

src/bin/sage-update-version: Also commit

371be50

build/bin/write-dockerfile.sh: ADD .upstream.d

423e48a

bootstrap-conda (src/environment-dev*.yml): Suppress pip section, for…

73ab316

… now

.ci/retrofit-worktree.sh: New, factored out from .github/workflows/bu…

f429993

…ild*.yml

src/tox.ini (pyright): Remove old version pin, do not rely on global …

47f6a98

…node installation

.github/workflows/build.yml: Simplify use of pyright by going through…

47e2783

… 'sage -tox'

mkoeppe self-assigned this Oct 11, 2023

mkoeppe added the c: scripts label Oct 11, 2023

mkoeppe force-pushed the ci_streamline branch from 78b6109 to d71920c Compare October 11, 2023 03:59

Fixup

94c8b88

mkoeppe force-pushed the ci_streamline branch from d71920c to d060052 Compare October 11, 2023 04:14

src/tox.ini (pyright): Do not restrict to subdirectories by default

8bf3bcb

mkoeppe added 10 commits October 12, 2023 09:13

pkgs/sage-conf_conda/setup.py: Always run configure

ddbbdc5

pkgs/sage-conf_conda/setup.py: Remove 'reusing configured SAGE_ROOT' …

d0cae80

…message

Merge branch 'tox_pyright_update' into ci_streamline

c555723

Merge branch 'conda_legacy_editable' into ci_streamline

19f029b

Merge branch 'bootstrap_conda_refactor' into ci_streamline

68e9997

Merge branch 'gh_actions_release' into ci_streamline

66a8be4

Merge branch 'ci_refactor_build' into ci_streamline

2e11b0f

build/bin/write-dockerfile.sh, tox.ini (docker-incremental): Make inc…

0bfb487

…remental build more incremental via git, via git, .ci/retrofit-worktree.sh

.github/workflows/build.yml: Merge doc-build.yml, doc-build-pdf.yml

8e25cbd

.github/workflows/build.yml: Merge ci-linux-incremental.yml

459da44

mkoeppe force-pushed the ci_streamline branch from f72073c to 459da44 Compare October 12, 2023 17:17

mkoeppe mentioned this pull request Oct 13, 2023

CI Build: Build a docker image incrementally, then test, docbuild, ... in separate jobs #36459

Closed

5 tasks

mkoeppe added s: needs work and removed s: needs review labels Oct 17, 2023

mkoeppe mentioned this pull request Oct 17, 2023

CI docbuild: Do not rebuild sagelib from scratch #36473

Merged

5 tasks

mkoeppe closed this Oct 21, 2023

github-actions bot removed the s: needs work label Oct 21, 2023

kwankyu added r: wontfix and removed r: wontfix labels Oct 21, 2023

This was referenced Oct 22, 2023

CI docbuild: Run containers explicitly with docker run, docker exec #36508

Closed

CI build, doc-build: Run containers explicitly, separate jobs for pyright, build, modularized tests, long tests #36498

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Merge workflows into build.yml #36446

CI: Merge workflows into build.yml #36446

mkoeppe commented Oct 11, 2023 •

edited

mkoeppe commented Oct 11, 2023

mkoeppe commented Oct 11, 2023

kwankyu commented Oct 11, 2023

tobiasdiez commented Oct 18, 2023

mkoeppe commented Oct 19, 2023

mkoeppe commented Oct 21, 2023 •

edited

CI: Merge workflows into build.yml #36446

CI: Merge workflows into build.yml #36446

Conversation

mkoeppe commented Oct 11, 2023 • edited

📝 Checklist

⌛ Dependencies

mkoeppe commented Oct 11, 2023

mkoeppe commented Oct 11, 2023

kwankyu commented Oct 11, 2023

tobiasdiez commented Oct 18, 2023

mkoeppe commented Oct 19, 2023

mkoeppe commented Oct 21, 2023 • edited

mkoeppe commented Oct 11, 2023 •

edited

mkoeppe commented Oct 21, 2023 •

edited