CI: Merge open blocker PRs in all CI workflows + other improvements #36338

mkoeppe · 2023-09-25T23:34:31Z

The new script .ci/merge-fixes.sh uses the GitHub CLI to retrieve open PRs with the "blocker" label from sagemath/sage and tries to merge them into the tested branch, ignoring any PRs that give merge failures.

We call this script in all CI workflows. Thus,

hotfixes for the linter (such as fix the linter once more #36327) or
fixes for build problems (such as Yet more spkg_configure for standard python packages #36276)

will take effect in the CI workflows of all PRs.

This can be seen in the runs of the workflows of the present PR (section "Merge CI fixes from sagemath/sage").

Additionally,

we fix debian-trixie, which failed because a system package is no longer available
we fix macos-conda-maximal by removing some fictitious distros/conda.txt files
we refactor ci-macos using a new reusable workflow; example runs:
- https://github.com/mkoeppe/sage/actions/runs/6308124398
- https://github.com/sagemath/cysignals/actions/runs/6316714387?pr=185
we update the ci-macos platforms, including a configuration for the new and problematic Xcode 15.0

📝 Checklist

The title is concise, informative, and self-explanatory.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

…w macos.yml, update macOS, Xcode versions

… is not needed in the container

dimpase · 2023-09-27T10:35:14Z

are we not reinventing the wheel here, by any chance?
https://github.blog/changelog/2023-07-12-pull-request-merge-queue-is-now-generally-available/

mkoeppe · 2023-09-27T16:56:44Z

are we not reinventing the wheel here, by any chance?
https://github.blog/changelog/2023-07-12-pull-request-merge-queue-is-now-generally-available/

Yes, @dimpase, I have considered many options, including Merge Queues. But moving to using Merge Queues is not actionable for the project; the release manager would have to adopt a changed workflow. As I wrote in #36338 (comment):

Yes, this PR is a practical solution. It is designed to avoid two approaches, each of which would be more elegant:

It does not create hotfix branches explicitly. Hence, there is no hotfix branch to maintain; hence there is not need for a maintainer of the hotfix branch.

It does not require communication with the release manager.

This careful design makes this PR an effective solution.

Inappropriate grandstanding

dimpase · 2023-09-27T18:31:52Z

I don't see how a merge queue with blockers (or positively reviewed blockers) is different in functionality from this PR.

mkoeppe · 2023-09-27T18:34:33Z

I don't see how a merge queue with blockers (or positively reviewed blockers) is different in functionality from this PR.

Well, Merge Queues are attached to a branch protection rule for a branch. A PR can only be merged in one branch, namely its target branch. So doing anything like this would interfere with the release manager's work.

mkoeppe · 2023-09-27T18:38:39Z

@dimpase You may want to try out merge queues with https://github.com/dimpase/primecountpy/pulls ...

dimpase · 2023-09-27T21:44:23Z

@dimpase You may want to try out merge queues with https://github.com/dimpase/primecountpy/pulls ...

I don't think it's a repo which needs them.

mkoeppe · 2023-09-27T21:47:45Z

But it's harmless to play with them there.

kwankyu · 2023-09-27T22:54:18Z

Thanks for the benevolent action, @vbraun!

### 📚 Description Follow-up on sagemath#35169. The container `sage-docker-fedora-31-maximal-with-targets` used for the PDF docbuild turned out to be not reliable. Here we replace it by `ubuntu-focal-standard-with-targets` and install texlive in it. We also copy over the incremental build from doc-build.yml and the method to get CI fixes from blocker tickets from sagemath#36338. This workflow is currently disabled in sagemath/sage. Example run: https://github.com/mkoeppe/sage/actions/runs/6318741659/job/17158468016     ### 📝 Checklist  - [x] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36338 - Depends on sagemath#36348  URL: sagemath#35373 Reported by: Matthias Köppe Reviewer(s): Dima Pasechnik

…conda/centos/archlinux system packages   Fixes breakage from sagemath#36338 seen in https://github.com/sagemath/sage/acti ons/runs/6332061118/job/17206153694: The 2nd stage (`...-with-targets`) does not find the container from the 1st stage (`...-with-targets-pre`) because they are keyed to the commit sha. Fixed here by keying it to the sha before merging the CI fixes. Also fixing the following failures: - `conda-forge-minimal` (as seen in https://github.com/sagemath/sage/act ions/runs/6207962004/job/16858998561); failed because the python3 depcheck packages were missing. Fixed now - check with `tox -e docker- conda-forge-minimal -- config.status` - `centos-stream-9-python3.9-minimal` (https://github.com/sagemath/sage/ actions/runs/6207962004/job/16858997494), `almalinux-9-python3.11-minimal` (https://github.com/sagemath/sage/actio ns/runs/6207962004/job/16858997663) and `-standard`; fixes sagemath#34786 - `archlinux-standard` - libgiac was missing in at least one run Also adding a few more uses of GH Action output groups for convenience. Tests running at https://github.com/mkoeppe/sage/actions/workflows/ci- linux.yml    ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: sagemath#36358 Reported by: Matthias Köppe Reviewer(s): Dima Pasechnik

### 📚 Description Follow-up on #35169. The container `sage-docker-fedora-31-maximal-with-targets` used for the PDF docbuild turned out to be not reliable. Here we replace it by `ubuntu-focal-standard-with-targets` and install texlive in it. We also copy over the incremental build from doc-build.yml and the method to get CI fixes from blocker tickets from #36338. This workflow is currently disabled in sagemath/sage. Example run: https://github.com/mkoeppe/sage/actions/runs/6318741659/job/17158468016     ### 📝 Checklist  - [x] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on #36338 - Depends on #36348  URL: #35373 Reported by: Matthias Köppe Reviewer(s): Dima Pasechnik

…ntos/archlinux system packages   Fixes breakage from #36338 seen in https://github.com/sagemath/sage/acti ons/runs/6332061118/job/17206153694: The 2nd stage (`...-with-targets`) does not find the container from the 1st stage (`...-with-targets-pre`) because they are keyed to the commit sha. Fixed here by keying it to the sha before merging the CI fixes. Also fixing the following failures: - `conda-forge-minimal` (as seen in https://github.com/sagemath/sage/act ions/runs/6207962004/job/16858998561); failed because the python3 depcheck packages were missing. Fixed now - check with `tox -e docker- conda-forge-minimal -- config.status` - `centos-stream-9-python3.9-minimal` (https://github.com/sagemath/sage/ actions/runs/6207962004/job/16858997494), `almalinux-9-python3.11-minimal` (https://github.com/sagemath/sage/actio ns/runs/6207962004/job/16858997663) and `-standard`; fixes #34786 - `archlinux-standard` - libgiac was missing in at least one run Also adding a few more uses of GH Action output groups for convenience. Tests running at https://github.com/mkoeppe/sage/actions/workflows/ci- linux.yml    ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: #36358 Reported by: Matthias Köppe Reviewer(s): Dima Pasechnik

@kwankyu

…arate jobs for pyright, build, modularized tests, long tests    Running the containers explicitly, instead of using the declarative `container:` feature of GH Actions gives us more control: - we can create more space on the host if necessary; we just scraped by an out of space condition in sagemath#36473 / sagemath#36469 (comment) - we can run some operations outside of the container but in the same job; this will make the separate "Get CI fixes" jobs unnecessary, addressing the cosmetic concerns from sagemath#36338 (comment), sagemath#36349 - it enables caching between the various workflows (as first discussed in sagemath#36446). We split out static type checking with pyright into its separate workflow: - **pyright.yml**: As static checking does not need a build of the Sage library, for PRs that do not make any changes to packages, there's nothing to build, and the workflow gives a fast turnaround just after 10 minutes. For applying the CI fixes from blocker tickets, this workflow uses the technique favored in sagemath#36349. The workflow **build.yml** first launches a job: - **test-new:** It builds incrementally (using a tox-generated `Dockerfile` and https://github.com/marketplace/actions/build-and-push- docker-images) and does the quick incremental test. This completes within 10 to 20 minutes when there's no change. After this is completed, more jobs are launched: - **test-mod:** It again builds incrementally and tests a modularized distribution. Later (with more from sagemath#35095), more jobs will be added to this matrix job for other distributions. - **test-long:** It again builds incrementally and runs the long test. The workflows **doc-build.yml** and **doc-build-pdf.yml** again build incrementally and then build the documentation. The diffing code for the HTML documentation now runs in the host, not the container, which simplifies things. (To show that diffing still works, we include a small change to the Sage library.) Splitting the workflow jobs implements @kwankyu's suggestion from: - sagemath#35652 (comment) (Fixes sagemath#35814) The steps "again builds incrementally" actually use the GH Actions cache, https://docs.docker.com/build/ci/github-actions/cache/#cache- backend-api. When nothing is cached and the 3 workflows are launched in parallel, they may each build the same thing. But when there's congestion that leads to the workflows to be run serially, the 2nd and 3rd workflow will be able to use the cache from the 1st workflow. This elasticity may be helpful in reducing congestion. There is a rather small per-project limit of 10 GB for this cache, so we'll have to see how effectively caching works in practice. See https://github.com/sagemath/sage/actions/caches   ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36938 (merged here) - Depends on sagemath#36951 (merged here) - Depends on sagemath#37351 (merged here) - Depends on sagemath#37750 (merged here)  URL: sagemath#36498 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

@kwankyu

…arate jobs for pyright, build, modularized tests, long tests    Running the containers explicitly, instead of using the declarative `container:` feature of GH Actions gives us more control: - we can create more space on the host if necessary; we just scraped by an out of space condition in sagemath#36473 / sagemath#36469 (comment) - we can run some operations outside of the container but in the same job; this will make the separate "Get CI fixes" jobs unnecessary, addressing the cosmetic concerns from sagemath#36338 (comment), sagemath#36349 - it enables caching between the various workflows (as first discussed in sagemath#36446). We split out static type checking with pyright into its separate workflow: - **pyright.yml**: As static checking does not need a build of the Sage library, for PRs that do not make any changes to packages, there's nothing to build, and the workflow gives a fast turnaround just after 10 minutes. For applying the CI fixes from blocker tickets, this workflow uses the technique favored in sagemath#36349. The workflow **build.yml** first launches a job: - **test-new:** It builds incrementally (using a tox-generated `Dockerfile` and https://github.com/marketplace/actions/build-and-push- docker-images) and does the quick incremental test. This completes within 10 to 20 minutes when there's no change. After this is completed, more jobs are launched: - **test-mod:** It again builds incrementally and tests a modularized distribution. Later (with more from sagemath#35095), more jobs will be added to this matrix job for other distributions. - **test-long:** It again builds incrementally and runs the long test. The workflows **doc-build.yml** and **doc-build-pdf.yml** again build incrementally and then build the documentation. The diffing code for the HTML documentation now runs in the host, not the container, which simplifies things. (To show that diffing still works, we include a small change to the Sage library.) Splitting the workflow jobs implements @kwankyu's suggestion from: - sagemath#35652 (comment) (Fixes sagemath#35814) The steps "again builds incrementally" actually use the GH Actions cache, https://docs.docker.com/build/ci/github-actions/cache/#cache- backend-api. When nothing is cached and the 3 workflows are launched in parallel, they may each build the same thing. But when there's congestion that leads to the workflows to be run serially, the 2nd and 3rd workflow will be able to use the cache from the 1st workflow. This elasticity may be helpful in reducing congestion. There is a rather small per-project limit of 10 GB for this cache, so we'll have to see how effectively caching works in practice. See https://github.com/sagemath/sage/actions/caches   ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36938 (merged here) - Depends on sagemath#36951 (merged here) - Depends on sagemath#37351 (merged here) - Depends on sagemath#37750 (merged here)  URL: sagemath#36498 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

@kwankyu

…arate jobs for pyright, build, modularized tests, long tests    Running the containers explicitly, instead of using the declarative `container:` feature of GH Actions gives us more control: - we can create more space on the host if necessary; we just scraped by an out of space condition in sagemath#36473 / sagemath#36469 (comment) - we can run some operations outside of the container but in the same job; this will make the separate "Get CI fixes" jobs unnecessary, addressing the cosmetic concerns from sagemath#36338 (comment), sagemath#36349 - it enables caching between the various workflows (as first discussed in sagemath#36446). We split out static type checking with pyright into its separate workflow: - **pyright.yml**: As static checking does not need a build of the Sage library, for PRs that do not make any changes to packages, there's nothing to build, and the workflow gives a fast turnaround just after 10 minutes. For applying the CI fixes from blocker tickets, this workflow uses the technique favored in sagemath#36349. The workflow **build.yml** first launches a job: - **test-new:** It builds incrementally (using a tox-generated `Dockerfile` and https://github.com/marketplace/actions/build-and-push- docker-images) and does the quick incremental test. This completes within 10 to 20 minutes when there's no change. After this is completed, more jobs are launched: - **test-mod:** It again builds incrementally and tests a modularized distribution. Later (with more from sagemath#35095), more jobs will be added to this matrix job for other distributions. - **test-long:** It again builds incrementally and runs the long test. The workflows **doc-build.yml** and **doc-build-pdf.yml** again build incrementally and then build the documentation. The diffing code for the HTML documentation now runs in the host, not the container, which simplifies things. (To show that diffing still works, we include a small change to the Sage library.) Splitting the workflow jobs implements @kwankyu's suggestion from: - sagemath#35652 (comment) (Fixes sagemath#35814) The steps "again builds incrementally" actually use the GH Actions cache, https://docs.docker.com/build/ci/github-actions/cache/#cache- backend-api. When nothing is cached and the 3 workflows are launched in parallel, they may each build the same thing. But when there's congestion that leads to the workflows to be run serially, the 2nd and 3rd workflow will be able to use the cache from the 1st workflow. This elasticity may be helpful in reducing congestion. There is a rather small per-project limit of 10 GB for this cache, so we'll have to see how effectively caching works in practice. See https://github.com/sagemath/sage/actions/caches   ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36938 (merged here) - Depends on sagemath#36951 (merged here) - Depends on sagemath#37351 (merged here) - Depends on sagemath#37750 (merged here) - Depends on sagemath#37277 (merged here)  URL: sagemath#36498 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

mkoeppe added 5 commits September 25, 2023 16:22

build/pkgs/sagemath_{coxeter3,mcqd,meataxe,tdlib}: Remove bad conda.txt

52508f6

tox.ini (debian-trixie): Use IGNORE_MISSING_SYSTEM_PACKAGES=yes

0a27b65

.github/workflows/ci-macos.yml: Refactor through new reusable workflo…

4ba2b6a

…w macos.yml, update macOS, Xcode versions

.ci/merge-fixes.sh: New

9cff8ff

Use .ci/merge-fixes.sh in all workflows

a9d84a8

mkoeppe added the c: scripts label Sep 25, 2023

mkoeppe self-assigned this Sep 25, 2023

mkoeppe force-pushed the tox_2023_09 branch 4 times, most recently from 0c0b7bd to 32b9f5a Compare September 26, 2023 00:24

.ci/merge-fixes.sh: Unshallow for merge

e4025c1

mkoeppe force-pushed the tox_2023_09 branch from 32b9f5a to e4025c1 Compare September 26, 2023 00:30

mkoeppe added the s: needs review label Sep 26, 2023

mkoeppe force-pushed the tox_2023_09 branch 4 times, most recently from 1f12c3c to d7c07c0 Compare September 26, 2023 01:53

mkoeppe added s: needs work and removed s: needs review labels Sep 26, 2023

mkoeppe force-pushed the tox_2023_09 branch 5 times, most recently from e4105ab to 37c96cf Compare September 26, 2023 03:47

build.yml, doc-build.yml: First create CI fixes as an artifact, so gh…

3949cc7

… is not needed in the container

mkoeppe force-pushed the tox_2023_09 branch from 37c96cf to 3949cc7 Compare September 26, 2023 04:02

mkoeppe added s: needs review and removed s: needs work labels Sep 26, 2023

mkoeppe changed the title ~~CI: Merge open blocker PRs in all CI workflows~~ CI: Merge open blocker PRs in all CI workflows + other improvements Sep 26, 2023

kwankyu added the s: positive review label Sep 27, 2023

tobiasdiez removed the s: positive review label Sep 27, 2023

mkoeppe added the s: needs review label Sep 27, 2023

mkoeppe mentioned this pull request Sep 27, 2023

Add needed pip packages to src/environment.yml, minimize conda environments #35593

Open

5 tasks

vbraun merged commit bb04839 into sagemath:develop Sep 27, 2023
71 checks passed

github-actions bot removed the s: needs review label Sep 27, 2023

mkoeppe deleted the tox_2023_09 branch September 27, 2023 22:24

mkoeppe added this to the sage-10.2 milestone Sep 27, 2023

mkoeppe mentioned this pull request Sep 28, 2023

.github/workflows/build.yml, doc-build.yml: Fix get_ci_fixes #36348

Merged

5 tasks

tobiasdiez mentioned this pull request Sep 28, 2023

ci: improve things around auto-apply of fixes #36349

Open

5 tasks

mkoeppe mentioned this pull request Sep 28, 2023

CI: Fix the multi-stage build in ci-linux.yml, fix conda/centos/archlinux system packages #36358

Merged

5 tasks

This was referenced Oct 22, 2023

CI build, doc-build: Run containers explicitly, separate jobs for pyright, build, modularized tests, long tests #36498

Merged

CI docbuild: Run containers explicitly with docker run, docker exec #36508

Closed

mkoeppe added the disputed PR is waiting for community vote, see https://groups.google.com/g/sage-devel/c/IgBYUJl33SQ label Dec 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Merge open blocker PRs in all CI workflows + other improvements #36338

CI: Merge open blocker PRs in all CI workflows + other improvements #36338

mkoeppe commented Sep 25, 2023 •

edited

dimpase commented Sep 27, 2023

mkoeppe commented Sep 27, 2023

dimpase commented Sep 27, 2023

mkoeppe commented Sep 27, 2023

mkoeppe commented Sep 27, 2023

dimpase commented Sep 27, 2023

mkoeppe commented Sep 27, 2023

kwankyu commented Sep 27, 2023

CI: Merge open blocker PRs in all CI workflows + other improvements #36338

CI: Merge open blocker PRs in all CI workflows + other improvements #36338

Conversation

mkoeppe commented Sep 25, 2023 • edited

📝 Checklist

⌛ Dependencies

dimpase commented Sep 27, 2023

mkoeppe commented Sep 27, 2023

dimpase commented Sep 27, 2023

mkoeppe commented Sep 27, 2023

mkoeppe commented Sep 27, 2023

dimpase commented Sep 27, 2023

mkoeppe commented Sep 27, 2023

kwankyu commented Sep 27, 2023

mkoeppe commented Sep 25, 2023 •

edited