cygwin, conda, homebrew: Fixes for CI

[mkoeppe]

Hopefully the last change for the CI for 9.1

Depends on #29530

CC: @dimpase @slel @embray

Component: porting: Cygwin

Author: Matthias Koeppe

Branch/Commit: 630576b

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29622

[mkoeppe]

Branch: u/mkoeppe/cygwin__fixes_for_ci

[mkoeppe]

Dependencies: #29530

[mkoeppe]

Commit: f9700fd

[mkoeppe]

Last 10 new commits:

652787b	.github/workflows/tox*.yml: Make copying logs from containers more robust
f8c6e7a	tox.ini: Commit a failed build container so that logs are copied from it
5c8e685	tox-optional.yml: Fix likewise
4f4fe60	ci-cygwin.yml: Split the file so that unrelated stages do not wait
27c63f1	Improve docker login instructions
dbf461d	Merge tag '9.1.rc2' into t/29530/make_docker_images_generated_by_github_ci_workflows_available
163562d	Rename cygwin workflows
97a96c6	ci-cygwin*.yml: Fix typos
9187128	tox.yml: Add docker-conda-forge
f9700fd	tox.yml: Add homebrew-macos-python3_xcode-gcc_spkg

[mkoeppe]

comment:4

This fixes some syntax errors in the cygwin build. Now it times out in stage 3. This would need to be rebalanced or split into several parallel stages

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

8406e25	Rebalance stages
99342f1	Update cygwin-minimal, cygwin-standard-python2
8df6b20	Fixup, rebalance

[sagetrac-git]

Changed commit from f9700fd to 8df6b20

[mkoeppe]

comment:7

getting closer - https://github.com/mkoeppe/sage/runs/639064085

[mkoeppe]

comment:8

Fatal error: can't write 24 bytes to section .text of build/temp.cygwin-3.1.4-x86_64-3.7/build/cythonized/sage/rings/laurent_series_ring_element.o: 'No space left on device'

[sagetrac-git]

Changed commit from 8df6b20 to ea92c28

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

ea92c28

Factor out .github/workflows/extra-sage-local.sh from ci-cygwin*.yml, remove archives after extracting

[sagetrac-git]

Changed commit from ea92c28 to 0d46553

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

0d46553

Fixup

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

2eb8351	.github/workflows/*.yml: Use max-parallel so that the multi-stage workflows get a chance to run
9b1214f	fiuxp

[sagetrac-git]

Changed commit from 0d46553 to 9b1214f

[sagetrac-git]

Changed commit from 9b1214f to d3b5f8c

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

801a0d0	Rebalance
bf15a04	Use tar --remove-files
016df8a	Prepare sage-local artifacts in /tmp
c400c37	.github/workflows/extract-sage-local.sh: Show df
d3b5f8c	Fix up listing tar files

[mkoeppe]

comment:13

This now manages to build Sage on Cygwin (see cygwin-stage-iii in https://github.com/mkoeppe/sage/runs/640373882)

[sagetrac-git]

Changed commit from d3b5f8c to 630576b

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

630576b

Update cygwin-minimal, cygwin-standard-python2

[mkoeppe]

comment:15

Ready for review

[mkoeppe]

comment:16

cygwin-minimal, stage-ii-a and stage-ii-c still time out (https://github.com/mkoeppe/sage/runs/641053384); more rebalancing should be done in the next release cycle. I stop for this ticket and 9.1.

Uploading and downloading of our large sage-local artifacts is somewhat brittle, a game of chance: For example, in https://github.com/mkoeppe/sage/runs/641209977 (cygwin-standard), stage-iii :

Exponential backoff for retry #5. Waiting for 33036.92082171844 milliseconds before continuing the upload at offset 1379926016
Total file count: 1 ---- Processed file #0 (0.0%)
Finished backoff for retry #5, continuing with upload
Total file count: 1 ---- Processed file #0 (0.0%)
An error has been caught http-client index 0, retrying the upload
Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:201:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}
Retry limit has been reached for chunk at offset 1379926016 to https://pipelines.actions.githubusercontent.com/4D1poTSDSSApU42GXYeGgQX6WjHwGPK9ZzdaE2SeHdlUs0ZBhW/_apis/resources/Containers/3201477?itemPath=sage-local-commit-b2c012bfed0fb9414e0db531acc43a3abf1ae1ba-cygwin-standard%5Csage-local-iii.tar
##[warning]Aborting upload for C:\tools\cygwin\tmp\sage-local-iii.tar due to failure
Total size of all the files uploaded is 1379926016 bytes

In the same run, for https://github.com/mkoeppe/sage/runs/641394549 (cygwin-standard-python2), the upload of stage-iii succeeded,
and stage-iv-a ... stage-iv-d downloaded it successfully.

• stage-iv-a (make ptest, https://github.com/mkoeppe/sage/runs/641394549) terminates, with a number of doctest errors and timeouts:

  sage -t src/doc/en/constructions/plotting.rst  # Timed out
  sage -t src/doc/fr/tutorial/interfaces.rst  # 1 doctest failed
  sage -t src/doc/ja/tutorial/interfaces.rst  # 2 doctests failed
  sage -t src/doc/pt/tutorial/interfaces.rst  # Timed out
  sage -t src/doc/ru/tutorial/interfaces.rst  # 6 doctests failed
  sage -t src/sage/coding/binary_code.pyx  # Timed out
  sage -t src/sage/doctest/control.py  # 1 doctest failed
  sage -t src/sage/functions/log.py  # 1 doctest failed
  sage -t src/sage/functions/orthogonal_polys.py  # 1 doctest failed
  sage -t src/sage/functions/other.py  # 2 doctests failed
  sage -t src/sage/dynamics/arithmetic_dynamics/projective_ds.py  # Timed out
  sage -t src/sage/interfaces/gap.py  # 1 doctest failed
  sage -t src/sage/groups/class_function.py  # Timed out
  sage -t src/sage/interfaces/giac.py  # 41 doctests failed
  sage -t src/sage/libs/gap/element.pyx  # 1 doctest failed
  sage -t src/sage/libs/glpk/error.pyx  # 2 doctests failed
  sage -t src/sage/matrix/matrix1.pyx  # 1 doctest failed
  sage -t src/sage/numerical/backends/glpk_backend.pyx  # 2 doctests failed
  sage -t src/sage/parallel/map_reduce.py  # Timed out after testing finished
  sage -t src/sage/repl/interpreter.py  # 3 doctests failed
  sage -t src/sage/rings/number_field/number_field_element.pyx  # Timed out
  sage -t src/sage/sandpiles/sandpile.py  # Timed out
  sage -t src/sage/symbolic/constants.py  # 3 doctests failed
  sage -t src/sage/symbolic/expression_conversions.py  # 1 doctest failed
  sage -t src/sage/symbolic/expression.pyx  # 1 doctest failed
  sage -t src/sage/tests/benchmark.py  # 2 doctests failed
  sage -t src/sage/tests/books/computational-mathematics-with-sagemath/sol/graphique_doctest.py  # 1 doctest failed
  sage -t src/sage/tests/cmdline.py  # 3 doctests failed
  

  Looks like the giac interface is broken.

• stage-iv-b builds a few optional packages; I have noted build failures in some separate tickets.

• stage-iv-c builds cbc, sage_numerical_backends_coin successfully.

• stage-iv-d (https://github.com/mkoeppe/sage/runs/641394565) builds a long list of optional packages, times out; I have noted build failures in some separate tickets.

[mkoeppe]

comment:17

Needs review. Let's get this into 9.1 please

[dimpase]

comment:19

this looks good; one minor question is how to get logs for particular failed spkg builds.
Is the latter possible/documented, and I'm just not looking at the right place?

[dimpase]

Reviewer: Dima Pasechnik

[mkoeppe]

comment:20

Replying to @dimpase:

one minor question is how to get logs for particular failed spkg builds.
Is the latter possible/documented, and I'm just not looking at the right place?

Do you see the artifact named "logs-...." in
https://github.com/mkoeppe/sage/actions/runs/94565308?

[mkoeppe]

comment:21

It contains both the logs of all packages,
and the contents of $SAGE_LOCAL/var/tmp/sage/build/ (all failed packages)

[mkoeppe]

comment:22

Alternatively, if one does not want to download these huge log archives, one can open the section "Print out logs for immediate inspection" (for example in https://github.com/mkoeppe/sage/runs/641394520?check_suite_focus=true) to see logs of packages with errors. (This already works before artifacts are made available.)

[dimpase]

comment:23

Replying to @mkoeppe:

It contains both the logs of all packages,
and the contents of $SAGE_LOCAL/var/tmp/sage/build/ (all failed packages)

ok - I only looked at "view raw logs" - and these are, despite the name, less complete than artefacts, right?

[mkoeppe]

comment:24

"View raw logs" just gives you what is displayed in the right pane. This is a GitHub Actions feature.
The log artefacts have a lot more, from my scripts.

[vbraun]

Changed branch from u/mkoeppe/cygwin__fixes_for_ci to 630576b