CI: Add portability and integration tests with Sage

[mkoeppe]

As promised in #1166 (comment) @albinahlback @edgarcosta, here's an updated GH Actions workflow for portability and integration testing using the reusable SageMath workflows.

Currently configured to:

• only run on tags, not branches / PRs; can also run manually
• build FLINT from scratch and run its test suite on all Linux platforms supported by SageMath
• build FLINT from scratch and run its test suite on some macOS configurations
• build Sage incrementally from a prebuilt Docker image from ghcr.io for one Linux platform and run the Sage test suite.

Preview of the workflow run: https://github.com/mkoeppe/flint2/actions/workflows/ci-sage.yml

[albinahlback]

How long does each run take?

Also, if we would instantiate a new tag, the "damage" could already have been done months back with multiple commits in between. Tracking bugs and issues, and reverting commits with possible would be way harder in this case when the time of a new tag comes around. As such, this would preferably be run at least at every merge, but I suppose that this would take too long.

Now, this is hopefully mostly important at the moment as we do have some change in the interface from 3.0 to the upcoming 3.1. Don't know if this would impact Sage.

Other than that, this looks good!

[mkoeppe]

How long does each run take?

Each of the Linux runs is between 10 and 15 minutes, the macOS runs are 16 minutes.

The integration testing with Sage probably around 1.5 hours. Currently fails trying to rebuild Singular - https://github.com/mkoeppe/flint2/actions/runs/7718792281/job/21040792339

Also, if we would instantiate a new tag, the "damage" could already have been done months back with multiple commits in between. Tracking bugs and issues, and reverting commits with possible would be way harder in this case when the time of a new tag comes around. As such, this would preferably be run at least at every merge, but I suppose that this would take too long.

Sure, that's something for the project to figure out.

For the portability testing, it would would also be possible to only run a few selected platforms on each PR, but all platforms on tags.

Or it could be configured as a cron job that runs it nightly or weekly.

For the integration testing with Sage, we would be able to test some subset of the test suite instead of the whole test suite.

I'll happy to help with adjusting the workflow file to implement the desired policy.

[albinahlback]

Each of the Linux runs is between 10 and 15 minutes, the macOS runs are 16 minutes.

Sounds good!

The integration testing with Sage probably around 1.5 hours. Currently fails trying to rebuild Singular - https://github.com/mkoeppe/flint2/actions/runs/7718792281/job/21040792339

We will look into this before releasing 3.1. Thanks!

Or it could be configured as a cron job that runs it nightly or weekly.

Seems like the best idea. I would prefer this.

For the integration testing with Sage, we would be able to test some subset of the test suite instead of the whole test suite.

This would be really good. Moreover, I don't think there is no need to test different OSs from our side as we have tests for different OSs already.

[mkoeppe]

For the failures on ubuntu-focal-standard, linuxmint-20.*-standard, fedora-{33,34,35}-standard, centos-stream-9-standard, almalinux-9-standard, I have opened:

• build/pkgs/flint 3.1.3 sagemath/sage#37203

[mkoeppe]

A failure:

https://github.com/mkoeppe/flint2/actions/runs/7718792281/job/21040792473#step:14:6853

  fmpz_mat_is_square...
  fmpz_mat_ismake[3]: *** [build/fmpz_mod_mat/test/main_TEST_RUN] Segmentation fault (core dumped)

[albinahlback]

A failure:

https://github.com/mkoeppe/flint2/actions/runs/7718792281/job/21040792473#step:14:6853

  fmpz_mat_is_square...
  fmpz_mat_ismake[3]: *** [build/fmpz_mod_mat/test/main_TEST_RUN] Segmentation fault (core dumped)

I don't see any specific reason to why this would fail. Looks like everything else works just fine. Moreover, I feel like this test is one of the last tests that would fail. I don't see anything compiler specific that could make this fail.

But if I understand this correctly, it is Ubuntu 22 with Ubuntu 14 toolchain? Why would someone use this combination?

[fredrik-johansson]

76b93b4 just fixed an uninitialized variable in the fmpz_mod_mat tests which potentially might have been responsible for that segfault.

[mkoeppe]

if I understand this correctly, it is Ubuntu 22 with Ubuntu 14 toolchain? Why would someone use this combination?

No, this is Ubuntu 14 with a backport toolchain https://github.com/sagemath/sage/blob/develop/tox.ini#L213 from https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test
It's running in a container on a GitHub runner; the OS version of that runner is not relevant.

[albinahlback]

No, this is Ubuntu 14 with a backport toolchain https://github.com/sagemath/sage/blob/develop/tox.ini#L213 from https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test It's running in a container on a GitHub runner; the OS version of that runner is not relevant.

Thanks for the clarification!

[edgarcosta]

Looks like some of the tests are still failing: https://github.com/mkoeppe/flint2/actions/workflows/ci-sage.yml

[mkoeppe]

Rebased on main for another run.

[edgarcosta]

You need to trigger the run on your repo

[mkoeppe]

Thanks for the reminder, I got distra...

[mkoeppe]

...cted, running now at https://github.com/mkoeppe/flint2/actions/runs/8382682177

[edgarcosta]

it looks like all checks have failed