Re-evaluate CI pipelines

[kotfu]

We currently use Travis, Appveyor and Azure Pipelines. We had Appveyor to test on Windows, and we added Azure as an experiment. Now Azure supports mac, linux, and windows. In addition, GitHub now has a native Actions which can run CI on mac, linux, and windows. We should re-evaluate which one(s) of these we want to use, and get rid of the rest.

[tleonhardt]

Based on the problems we have had over the years (or lack thereof) I'd rank the existing CI services we use from best to worst as:

1. Travis CI - excellent for Linux, crappy for Mac
2. Azure Pipelines - good for Mac, but haven't tested Windows or Linux
3. AppVeyor - kinda sucks and would be the best first candidate for replacement

I have zero experience with GitHub Actions, but it sounds very intriguing. I think we should try it.

It would probably be beneficial to redo all of our CI integrations from the ground up anyways since most of them were created by Federico and he is no longer involved with the project.

[kotfu]

I didn't know anything about GitHub Actions either, so I tried it out at https://github.com/tomcatmanager/tomcatmanager/actions. The UI still feels a little rough, but the builds seem to run pretty quickly and have been reliable for the limited time I've used it.

One of the current limitations is that you can't delete the build history for anything, so if you make a test job, or want to rename it, you have some cruft that will remain until they update the UI. However, there are some really nice things about it. Codecov has a nice action that uploads coverage reports, and you can keep the Codecov secret in Github Secrets.

It's all configured by a yaml file which looks very similar to travis or azure, with one really nice improvement. It can do multi-item matrices. So you can have a matrix that builds all python versions on all 3 platforms very easily. See https://github.com/tomcatmanager/tomcatmanager/blob/develop/.github/workflows/test.yml for an example yaml file.

[tleonhardt]

Ok, you are making GitHub Actions sound pretty attractive.

Perhaps we could start small by moving our Windows testing from AppVeyor to GitHub Actions and see how we like it after a month or so? Maybe we could even leave AppVeyor in place while we evaluate GitHub Actions?

Then if we are happy we can evaluate moving macOS and Linux testing there as well one step at a time?

[kotfu]

Yup. Super easy to just add Github Actions for Windows testing and leave AppVeyor running too. No problem to incrementally migrate either.

We can use this opportunity to re-evaluate our testing strategy as well. For example, code coverage only needs to run on one version of python on one OS. Same with flake8 and doc8.

We could save a few carbon molecules by only testing on linux on each commit, and test on all platforms for each pull request.

Lots of options.

[tleonhardt]

We are having Windows failures on AppVeyor at the moment from what appears to be a pip issue but it isn't clear and isn't due to any of our code.

So moving Windows testing to GitHub Actions might be a very welcome thing.

[tleonhardt]

Lately TravisCI has been super slow. So it might be a better candidate for first moving to GitHub Actions

[tleonhardt]

The travis-ci.org site now has a warning at the top that says:

Please be aware travis-ci.org will be shutting down in several weeks, with all accounts migrating to travis-ci.com. Please stay tuned here for more information.

From what I read they are giving everyone 1000 minutes of CPU time (or something like that) for free and then open source projects will need to get "validated" in order to receive free open-source credits.

So this whole re-organization and shutting down of the old infrastructure may in-part explain why Travis has been so shitty lately.