Change Gammapy CI to Azure Pipelines?

[cdeil]

I think we should consider changing the Gammapy CI to Azure pipelines.

https://blog.github.com/2018-09-10-azure-pipelines-now-available-in-github-marketplace/

Numpy and other projects are trying it out, to quote from here:

The "free for open source" tier is 10 concurrent jobs that can run up to 60
minutes each (although the website says 30 minutes), and no limit on
minutes per month. Like the other CI services, it uses YAML files to
define the build and test process. I found the documentation of the YAML
pipeline spec to be large, but still missing a number of small details and
explanations about how features worked together. That said, the build
performance is significantly better than Travis CI and AppVeyor for all 3
platforms, and the convenience of all major platforms going through one
service is very nice. Despite not liking the UI quite as much as Travis
CI, I think Numba will likely switch completely over to Azure assuming the
free offering continues to be this good.

Currently we are using Appveyor for Windows, and travis-ci for Linux and Mac. Appveyor isn't set up properly (#818, #431). travis-ci is very slow (maybe because builds aren't configured well and run re-run way to much install stuff on each build). The dev docs build still isn't done in CI, but only manually locally (#1241). So Gammapy CI needs to be improved, and this issue is to discuss which direction to take (improve existing CI setup or try Azure pipelines).

Note that trying Azure pipelines comes at no risk, we can just add it for a while in parallel to what we have to try it out. I'm also not worried about long-term risk, Microsoft (but also travis-ci and Appveyor) seem like awesome and trustworthy free CI services, and should things change in a few years, well, then we can change again in a few years to something else. But Microsoft has clearly made a commitment to Github and Azure and open source and there's no reason to suspect that this will change any time soon.

We currently use https://github.com/astropy/ci-helpers , which as I understand is a set of helper scripts to install and run everything, almost like a little CI framework to configure things on top of travis-ci and Appveyor, using conda for install. @bsipocz or @astrofrog - is it correct that those don't work with Azure pipelines, so if we use those, we'd be on our own in terms of CI setup?

Azure pipelines do seem to support conda environments:
https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/package/conda-environment

From the Gammapy team, maybe @Bultako @adonath or @registerrier or anyone interested in Gammapy CI and likely to be involved in maintaining it can comment here.

Personally I'm +1 to try Azure Pipelines. For now, I plan to try it out on a tiny personal project I started recently (https://github.com/cdeil/multinorm) to get some experience with it.

[astrofrog]

If Azure pipelines already support conda environments out of the box, then you may not need to use ci-helpers, since installing dependencies afterwards should be reasonably easy.

[bsipocz]

@cdeil - ci-helpers doesn't support azure pipelines yet, but we're experimenting with it. It looks super promising so far.

[cdeil]

I'm starting this in the next days: #1913 .

[cdeil]

We have a working Linux build on Azure pipelines for Gammapy:

https://dev.azure.com/gammapy/gammapy/_build/latest?definitionId=1

So next steps here are to configure these builds:

• Python 3 on Windows, then remove Appveyor (see AppVeyor build is in personal account #818 )
• Dev docs build and push to Github pages (see Move docs from ReadTheDocs to Github pages #1241)

travis-ci is running well at the moment, migrating that can be done later.

[cdeil]

I'm closing this issue. We now have Azure Pipelines and travis-ci as CI systems. On both currently all builds are passing. The related #1241 is still open.

Moving all builds to on CI system isn't so easy. travis-ci doesn't support Windows build agents. We probably could move everything to Azure pipelines, but it's a lot of work to configure all the builds, and e.g. currently they don't support cron-based jobs yet. I'm OK for now with having these two CI systems.

[astrofrog]

@cdeil - just regarding your comment:

travis-ci doesn't support Windows build agents

Travis does now support Windows and ci-helpers supports Windows on Travis too.

[cdeil]

Ah, good to know. So moving all CI to travis-ci is an option as well.

[bsipocz]

Yes. Though travis is not very great with artifacts, which is my understanding azure does to a point.

[cdeil]

One more thing I forgot to mention: Azure pipeline is also designed to have release pipelines. The Gammapy release procedure is quite complex, several files have to be generated, webpages updated, git tag, PyPI upload, it's maybe 15 or 20 commands one has to type. This could be scripted as a semi-automated Azure pipeline, where the release manager only reviews logs of certain steps and approves the execution of the next steps in the pipeline (eventual release). Not sure if that really works in practice (i.e. one can customise steps if needed, or abort the release), but it we wanted this, it should be tried on Azure pipelines.

[bsipocz]

Fair enough, I also thinks that azure is very promising, though difficult to see it would solve any of the actual pain in astropy's releasing process, on the contrary would just add an extra layer of complexity.