[RFC] [ci] remove Azure DevOps CI jobs?

[jameslamb]

Description

@shiyu1994 informed me this week (in the private maintainer chat) that he has left Microsoft.

As a result, there are now 0 active contributors to LightGBM employed by Microsoft.

That poses a serious risk to development on this project. Development in this project has been disrupted many times over the last few years with problems of the form "someone needs to take an administrative action that only Microsoft employees have permission to do".

Most recently, Azure DevOps linux CI jobs not working for 2 months: #6918

Even if/when Microsoft assigns new maintainers to the project, based on my experience over the last few years, I am not optimistic that they'll be very responsive. For example, my requests for financial/development support to build and test aarch64 macOS wheels received 0 response from Microsoft for multiple years: #5328. And other than @shiyu1994 (who was also actively developing LightGBM), I have seen very little other involvement from Microsoft in this project's day-to-day maintenance over the last few years.

To try to reduce the risk of disruption to development, I'm proposing that we remove all Azure DevOps CI jobs in the project.

Benefits of this work

• reduces the risk of lengthy periods where commits cannot be merged in this project

Acceptance criteria

• LightGBM CI does not use Azure DevOps

Approach

For all of the jobs in https://github.com/microsoft/LightGBM/blob/1684419f6d2221bc355f883f27c84581359d126b/.vsts-ci.yml, do some mix of the following:

• run them on GitHub Actions (with GitHub-hosted runners)
• run them on AppVeyor (with AppVeyor-hosted runners)
• stop running them completely

Notes

The Azure DevOps CI jobs have been most useful for allowing LightGBM to run more jobs concurrently than we could on only GitHub-hosted runners.

A few years ago, we were worried about being limited to 20 concurrent GitHub Actions jobs across the entire repo (#3672 (comment)). I think that limit should be much higher now.

Table from https://docs.github.com/en/actions/administering-github-actions/usage-limits-billing-and-administration

I don't know for sure, but I strongly suspect that the microsoft organization is on an Enterprise GitHub plan. If I'm right about that, our limit would be 500 total concurrent jobs and 50 concurrent macOS jobs... I think we could make LightGBM at it's current needs and amount of activity work within those limits, with most of the jobs from Azure DevOps transitioned to GitHub Actions.

[jameslamb]

Tagging other maintainers for your thoughts:

@StrikerRUS @guolinke @shiyu1994 @jmoralez @borchero

[guolinke]

I am +1

[jmoralez]

I'm +1 as well

[StrikerRUS]

The main problem of such migration will be artifacts creation, I guess. But OK, let's try to do this.

[jameslamb]

I know :/

But at least we'd be able to fix any issues ourselves and not be blocked the way we were in #6918

Ok thank you all, I will start on this this week. I'll break it up into a few PRs.

[borchero]

Little late to the party but I agree that, under the circumstances, it's a smart move to move off of Azure CI jobs. Thanks @jameslamb!

[letmaik]

I'm temporarily taking care of the CI runners for lightgbm in Microsoft (one Azure DevOps CPU pool (8-core), one GitHub Actions GPU runner (V100)).

I agree that the project should move fully to GitHub Actions and use the hosted runners, it will simplify things.

However, this won't work for the GPU runner which backs the CUDA GitHub Actions workflow (https://github.com/microsoft/LightGBM/blob/master/.github/workflows/cuda.yml). Unfortunately (for cost attribution reasons), using the separately paid GPU Larger Runner of GitHub (T4 GPUs) is not an option. I'm currently checking who can own the V100 runner inside Microsoft. If you see any issues with the GPU runner, feel free to tag me.

[jameslamb]

Nice to meet you @letmaik , thanks very much for the help! I will try to make some progress on moving the other (non-GPU) jobs off of Azure DevOps.

And please @ me any time for help on moving the CUDA jobs to the GPU Large Runner over in #6958 (glad that seems like it might be an option now!).