[ci] run Azure linux tasks on self-hosted runner pool

[jameslamb]

To help with #3519, @guolinke has created a pool of self-owned runners for CI jobs on Azure DevOps. #3519 (comment)

This PR proposes moving the linux jobs we currently run on Azure DevOps to that pool. This should have no effect on those jobs, since they are containerized and run in containers based on the image from https://github.com/guolinke/lightgbm-ci-docker.

Running on this pool should decrease our overall CI time. See some of the comments from #3672 (comment). Azure allows us 10 concurrent builds across all operating systems for all jobs using microsoft-hosted runners. After this change, those 10 concurrent builds would only be used by Mac and Windows jobs.

Notes from Reviewers

I think this PR needs approval from both @StrikerRUS and @guolinke . This will lead to 3 runs on the sh-ubuntu pool per commit pushed, and I want to be sure @guolinke is ok with whatever the cost implications of that is.

[StrikerRUS]

I'm very unsure about what I'm saying 😃
But seems the cost is calculated only for usage of virtual machines, all parallel jobs are free

[StrikerRUS]

Can confirm that this PR utilized agents from sh-ubuntu pool:

[guolinke]

great! BTW, do we need the windows self-hosted pool?

[jameslamb]

great! BTW, do we need the windows self-hosted pool?

The Windows self-hosted pool would definitely be helpful too, but I haven't moved to it yet because differences between that VM image and the microsoft-hosted runner's image might affect artifacts. Like @StrikerRUS said in #3672 (comment),

Artifacts are something that are quite sensitive to the environments where they are built.

This wasn't a concern with linux because we use docker for isolation.

I think we should not use the Windows pool until we notice a problem with long build times because of a lot of jobs queued up. So if having it is costing you money I think you could probably remove it.

[StrikerRUS]

Also, given that we moved all Linux jobs to self-hosted pool, now we have only macOS and Windows jobs on Microsoft-hosted pool. And on Microsoft-hosted pool we have 10 free parallel jobs, but only 5 of them can be macOS jobs. So, I think it makes sense to keep Windows jobs on Microsoft-hosted pool to fully utilize it.

10 free Microsoft-hosted jobs across different OSes, but always only 5 for macOS;

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.