VSCode-python pipelines use so many build agents they block other Microsoft projects

[miniksa]

From the microsoft/terminal repo, I noticed we were unable to dispatch builds for a while this afternoon. When I checked the job limits, it appears that this pipeline build/ci/vscode-python-nightly-uitest.yaml and build/ci/vscode-python-nightly-ci.yaml explodes into dozens of machines.

Our pool only has 200 machines available at any given moment.

In this run: https://dev.azure.com/ms/vscode-python/_build/results?buildId=61919&view=results

It looks like 30-50 of them are assigned to just one nightly build. And not even at night. In the middle of the day.

There's also this run happening at the same time:
https://dev.azure.com/ms/vscode-python/_build/results?buildId=61924&view=results

It also appears to be using 30-50 machines.

So it appears as if >50% of the pool is in use by your pipelines.

Could you please adjust your pipeline to serialize some of these tests such that the rest of us can use the resources as well? Or move the nightlies to happen over night so there are agents for CI during working hours?

Thanks!
--Michael

[miniksa]

Also, I just realized. 4pm is midnight GMT. If they're set for midnight GMT, can you set them to midnight Pacific time, please?

[DonJayamanne]

Hmm, when this was setup we were under the impression that each pipeline gets 200 machines. I.e. we didn't realize its something we had to share with other teams/products.

[ericsnowcurrently]

FYI, with that change we are now using the following number of build agents:

• per-PR: 13 -> 8 (for 8 jobs)
• per-merge: 39 -> 8 (for 24 jobs, so the pipeline takes roughly 3 times as long to run)
• nightly: 84 -> 12 (for 84 jobs, so the pipeline takes roughly 7 times as long to run)

@miniksa, at this point we shouldn't be impacting other users of the agent pool nearly as severely. If you see any problems please let us know.