Concurrency limit for CI jobs

[potiuk]

Yesterday's problem with CI builds has been confirmed as hitting concurrency limit by GitHub support.

According to this page Limits we have 180 concurrent job limits at the Apache Enterprise-level account. Seems that we are competing with other projects (similarly as it was in Travis).

I started a thread on the builds@apache.org list https://lists.apache.org/thread.html/r1708881f52adbdae722afb8fea16b23325b739b254b60890e72375e1%40%3Cbuilds.apache.org%3E

and opened an accompanying ticket in JIRA: https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20978

GitHub Support also suggested (as in our internal discussions), Self-hosting runners.
https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners

[potiuk]

cc: @kaxil @ashb @dimberman @mik-laj @turbaszek (and others).

[potiuk]

@aijamalnk Re: self-hosted runners -> Would it be possible to come back about the discussion on Google Donating some credits to us ? The previous ones expired, and we did not use them to the full extent as we were not ready, but right now with the current GitHub Actions full support for running self-hosted runners in Google Cloud https://github.blog/2020-08-04-github-actions-self-hosted-runners-on-google-cloud/ I am sure we can start using those immediately.

[potiuk]

Seems also that splitting to smaller jobs has this side effect that it makes competing with other projects much harder. If there are many projects that are holding their slots for a long time, we can get starved rather heavily ( I think this is what is happening now). I am thinking about combining the jobs back, but in a smart way.

Instead of having full set of tests as matrix strategy, I think we can run them sequentially in one job and perform clean-up after each "type" of tests. This way we can avoid the Pylint memory leaks and in-container memory leaks. I can draft it quickly after we get 2.0alpha out.

We can clean-up file-system and run docker system prune between each type of tests. I think that would be the best of both worlds.

WDYT @kaxil @ashb @dimberman ?

[ashb]

@potiuk Sounds good! lets try it!

[kaxil]

Yes please, +1

[potiuk]

Got it working - also combined the jobs for kubernetes the same way.

[potiuk]

Closed - already superseded