Request: reduce the amount of time used on Apache shared runners

[alamb]

TLDR: Could you please consider reducing the amount of github hosted runner time this project is consuming? 🙏

Hi,

I am a member of the Apache DataFusion and Arrow PMCs. Our CI jobs that run on github hosted runners are taking hours to start due to other jobs running in the (shared) apache organization which is severely impacting our project's progress.

While investigating what our jobs were waiting for, I was continually pointed at this repository.

117 Runner-Days of time in the last 24 hours, 20x more than DataFusion (updated)

I have a script that computes the total amount of github hosted runner time in hours spend for each project (attached below)

The airflow project is currently using almost 20x more than the next highest project I can find, to the tune of almost 120 days of runner time in each 24 hour window.

On average, at any time, Apache Airflow has 128 jobs running on the shared runners

andrewlamb@Andrews-MacBook-Pro-3:~/Downloads$ python3 usage.py --time 24h apache/airflow
1535 runs of apache/airflow created since 2026-10-21T09:36:30Z (24.0 h window)
jobs loaded: 1469 runs from cache, 66 from the API, 0 failed  (cache: gh-runner-usage-cache)

conclusion     jobs  share runner-min runner-days
success       19020  58.7%     169845       117.9
skipped       10557  32.6%          0         0.0
cancelled      2417   7.5%      12016         8.3
failure         256   0.8%       2941         2.0
queued           70   0.2%          0         0.0
in_progress      69   0.2%          0         0.0
total         32389            184802       128.3

average concurrent jobs over the window: 128

Here is the script (and the data to save you some GH API credits): usage-airflow.zip

You can see more analysis / discussion on these issues

• CI takes hours to start datafusion#25088
• ASF Slack Discussion: https://the-asf.slack.com/archives/CBX4TSBQ8/p1792520326623569
• https://issues.apache.org/jira/projects/INFRA/issues/INFRA-28560?filter=allopenissues

I looked at this repo briefly to see if there was something obvious that was consuming all the time that maybe could reduce the time spent. I think it may just be that each commit runs significant amount of work on CI.

I noticed that each commit in stacked PRs kicks off the entire CI suite, for example that I discussed with @ephraimbuddy in this PR) but I don't think this explains the entire usage

• Add HTTP publication to the Dag processor manager #74793

cc @kaxil and @potiuk as I see your names on many approvals / PRs.

Thank you for your consideration

[hubcio]

+1 from apache/iggy PMC. it would be great if you would consider to tone down the usage, due to recent long queues i created issue to infra.

[ephraimbuddy]

I noticed that each commit in stacked PRs kicks off the entire CI suite, for example that I discussed with @ephraimbuddy in this PR) but I don't think this explains the entire usage

I'm going to check this but I don't think it's each commit triggering every other PRs CI in the stack. I think when the PRs are restacked is when all their respective CIs kicks off.

[potiuk]

Thanks for this issue - it is absolutely what we are doing now. In case you have not noticed we are well in the process of doing just that (and already implemented some of earlier - easier optimisations). I thin everyone (including @hubcio and @alamb in your projects) - you should have similar approach we do - where we, proactively - before issues like that - plan and relentlessly execute optimizations efforts.

I would really love if all the projects do that - including those projects which experienced the queuing time now. Everyone's small optimisation count, and I hope Airflow blueprinit on how to do that might be followed.

All of the optimizations we tried and plan are also described in this PR to the infra website - apache/infrastructure-website#320 where we share and ask others to try similar things.

• Document fork CI for contributors and maintainers #74825 and https://lists.apache.org/thread.html/nfno4r1jqx5dj75v867prjjgg6o77odx and https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974711/AIP-120+Run+CI+for+external+contributions+in+contributor+forks

• Run maintainer and canary CI on self-hosted RunsOn runners #74889 tha - https://lists.apache.org/thread.html/lfgvvffhtzgpbyy3dm197vftsg448qpx and https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974652/AIP-118+Self-hosted+runners+for+Apache+Airflow+CI

My estimate is that it will reduce Airflow usage by about 80% overall.

Let me close the issue - because it's been outdated before it's been opened.

[alamb]

Thank you -- it is great you are already working on it. I agree that it will be helpful for all projects to reduce the runner usage

We are madly trying to do the same thing in our projects. For example, we recently switch DataFusion so CI mostly runs in contributor forksL

• ci: run the Rust workflow for fork pull requests on the fork datafusion#25591 (thanks to @kumarUjjawal for that)

[alamb]

And to be clear DataFusion has used lots of runner time in the past as well, though mostly of our heavy jobs are now on Hosted RunsOn runners

[potiuk]

And yes - we usually have 10x more traffic / PRs than the next biggest project - but last week is an outlier as we prepare for 3.4.0 release and there is a lot of cherry-picking involved. Usually - and i keep on monitoring it at https://infra-reports.apache.org/#ghactions - we hover around 6 FT runners, but this last 7 days we have 17 - that is indeed "a lot".

If you look at last 30 days average is 10 FT. I hope to reduce it to ~ 5 at peak. Most of the "Release preparation" jobs will be anyhow taken over by our self-hosted runners when we enable it - I hope this weekend.

[potiuk]

cc: @shahar1 -> let's focus on switching to the self-hosted over the weekend. Ping me on slack any time and we will complete it faster - this will defintely help @alamb and @hubcio and others - and possibly we can share our "blueprint" - step-by-step how to do the RunsOn enablement when you have AWS credits. This might be super helpful for infra and other ASF teams to follow it.

[alamb]

BTW my post above mis stated -- this repo uses 20x more runner time than Datafusion, not the next highest repo

The next highest repo I found was apache/beam actually which uses like 1/2 of this runner time

andrewlamb@Andrews-MacBook-Pro-3:~/Downloads$ python3 usage.py --time 24h apache/beam
2700 runs of apache/beam created since 2026-10-21T09:59:46Z (24.0 h window)
jobs loaded: 706 runs from cache, 1994 from the API, 0 failed  (cache: gh-runner-usage-cache)

conclusion     jobs  share runner-min runner-days
success        4864  80.3%      85981        59.7
cancelled       691  11.4%       7720         5.4
skipped         327   5.4%          0         0.0
failure         149   2.5%       5826         4.0
in_progress      27   0.4%          0         0.0
neutral           3   0.0%          0         0.0
total          6061             99528        69.1

average concurrent jobs over the window: 69