Adopt a UCI-sponsored self-hosted GitHub Actions runner

[Yicong-Huang]

Hi everyone,

I'd like to start a discussion on whether Apache Texera (Incubating) should adopt a self-hosted GitHub Actions runner sponsored and hosted by UC Irvine (UCI).

Motivation

The motivation of adopting self-hosted runners are:

• CI queueing. Apache projects share the same pool of GitHub-hosted runners, so our jobs frequently wait in the queue during busy periods despite our relatively low usage.
• Benchmark reproducibility. GitHub-hosted runners vary in hardware and share resources with unrelated workloads, making benchmark results noisy.

We do not have a GitHub Actions quota issue. Texera currently uses about 240k minutes/month (~60k/week), only 24% of ASF's weekly allocation.

Existing practice in ASF

• Apache Airflow uses self-hosted GitHub Actions runners.
• Apache Flink runs CI on sponsored Azure Pipelines.
• Apache Arrow maintains dedicated benchmark machines.

I do not recommend Spark's fork-CI model, where contributors run CI from their own forks. Besides the contributor experience, it does not solve benchmark reproducibility.

Proposal

Use a UCI-sponsored self-hosted runner for trusted workflows.

Proposed Machine Spec (Up to revisions)

• 64 CPU cores
• 256 GB RAM
• Ephemeral GitHub Actions runners

Routing

• Fork PRs → ubuntu-latest
• merge_group / trusted branches → self-hosted

This follows the same general approach as Apache Airflow.

Compared to GitHub-hosted runners, this removes shared-runner contention while also enabling warm build caches and a consistent benchmark environment.

Risks & ownership

Benchmark isolation. CPU pinning alone may not be sufficient. One option is to reserve dedicated benchmark windows.

Host failure. CI can fall back to GitHub-hosted runners by changing the runner label.

Security. Self-hosted runners remain ephemeral and execute only trusted workflows. Fork PRs continue to use GitHub-hosted runners.

Ownership. The machine would be hosted and maintained by UC Irvine. ASF Infra would only manage the runner registration.

Discussion

I'd appreciate feedback on:

• whether this direction makes sense;
• benchmark isolation;
• operational ownership;
• whether a single machine is sufficient.

If there is consensus, I'll follow up with a [VOTE] on the dev mailing list and an ASF INFRA request.

[parshimers]

I maintain an older but analogous system for AsterixDB's build (Jenkins instead of GHA). I will say that it is a bit of a task to manage, and often one that is kind of a chore. Often I think folks really only notice it when it breaks, not when something is done to improve or maintain it. That's just a perennial problem in any infrastructure, however. My views on this might be biased by being tasked with Jenkins which is really a troublesome system; GHA should be less annoying.

For benchmarks it makes a lot of sense.. It's also easier to just be able to alter or look at something directly if you manage it. It's come in handy many times, particularly in the earlier days of AsterixDB when there were more odd bugs lurking about that required special setups to pin down.

I think a machine of that spec should be sufficient. I actually migrated all of the previous builders for AsterixDB onto a machine of similar spec a few years ago (Threadripper 5975WX with 256GB, using VMs to parallelize) and it has been just fine. Before that it was an assortment of discrete machines (NUCs and others) but this lead to a lot of maintenance overhead, because they weren't uniform and ephemeral (e.g. PXE booted, or something similar).

[sshiv012]

NVM, Yes, whatever this dude says! He's the Jenkins whisperer :)

[parshimers]

I only whisper to it because it bites me if I speak normally :) I think I owe you a soda, because we raised 99% of the same points. Your point about making the builders ephemeral VMs is a very good one; I would definitely do that first. I am in the middle of trying to migrate some existing things to that, and it's a lot easier to do from the first place rather than migrating.

[Yicong-Huang]

yeah I had experiences with Jenkins and we chose GitHub Actions for its simplicity and native integration with GitHub. I do concern about management overhead on self-hosted runners, and I am trying to propose some ideas in the above "Risks & ownership" section. We may need to have fall back plans such that when self-runners time out, fall back to GitHub default runners, which eventually adds more end to end time on CI, I hope that happens not that frequently.

Thanks for confirming the hardware resource! I am a bit concerned on Texera's future CI resources, thus wanted to be conservative but also make sure it is enough for the near future as we add more legs in testing infra.

building ephemeral VMs is almost a must have for security purposes.

[sshiv012]

Hello Yicong, thank you for starting this discussion. I think a UCI-sponsored runner could be valuable, particularly for reducing queueing and making performance tests more reproducible. We operate something similar for Apache AsterixDB at UCI. A UCI-hosted Jenkins installation receives repository webhooks and can execute builds in parallel, reporting test results back to Gerrit (also UCI-hosted) as verification votes. AsterixDB uses Gerrit as its primary review system and mirrors the repository to GitHub, so the integration differs from Texera’s GitHub-centered workflow, but the operational experience is relevant: dedicated hardware gives us greater control over parallelism, installed software, caches, and performance-test conditions.

For Texera, I would initially favor keeping GitHub Actions as the workflow and reporting layer and supplying ephemeral UCI-hosted Actions runners underneath it. Moving to a separate Jenkins controller would introduce another CI configuration, user interface, authentication model, and failure surface. Jenkins becomes more attractive if we later need capabilities that are difficult to express in Actions, for example, managing a pool of machines, scheduling long-running benchmark campaigns, or coordinating deployments to persistent testing environments. Jenkins also has support/integration with Github Actions.

Some things worth noting would be

• security issues : https://infra.apache.org/self-hosted-runners.html. ASF permits self-hosted Actions runners but explicitly warns about their security risks. Fork PRs should never execute on the UCI machines. Trusted jobs should also receive minimal credentials and run in disposable VMs or containers, with no access to UCI internal networks or the host.
• staging env: The same UCI infrastructure could eventually support a shared Texera staging environment, which would be useful for integration testing and demonstrations. However, I would keep staging logically and operationally separate from CI runners. Machines that execute repository code should not share credentials, writable storage, or a network trust boundary with a persistent deployment.

[Yicong-Huang]

Thanks @sshiv012 for bringing these context! We want to keep using Github Action workflows to manager our CI and it should be straightforward to configure self-hosted runners. Thanks for confirming the experience from AsterixDB side.

Regarding the two notes:

1. security issue, yes I am fully aware of these. Thus in my proposal we want to keep Fork PRs → ubuntu-latest with the default runners, and only route trusted jobs to self-hosted runners. We need to pay additional attention to the UCI network and minimal credentials, thanks for the reminder!
2. staging env, actually, we have an ongoing effort led by @Ma77Ball and we are already testing it. The idea is to have 1 staging env for main (dev branch) and 1 staging server for an active release branch (e.g., release/v1.2). @Ma77Ball please share more details and link to references.