Description
What would you like added?
Something like a new jobs.<job_id>.runs-on.tags definition that would let me specify a variable in the workflow that is passed on to the arc-runner-set-listener.
I would like to spark a discussion with this thread to align my contribution efforts correctly :)
Why is this needed?
Hi all!
My goal is to spawn VMs with different Linux kernels for the workflow steps to be executed in.
For this I deployed ARC with a custom modified runner image (kubevirt-actions-runner)[https://github.com/zhaofengli/kubevirt-actions-runner/tree/main] which spawns ephemeral KubeVirt VMs for the actions runner to work in.
This is a simplified example of my workflow so far:
name: Run kernel tests
on:
pull_request:
env:
KERNEL_REF: "${{ github.event.pull_request.head.sha }}"
KERNEL_TREE: "https://github.com/${{ github.repository }}"
#This workflow requires two k8s actions-runner-controllers (ARC) to be active.
#One for building the kernel and the other one for spawning the VM resource.
jobs:
build-kernel:
#This step runs in a container in the k8s cluster
runs-on: arc-kernel-builder
steps:
- name: Checkout blktests-ci
uses: actions/checkout@v4
with:
repository: linux-blktests/blktests-ci
path: blktests-ci
- name: Build kernel and package it into a containerimage
run: |
cd blktests-ci/playbooks/roles/kernel-builder-k8s-job/templates
docker build \
--build-arg KERNEL_TREE=${KERNEL_TREE} \
--build-arg KERNEL_REF=${KERNEL_REF} \
-t linux-kernel-containerdisk \
-f Dockerfile.linux-kernel-containerdisk . 2>&1 | tee build.log
#Setting KERNEL_VERSION var which is latern needed for notifying the VM what kernel to pick up
cat build.log | grep KERNEL_VERSION | awk '{print $3}' | grep KERNEL_VERSION >> $GITHUB_ENV
- name: Push the new Fedora containerimage with the freshly build kernel
run: |
docker tag linux-kernel-containerdisk registry-service.docker-registry.svc.cluster.local/linux-kernel-containerdisk:${KERNEL_VERSION}
docker push registry-service.docker-registry.svc.cluster.local/linux-kernel-containerdisk:${KERNEL_VERSION}
- name: Notifying the next job to pick up the Fedora image with the correct tag that we just build
run: |
echo "${KERNEL_VERSION}"
run-tests-on-kernel:
#This job _should_ run in a VM with the previously compiled kernel in the k8s cluster
runs-on: arc-vm-runner-set
needs: build-kernel
steps:
- name: Print VM debug info
run: |
uname -a
cat /etc/os-release
Notes:
- The GitHub workflow itself builds the kernel I want to test in a prepending job (which is
runs-ona different label/machine). The kernel artifacts are then uploaded to an artifactory. - In the second job (which depends on the fist job) I am specifying
runs-on: arc-vm-runner-setthat triggers ARC to spawn the VM.
For this the custom runner image does some REST API calls to grep the previous build kernel artifact version which is used to spawn the VM.
This notion of printing out the KERNEL_VERSION which was just build and then doing REST API calls to grep it in the next depending job is a hacky setup which breaks with multiple concurrent workflow runs. (VMs with a given kernel pick up the test run from a workflow that build a different kernel).
Ideally I would want to have something like a new jobs.<job_id>.runs-on.tags definition that would let me specify a (kernel version) variable in the workflow that is passed on to the arc-runner-set-listener.
The actions runner within the arc-vm-runner-set could then be configured to just picks up jobs with that specific tags.
That way I could force the VM that is being spawned to just pick up the kernel testing job for the same workflow run where the same kernel was build in.
After looking through the ARC code it looks to me that the arc-runner-set-listener only passes information like the jitconfig, some annotations (patch-id, runner-group-name, runner-scale-set-name, runner-spec-hash) and the static env vars from the values.yaml to the runner pod.
I would be more than happy to drive this new feature if this sounds like a sane idea.
Otherwise, If I am overlooking something please let me know.
I would like to spark a discussion with this thread to align my contribution efforts correctly :)
If someone has a completely different workflow idea on how to achieve what I described I would love to hear that as well :)