Add automated Kernel CI workflow #675

shreeya-patel98 · 2025-11-07T11:05:49Z

Implements a 5-stage GitHub Actions pipeline for automated kernel testing and PR creation.
Uses kernel-container-build automated-testing-v1 branch for build/test tooling.

Stage 1: Build (15-30 min)

Checkout kernel source + kernel-container-build repo (automated-testing-v1)
Build kernel in CIQ builder container with kABI checking
Convert built container to QCOW2 VM image
Upload: kernel-build.log, QCOW2 image

Stage 2: Boot Verification (2-5 min)

Download QCOW2 image
Boot kernel in QEMU (KVM or TCG) and validate login prompt appears
Upload: boot logs

Stage 3: Kernel Selftests (40-60 min)

Download QCOW2 image
Execute comprehensive kselftests in QEMU with dual serial consoles
Upload: kselftest TAP logs, dmesg output

Stage 4: Compare Results (1-2 min)
Purpose: Detect test regressions by comparing against base branch

Steps:

Checkout with full history (fetch-depth: 0)
Download current kselftest logs
Fetching base branch:
- For pushes: Fetches the base branch where the patches needs to be
  merged using the user branch name.
  Example :- If user branch pushed is {shreeya}_ciqlts9_2 then
  ciqlts9_2 is the base branch.
- For force-pushes: Doesn't do the base branch fetching again, instead uses the
  same base branch as detected during the first PR creation and also
  compares the result against that base branch.
- Outputs: base_branch (reused by PR stage)
Download baseline logs from base branch (searches last 5 successful runs)
Compare results:
- Counts passing/failing tests (before/after)
- Fails if >±3 tests changed
- Outputs: comparison_status, comparison_message

Stage 5: Create Pull Request (1-2 min)
Purpose: Auto-create/update PR after all tests pass

Prerequisites: Only runs if build + boot + kselftest passed, no regressions detected

Steps:

Check all stages passed and comparison_status != failed
Checkout (shallow: fetch-depth: 100) for commit messages
Download all artifacts (build/boot/test logs)
Extract statistics (pass/fail counts, build times)
Get commit info:
- Single commit: Use commit message
- Multiple commits: Create summary
Create/Update PR:
- Reuses base_branch from compare-results (no duplication!)
- Generate PR body with test results via create-pr-body.sh
- Creates new PR or updates existing one in case of force pushes

Note :-

To skip this github action workflow, add [ci skip] or [skip ci]
into the head commit when pushing the patches.
For the CI to generate automatic PRs against a valid branch,
one needs to have the user branch name in the format of
{any_name}_basebranch.
Braces and base branch name in the user branch is necessary or else
PR creation stage will fai}_basebranch.
Braces and base branch name in the user branch is necessary or else
PR creation stage will fail.

Signed-off-by: Shreeya Patel spatel@ciq.com

.github/workflows/kernel-build-and-test-x86_64.yml

.github/scripts/create-pr-body.sh

.github/workflows/kernel-build-and-test-x86_64.yml

.github/scripts/create-pr-body.sh

.github/workflows/kernel-build-and-test-x86_64.yml

PlaidCat · 2025-11-12T23:49:00Z

.github/workflows/kernel-build-and-test-x86_64.yml

+
+    - name: Download baseline kselftest logs from base branch
+      if: steps.base_branch.outputs.base_branch != ''
+      uses: dawidd6/action-download-artifact@v3


this appear to be a user action, we should do a code audit of this.

@bmastbergen has mentioned this before too, maybe as CIQ we should be authoring our own sort of actions like this so that we can contribute to additional community members and we know a company is behind an action code we're running

It makes me nervous using third-party actions from who knows where. That being said, they can be extremely useful. I guess the official download artifacts action can't download artifacts from other workflows/branches/PRs which we obviously need here. I don't think we need to hold this PR up by writing our own version of this right now, unless its really trivial. I got as far as "oh great it uses node.js" when looking at dawidd6/action-download-artifact

yes, I wasn't able to find any quick solution for this.

Ok, this looks like relatively stable develeopment and and has users here and there committing beyond the repo owner.

https://github.com/dawidd6/action-download-artifact/graphs/contributors

It also seeming to be used a lot
https://github.com/dawidd6/action-download-artifact/network/dependents

However can we try using the latest release v11 or at least v6
GHSA-5xr6-xhww-33m4

Update the version and I'll approve.

Updated the version to v11, will test it once

Implements a 5-stage GitHub Actions pipeline for automated kernel testing and PR creation. Uses kernel-container-build automated-testing-v1 branch for build/test tooling. Stage 1: Build (15-30 min) - Checkout kernel source + kernel-container-build repo (automated-testing-v1) - Build kernel in CIQ builder container with kABI checking - Convert built container to QCOW2 VM image - Upload: kernel-build.log, QCOW2 image Stage 2: Boot Verification (2-5 min) - Download QCOW2 image - Boot kernel in QEMU (KVM or TCG) and validate login prompt appears - Upload: boot logs Stage 3: Kernel Selftests (40-60 min) - Download QCOW2 image - Execute comprehensive kselftests in QEMU with dual serial consoles - Upload: kselftest TAP logs, dmesg output Stage 4: Compare Results (1-2 min) Purpose: Detect test regressions by comparing against base branch Steps: 1. Checkout with full history (fetch-depth: 0) 2. Download current kselftest logs 3. Fetching base branch: - For pushes: Fetches the base branch where the patches needs to be merged using the user branch name. Example :- If user branch pushed is {shreeya}_ciqlts9_2 then ciqlts9_2 is the base branch. - For force-pushes: Doesn't do the base branch fetching again, instead uses the same base branch as detected during the first PR creation and also compares the result against that base branch. - Outputs: base_branch (reused by PR stage) 4. Download baseline logs from base branch (searches last 5 successful runs) 5. Compare results: - Counts passing/failing tests (before/after) - Fails if >±3 tests changed - Outputs: comparison_status, comparison_message Stage 5: Create Pull Request (1-2 min) Purpose: Auto-create/update PR after all tests pass Prerequisites: Only runs if build + boot + kselftest passed, no regressions detected Steps: 1. Check all stages passed and comparison_status != failed 2. Checkout (shallow: fetch-depth: 100) for commit messages 3. Download all artifacts (build/boot/test logs) 4. Extract statistics (pass/fail counts, build times) 5. Get commit info: - Single commit: Use commit message - Multiple commits: Create summary 6. Create/Update PR: - Reuses base_branch from compare-results (no duplication!) - Generate PR body with test results via create-pr-body.sh - Creates new PR or updates existing one in case of force pushes Note :- 1. To skip this github action workflow, add [ci skip] or [skip ci] into the head commit when pushing the patches. 2. For the CI to generate automatic PRs against a valid branch, one needs to have the user branch name in the format of {any_name}_basebranch. 3. Braces and base branch name in the user branch is necessary or else PR creation stage will fai}_basebranch. 3. Braces and base branch name in the user branch is necessary or else PR creation stage will fail. Signed-off-by: Shreeya Patel <spatel@ciq.com>

Script to generate detailed PR descriptions with kselftest results. Signed-off-by: Shreeya Patel <spatel@ciq.com>

roxanan1996 · 2025-11-21T13:33:18Z

.github/workflows/kernel-build-and-test-x86_64.yml

+
+    - name: Create Pull Request
+      env:
+        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}


Please include this fix cddb0db

So that the pull request workflow is triggered

roxanan1996

I requires a fix to trigger the pull request workflow when the pull request is created

roxanan1996 · 2025-11-21T15:44:24Z

.github/workflows/kernel-build-and-test-x86_64.yml

+        BRANCH_NAME="${{ github.ref_name }}"
+
+        # Define whitelist of valid base branches
+        VALID_BASES="ciqlts9_2 ciqlts9_4 ciqlts8_6"


Minor: Can you put a TODO here? I hope soon to use a centralized place from where you can take the base branches.

roxanan1996 · 2025-11-21T15:58:22Z

.github/scripts/create-pr-body.sh

+This PR has been automatically created after successful completion of all CI stages.
+
+## Commit Message(s)
+\`\`\`


I personally find this very hard to read, especially for pull requests with multiple commits.
See #716.

I would remove the ```.

And guard every separate commit with ```.

This is how everyone has been doing it "manually"
Examples:
#711
#703

Yes, I agree with you on this but I thought of not spending too much time in making this look very nice at this stage. My focus was to enable this on multiple platforms first as we have only been able to work on ciqlts9_2 as of now so there is a lot of things to be do yet.
Testing these kind of changes also takes a lot of time because I have to go through all the stages to only test the formatting :(
Anyway since I already had a comment to work on from Maple, I made these changes too and also the ones that you pointed out below.

Yeah lets get the 80-90% done here and make sure it works frequently

roxanan1996 · 2025-11-21T16:28:49Z

.github/workflows/kernel-build-and-test-x86_64.yml

+          echo "commit_subject=Multiple patches tested ($COMMIT_COUNT commits)" >> $GITHUB_OUTPUT
+
+          # Get all commit messages and save to file
+          git log origin/$BASE_BRANCH..HEAD --pretty=format:"### %s%n%n%b%n---" > /tmp/commit_message.txt


As mentioned in a previous comment, I would guard every separate commit with ```.
Moreover, the full commit message adds a lot of noise.

This should work (adapted from my local script to create pull requests)

COMMIT_SUBJECT=$(cat /tmp/commit_subject.txt) echo "commit_subject=$COMMIT_SUBJECT" >> $GITHUB_OUTPUT - # Save full commit message to file - git log -1 --pretty=%B > /tmp/commit_message.txt else # Multiple commits: create summary echo "commit_subject=Multiple patches tested ($COMMIT_COUNT commits)" >> $GITHUB_OUTPUT - - # Get all commit messages and save to file - git log origin/$BASE_BRANCH..HEAD --pretty=format:"### %s%n%n%b%n---" > /tmp/commit_message.txt fi + # Get all commit messages and save to file (in reverse order) + for commit in $(git log origin/$BASE_BRANCH..HEAD --format=%h | tac) + git log -1 $commit --format=%B | awk 'BEGIN{print "```"} /^$/{empty++} empty==2{exit} {print} END{print "```"}' >> /tmp/commit_message.txt + - name: Create Pull Request

Thanks, I've added this change but will need to test it.

shreeya-patel98 requested a review from a team November 7, 2025 11:05

shreeya-patel98 self-assigned this Nov 7, 2025

shreeya-patel98 requested review from PlaidCat, bmastbergen and kerneltoast as code owners November 7, 2025 11:05

shreeya-patel98 force-pushed the shreeya_kernelci_main branch from 53a82b2 to 965ef63 Compare November 11, 2025 14:23

roxanan1996 reviewed Nov 11, 2025

View reviewed changes

.github/workflows/kernel-build-and-test-x86_64.yml Show resolved Hide resolved

roxanan1996 reviewed Nov 12, 2025

View reviewed changes

.github/scripts/create-pr-body.sh Show resolved Hide resolved

.github/workflows/kernel-build-and-test-x86_64.yml Outdated Show resolved Hide resolved

.github/workflows/kernel-build-and-test-x86_64.yml Outdated Show resolved Hide resolved

PlaidCat reviewed Nov 12, 2025

View reviewed changes

.github/scripts/create-pr-body.sh Outdated Show resolved Hide resolved

PlaidCat reviewed Nov 12, 2025

View reviewed changes

.github/scripts/create-pr-body.sh Show resolved Hide resolved

PlaidCat reviewed Nov 12, 2025

View reviewed changes

.github/workflows/kernel-build-and-test-x86_64.yml Show resolved Hide resolved

PlaidCat reviewed Nov 12, 2025

View reviewed changes

.github/workflows/kernel-build-and-test-x86_64.yml Show resolved Hide resolved

PlaidCat reviewed Nov 12, 2025

View reviewed changes

shreeya-patel98 force-pushed the shreeya_kernelci_main branch from 965ef63 to f8d9d1c Compare November 18, 2025 11:46

shreeya-patel98 added 2 commits November 20, 2025 12:02

Add PR body generation script

9982c5c

Script to generate detailed PR descriptions with kselftest results. Signed-off-by: Shreeya Patel <spatel@ciq.com>

shreeya-patel98 force-pushed the shreeya_kernelci_main branch from f8d9d1c to 9982c5c Compare November 20, 2025 12:13

roxanan1996 reviewed Nov 21, 2025

View reviewed changes

roxanan1996 requested changes Nov 21, 2025

View reviewed changes

roxanan1996 reviewed Nov 21, 2025

View reviewed changes

Add automated Kernel CI workflow #675

Are you sure you want to change the base?

Add automated Kernel CI workflow #675

Conversation

shreeya-patel98 commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PlaidCat Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreeya-patel98 Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PlaidCat Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roxanan1996 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreeya-patel98 Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

shreeya-patel98 commented Nov 7, 2025 •

edited

Loading

PlaidCat Nov 12, 2025 •

edited

Loading

shreeya-patel98 Nov 20, 2025 •

edited

Loading

PlaidCat Nov 20, 2025 •

edited

Loading

shreeya-patel98 Nov 21, 2025 •

edited

Loading