CNTRLPLANE-3766: add chai-bot CI daily health report instructions#8918
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds a generated HyperShift CI job registry, a scheduled workflow that refreshes and publishes that registry, and a Markdown spec for a daily CI health report. The report defines build collection fallbacks, pass-rate and trend calculations, top-level summary formatting, threaded failure analysis for low-health categories, and a fixed set of failure-pattern hints. 🚥 Pre-merge checks | ✅ 11✅ Passed checks (11 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (3)
.chai-bot/hypershift_ci_daily_health_report.md (3)
17-19: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd language hints to fenced code blocks.
markdownlint flags these fences (Lines 17, 32, 65, 76, 100) for missing a language annotation.
📝 Suggested fix (apply to each flagged fence)
-``` +```text https://raw.githubusercontent.com/openshift/hypershift/refs/heads/main/.chai-bot/ci-status-jobs.yamlAlso applies to: 32-34, 65-73, 76-78, 100-104
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.chai-bot/hypershift_ci_daily_health_report.md around lines 17 - 19, Add language hints to all fenced code blocks in the report markdown, since markdownlint is flagging the fences as missing annotations. Update each affected fence in the hypershift CI daily health report so the blocks are explicitly tagged with an appropriate language (for example, the referenced URL-only fence and the other flagged snippets), keeping the content unchanged while ensuring the fences are identifiable in the markdown.Source: Linters/SAST tools
7-7: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPlatform list in the Goal doesn't match the actual registry categories.
Line 7 enumerates platforms as "AWS, Azure, KubeVirt, OpenStack, IBM/PowerVS, MCE Agent, HCM, GKE" as if each were its own report category, but the registry actually groups Azure+KubeVirt together per version, and GKE is a job inside the combined "HCM & Other" category rather than a standalone category. This mismatch could confuse whoever implements/maintains the categorization logic against
ci-status-jobs.yaml.✏️ Suggested wording fix
-Monitor periodic Prow CI jobs for HyperShift across all supported platforms (AWS, Azure, KubeVirt, OpenStack, IBM/PowerVS, MCE Agent, HCM, GKE). Compute per-category pass rates from the last 10 completed builds, identify trends, and post a summary to the channel. Provide threaded failure analysis for categories below 80%. +Monitor periodic Prow CI jobs for HyperShift across the categories defined in the job registry (AWS 5.0/4.23, Azure & KubeVirt 5.0/4.23, OpenStack, IBM/PowerVS, MCE Agent, HCM & Other). Compute per-category pass rates from the last 10 completed builds, identify trends, and post a summary to the channel. Provide threaded failure analysis for categories below 80%.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.chai-bot/hypershift_ci_daily_health_report.md at line 7, The Goal text is listing platforms as if they match the reporting categories, but the registry grouping is different. Update the wording in the hypershift_ci_daily_health_report.md goal to reflect the actual category structure used by ci-status-jobs.yaml, especially the combined Azure+KubeVirt per-version grouping and GKE being part of the “HCM & Other” category. Keep the intent of monitoring all supported jobs, but make the category names and breakdown align with the real registry taxonomy so implementers can map the report logic correctly.
130-223: 🗄️ Data Integrity & Integration | 🔵 Trivial | 🏗️ Heavy liftFallback registry duplicates the source-of-truth YAML with no sync safeguard.
This embedded fallback (Lines 134-223) is a manual copy of
.chai-bot/ci-status-jobs.yaml. Since Step 1's fallback logic silently switches to this copy on fetch failure, any future addition/removal of jobs in the primary registry that isn't mirrored here will cause the bot to silently report against a stale job list during outages, with no indication to readers that fallback data was used.Consider either (a) adding an explicit note in the top-level message when the fallback registry is used, so staleness is visible, or (b) adding a periodic CI check that diffs this embedded block against
ci-status-jobs.yamlto catch drift.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.chai-bot/hypershift_ci_daily_health_report.md around lines 130 - 223, The embedded fallback registry in the daily health report is a manual copy of the source-of-truth job YAML and can drift silently when Step 1 falls back to it. Update the logic around the fallback registry block in hypershift_ci_daily_health_report.md so readers are explicitly told when the embedded registry is being used, or add a CI safeguard that compares this fallback content against ci-status-jobs.yaml to detect mismatches. Reference the fallback registry section and the Step 1 fallback path so the change is applied where the stale data can be selected.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In @.chai-bot/hypershift_ci_daily_health_report.md:
- Around line 17-19: Add language hints to all fenced code blocks in the report
markdown, since markdownlint is flagging the fences as missing annotations.
Update each affected fence in the hypershift CI daily health report so the
blocks are explicitly tagged with an appropriate language (for example, the
referenced URL-only fence and the other flagged snippets), keeping the content
unchanged while ensuring the fences are identifiable in the markdown.
- Line 7: The Goal text is listing platforms as if they match the reporting
categories, but the registry grouping is different. Update the wording in the
hypershift_ci_daily_health_report.md goal to reflect the actual category
structure used by ci-status-jobs.yaml, especially the combined Azure+KubeVirt
per-version grouping and GKE being part of the “HCM & Other” category. Keep the
intent of monitoring all supported jobs, but make the category names and
breakdown align with the real registry taxonomy so implementers can map the
report logic correctly.
- Around line 130-223: The embedded fallback registry in the daily health report
is a manual copy of the source-of-truth job YAML and can drift silently when
Step 1 falls back to it. Update the logic around the fallback registry block in
hypershift_ci_daily_health_report.md so readers are explicitly told when the
embedded registry is being used, or add a CI safeguard that compares this
fallback content against ci-status-jobs.yaml to detect mismatches. Reference the
fallback registry section and the Step 1 fallback path so the change is applied
where the stale data can be selected.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 601d32ec-b86b-458e-a49d-337c782b7bf9
📒 Files selected for processing (2)
.chai-bot/ci-status-jobs.yaml.chai-bot/hypershift_ci_daily_health_report.md
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8918 +/- ##
=======================================
Coverage 43.45% 43.45%
=======================================
Files 771 771
Lines 95718 95718
=======================================
Hits 41597 41597
Misses 51234 51234
Partials 2887 2887
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
celebdor
added
the
area/ci-tooling
georgelipceanu
left a comment
georgelipceanu
left a comment
There was a problem hiding this comment.
Left a small comment 😄.
| **Fallback method**: If Prow tools return no results, scrape the Prow job-history page: | ||
| ``` | ||
| https://prow.ci.openshift.org/job-history/gs/test-platform-results/logs/{JOB_NAME} | ||
| ``` | ||
| The page contains a JavaScript variable `var allBuilds = [...]` with objects containing `{ID, Result, Started, Duration}`. Parse this to extract build results. | ||
|
|
||
| For each build, record: | ||
| - Date (MonDD format, e.g., "Jul02") | ||
| - Result: `SUCCESS` or `FAILURE` | ||
| - Build ID (for linking to specific runs) |
There was a problem hiding this comment.
small nit but could it also be an idea to have a 2nd fallback to testgrid.k8s.io/redhat-hypershift in case Prow is down for whatever reason? although testgrid isn't our official "source of truth" we still use it in other skills (example in daily.md#L50) and it still is a useful tool. It's not entirely necessary but maybe something to consider, LGTM otherwise
|
|
||
| For each build, record: | ||
| - Date (MonDD format, e.g., "Jul02") | ||
| - Result: `SUCCESS` or `FAILURE` |
There was a problem hiding this comment.
Step 2 filters only "running/pending" and records exactly two outcomes: SUCCESS or FAILURE. Prow has two additional terminal states — ABORTED (preempted by Boskos timeout or resource pressure) and ERROR (infrastructure failure before the job runs). Both pass the filter and land in the failure bucket. A Boskos capacity wave can push a healthy category through the 80% threshold, triggering spurious threaded failure analysis and on-call noise for infra issues, not product regressions. Fix: explicitly filter or separately categorize ABORTED/ERROR builds.
|
|
||
| ### Step 2 — Collect Build History | ||
|
|
||
| For each job in the registry, collect the **last 10 completed builds** (skip any still running/pending). |
There was a problem hiding this comment.
Step 2 collects "last 10 completed builds." Step 3 computes a trend by comparing "most recent 7 days vs prior 7 days" — a 14-day window. For a once-daily job, 10 builds = 10 days; the prior-7-day window gets at most 3 data points. For twice-daily jobs (e.g., AWS conformance), 10 builds = 5 days and the entire prior window is empty — the bot always emits ➡️ Stable regardless of actual trend.
Fix: collect at least 20 builds, or reframe the trend as "last 5 vs prior 5 builds."
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.chai-bot/hypershift_ci_daily_health_report.md:
- Around line 15-23: The registry fetch step is using the wrong GitHub raw URL,
causing the fetch to fail and forcing the report to use the embedded fallback
registry. Update the URL used by the fetch logic in the report generation flow
to the main-branch raw path, and keep the existing fetch_web_content/YAML
parsing and fallback handling intact so the header only shows the fallback
warning when it is actually used.
- Around line 31-44: The TestGrid fallback in this report flow is too shallow
for Step 2/5 because it may only provide status without the last 20 completed
builds, build IDs, or failing-log links. Update the fallback logic around the
Prow history parsing path so it resolves build history from an underlying source
such as the Prow results_url_template or equivalent job-history data, and only
use TestGrid when it can supply those fields. Ensure the threaded analysis path
uses the same build-history source behind the fallback, not a status-only check.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 967f9a22-f0b7-4d0b-aefe-5825303b58ee
📒 Files selected for processing (1)
.chai-bot/hypershift_ci_daily_health_report.md
| Fetch the job registry from GitHub: | ||
|
|
||
| ```text | ||
| https://raw.githubusercontent.com/openshift/hypershift/refs/heads/main/.chai-bot/ci-status-jobs.yaml | ||
| ``` | ||
|
|
||
| Use `fetch_web_content` to retrieve this file. Parse the YAML to extract categories and their job lists. | ||
|
|
||
| **Fallback**: If the fetch fails, use the embedded registry at the bottom of this document. If using the fallback, add `(⚠️ using fallback registry)` to the report header so readers know the data source may be stale. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== File excerpt ==\n'
sed -n '1,40p' .chai-bot/hypershift_ci_daily_health_report.md
printf '\n== URL probe ==\n'
python3 - <<'PY'
import urllib.request, urllib.error, ssl
url = "https://raw.githubusercontent.com/openshift/hypershift/refs/heads/main/.chai-bot/ci-status-jobs.yaml"
ctx = ssl.create_default_context()
try:
with urllib.request.urlopen(url, context=ctx, timeout=20) as r:
print("status:", r.status)
print("final_url:", r.geturl())
print("content_type:", r.headers.get_content_type())
data = r.read(200).decode("utf-8", "replace")
print("first_bytes:", data[:200].replace("\n", "\\n"))
except Exception as e:
print(type(e).__name__ + ":", e)
PYRepository: openshift/hypershift
Length of output: 2309
🏁 Script executed:
python3 - <<'PY'
import urllib.request, ssl, urllib.error
url = "https://raw.githubusercontent.com/openshift/hypershift/refs/heads/main/.chai-bot/ci-status-jobs.yaml"
ctx = ssl._create_unverified_context()
try:
with urllib.request.urlopen(url, context=ctx, timeout=20) as r:
print("status:", r.status)
print("final_url:", r.geturl())
print("content_type:", r.headers.get_content_type())
data = r.read(500).decode("utf-8", "replace")
print("body_preview:", data.replace("\n", "\\n"))
except Exception as e:
print(type(e).__name__ + ":", e)
PYRepository: openshift/hypershift
Length of output: 195
Fix the registry fetch URL
Use https://raw.githubusercontent.com/openshift/hypershift/main/.chai-bot/ci-status-jobs.yaml instead; the current refs/heads/main raw URL returns 404, so Step 1 will always fall back to the embedded registry and may report stale data.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.chai-bot/hypershift_ci_daily_health_report.md around lines 15 - 23, The
registry fetch step is using the wrong GitHub raw URL, causing the fetch to fail
and forcing the report to use the embedded fallback registry. Update the URL
used by the fetch logic in the report generation flow to the main-branch raw
path, and keep the existing fetch_web_content/YAML parsing and fallback handling
intact so the header only shows the fallback warning when it is actually used.
| jobs: | ||
| - periodic-ci-openshift-hypershift-release-4.22-periodics-hcm-e2e-aws | ||
| - periodic-ci-openshift-hypershift-release-4.22-periodics-hcm-e2e-aws-upgrade-minor | ||
| - periodic-ci-openshift-hypershift-release-5.0-periodics-hcm-azure-e2e-aks-upgrade-minor |
There was a problem hiding this comment.
how is this generated and how do we ensure it keeps up to date?
There was a problem hiding this comment.
could that maybe be another scheduled task that figures the jobs deterministically?
There was a problem hiding this comment.
The initial list was hand-curated. I've added hack/ci/update-job-registry.py which does a sparse checkout (tree:0 filter) of openshift/release, extracts all periodic job names from the hypershift job configs, and categorizes them by platform keywords. A nightly GitHub Action (.github/workflows/update-ci-job-registry.yaml) runs it and opens a PR if the registry changed — so it stays in sync automatically.
There was a problem hiding this comment.
Yes, exactly — a nightly GH Action now does this. The categorization is deterministic via keyword matching rules in the script (mce-* → MCE Agent, openstack → OpenStack, etc). The script picks up the 3 most recent release versions automatically.
|
|
||
| Use `fetch_web_content` to retrieve this file. Parse the YAML to extract categories and their job lists. | ||
|
|
||
| **Fallback**: If the fetch fails, use the embedded registry at the bottom of this document. If using the fallback, add `(⚠️ using fallback registry)` to the report header so readers know the data source may be stale. |
There was a problem hiding this comment.
what's the point of decoupling the registry in a separate file and having two sources? chai bot needs to fetch_web_content this instructions anyway, why not just provide the registry with them all together and maintain only that source?
There was a problem hiding this comment.
Good point. With the auto-generation, the separation makes sense: the registry is machine-generated (nightly GH Action regenerates it from openshift/release), while the report instructions are human-authored and stable. I've removed the embedded fallback entirely since the registry is now kept fresh automatically.
|
/override e2e-aks |
|
@celebdor: /override requires failed status contexts, check run or a prowjob name to operate on.
Only the following failed contexts/checkruns were expected:
If you are trying to override a checkrun that has a space in it, you must put a double quote on the context. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/override ci/prow/e2e-aks |
|
@celebdor: Overrode contexts on behalf of celebdor: ci/prow/e2e-aks, ci/prow/e2e-aws, ci/prow/e2e-aws-upgrade-hypershift-operator, ci/prow/e2e-azure-v2-self-managed, ci/prow/e2e-kubevirt-aws-ovn-reduced, ci/prow/e2e-v2-aws, ci/prow/e2e-v2-gke DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.chai-bot/hypershift_ci_daily_health_report.md (1)
44-45: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winSurface the ABORTED/ERROR warning outside threaded analysis.
As written, the >30% infra-health note only appears in Step 5, but Step 5 runs only for categories below 80% pass rate. That means a category can look healthy while its infra signal is never shown.
Also applies to: 97-120
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.chai-bot/hypershift_ci_daily_health_report.md around lines 44 - 45, The ABORTED/ERROR infrastructure-health note in the daily health report is only emitted inside the threaded analysis path, so it can be skipped for otherwise healthy categories. Update the report logic that builds the hypershift CI summary to surface the >30% ABORTED/ERROR warning outside the Step 5 branch, using the existing pass-rate/infra classification flow so it is shown whenever the threshold is exceeded. Keep the handling in the same report-generation area that computes job result buckets and threaded analysis so the warning is independent of whether the category drops below 80% pass rate.
🧹 Nitpick comments (4)
hack/ci/update-job-registry.py (1)
130-196: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd unit tests for the categorization logic.
categorize_job/build_categoriesare described as "the single source of truth" for how jobs map to report categories, but there's no accompanying test file exercising the platform matchers, version-split logic, or description formatting.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/ci/update-job-registry.py` around lines 130 - 196, Add unit tests for the categorization flow in categorize_job and build_categories, covering each PLATFORM_RULES matcher, the VERSION_SPLIT_PLATFORMS branch, and the description formatting paths. Create tests that verify uncategorized jobs are skipped, version filtering via selected_versions works, split platforms emit one Category per version, and non-split platforms aggregate versions into a single category with the expected description..github/workflows/update-ci-job-registry.yaml (3)
19-19: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPin the PyYAML version for reproducible nightly runs.
An unpinned
pip install pyyamlcan silently pull a new major version on any given night, causing an unattended failure or behavior change with no corresponding code diff to review.♻️ Proposed fix
- run: pip install --break-system-packages pyyaml + run: pip install --break-system-packages "pyyaml==6.0.2"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/update-ci-job-registry.yaml at line 19, The workflow step in update-ci-job-registry.yaml installs PyYAML without a version pin, which can make nightly runs non-reproducible. Update the pip install command in the workflow to pin PyYAML to a specific compatible version so the update-ci-job-registry job uses the same dependency each run. Keep the change localized to the existing run step that installs pyyaml.
12-15: 🩺 Stability & Availability | 🔵 TrivialConsider adding
concurrencyandtimeout-minutes.Two overlapping runs (manual
workflow_dispatchplus the nightly schedule, or a slow git clone) could race on the sameautomated/update-ci-job-registrybranch or tie up the runner indefinitely. Aconcurrency:group on the workflow and atimeout-minutes:on the job would bound both risks.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/update-ci-job-registry.yaml around lines 12 - 15, The update-ci-job-registry workflow job currently has no safeguards against overlapping executions or runaway runtime. Add a workflow-level concurrency group for this workflow so manual and scheduled runs cannot race on the same automated/update-ci-job-registry branch, and add a timeout-minutes limit to the update-registry job to bound slow clones or stalled steps. Use the existing update-registry job definition in the workflow to place these settings.
45-45: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winAvoid interpolating
${{ }}expressions directly into shell command strings.
${{ steps.generate.outputs.release_sha }}is expanded verbatim into therun:script before the shell parses it (commit message and PR body). The current value (a short git SHA) is low-risk, but this is a recognized GitHub Actions injection pattern — routing it through anenv:variable and referencing$RELEASE_SHAavoids the class of issue entirely at negligible cost.🔒️ Proposed fix
- name: Create pull request if: steps.diff.outputs.changed == 'true' env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + RELEASE_SHA: ${{ steps.generate.outputs.release_sha }} run: | BRANCH="automated/update-ci-job-registry" git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]`@users.noreply.github.com`" git checkout -B "${BRANCH}" git add .chai-bot/ci-status-jobs.yaml - git commit -m "chore(ci): update periodic job registry (release@${{ steps.generate.outputs.release_sha }})" + git commit -m "chore(ci): update periodic job registry (release@${RELEASE_SHA})" git push --force-with-lease origin "${BRANCH}" if gh pr view "${BRANCH}" --json state --jq .state 2>/dev/null | grep -q OPEN; then echo "PR already exists, pushed update." else gh pr create \ --title "NO-JIRA: update periodic job registry" \ - --body "Auto-generated by the nightly \`update-ci-job-registry\` workflow from [openshift/release@${{ steps.generate.outputs.release_sha }}](https://github.com/openshift/release/commit/${{ steps.generate.outputs.release_sha }}). + --body "Auto-generated by the nightly \`update-ci-job-registry\` workflow from [openshift/release@${RELEASE_SHA}](https://github.com/openshift/release/commit/${RELEASE_SHA}). Review the diff to see which jobs were added or removed." \ --label approved --label lgtm fiAlso applies to: 52-56
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/update-ci-job-registry.yaml at line 45, The git commit command in the update-ci-job-registry workflow interpolates steps.generate.outputs.release_sha directly inside the run script, which should be avoided. Move that value into an env variable for the affected commit/PR body steps and reference the shell variable instead, so the workflow uses the generated release SHA without embedding a GitHub Actions expression in the shell command. Apply the same pattern anywhere the release SHA is reused in this job.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/update-ci-job-registry.yaml:
- Around line 51-58: Remove the automatic approved/lgtm labeling from the gh pr
create step in the update-ci-job-registry workflow, since those labels should
only be applied by human review actions. Update the PR creation logic to create
the PR without those labels and rely on normal reviewer/approver flow instead;
locate the change in the gh pr create block within the workflow job that
generates the registry update PR.
- Line 16: The workflow is using a mutable tag for actions/checkout, which
should be pinned to a full commit SHA for CI security. Update the uses entry in
update-ci-job-registry.yaml from actions/checkout@v4 to a specific commit SHA,
and keep the reference anchored on the checkout step so it remains easy to find
if the workflow changes.
- Around line 40-46: The push in the update-ci-job-registry workflow is missing
a fetched remote-tracking ref for the automated/update-ci-job-registry branch,
so --force-with-lease may fail unexpectedly. Update the workflow step around git
checkout -B and git push so it first fetches the target branch from origin or
otherwise supplies an explicit expected SHA before pushing. Keep the fix
localized to the branch update sequence in the workflow job.
In `@hack/ci/update-job-registry.py`:
- Around line 77-98: The git subprocess calls in sparse_checkout currently have
no timeout, so a stalled clone or checkout can hang the nightly job
indefinitely. Update sparse_checkout to pass a timeout to each subprocess.run
invocation, including the git clone, git sparse-checkout set, git checkout, and
git rev-parse calls, using the existing sparse_checkout function as the place to
apply the change.
- Around line 267-277: main() currently writes the regenerated registry even
when many jobs remain uncategorized, so add a coverage threshold check before
render_yaml/output in the update-job-registry flow. Use the existing all_jobs,
categories, and total_categorized values to compute uncategorized coverage, and
fail the run with a non-zero exit if the uncategorized share is too high; keep
the write to args.output only after the check passes. Make the guard explicit
near build_categories/render_yaml so the update is blocked when categorization
drifts sharply, even with the approved/lgtm auto-selection.
---
Outside diff comments:
In @.chai-bot/hypershift_ci_daily_health_report.md:
- Around line 44-45: The ABORTED/ERROR infrastructure-health note in the daily
health report is only emitted inside the threaded analysis path, so it can be
skipped for otherwise healthy categories. Update the report logic that builds
the hypershift CI summary to surface the >30% ABORTED/ERROR warning outside the
Step 5 branch, using the existing pass-rate/infra classification flow so it is
shown whenever the threshold is exceeded. Keep the handling in the same
report-generation area that computes job result buckets and threaded analysis so
the warning is independent of whether the category drops below 80% pass rate.
---
Nitpick comments:
In @.github/workflows/update-ci-job-registry.yaml:
- Line 19: The workflow step in update-ci-job-registry.yaml installs PyYAML
without a version pin, which can make nightly runs non-reproducible. Update the
pip install command in the workflow to pin PyYAML to a specific compatible
version so the update-ci-job-registry job uses the same dependency each run.
Keep the change localized to the existing run step that installs pyyaml.
- Around line 12-15: The update-ci-job-registry workflow job currently has no
safeguards against overlapping executions or runaway runtime. Add a
workflow-level concurrency group for this workflow so manual and scheduled runs
cannot race on the same automated/update-ci-job-registry branch, and add a
timeout-minutes limit to the update-registry job to bound slow clones or stalled
steps. Use the existing update-registry job definition in the workflow to place
these settings.
- Line 45: The git commit command in the update-ci-job-registry workflow
interpolates steps.generate.outputs.release_sha directly inside the run script,
which should be avoided. Move that value into an env variable for the affected
commit/PR body steps and reference the shell variable instead, so the workflow
uses the generated release SHA without embedding a GitHub Actions expression in
the shell command. Apply the same pattern anywhere the release SHA is reused in
this job.
In `@hack/ci/update-job-registry.py`:
- Around line 130-196: Add unit tests for the categorization flow in
categorize_job and build_categories, covering each PLATFORM_RULES matcher, the
VERSION_SPLIT_PLATFORMS branch, and the description formatting paths. Create
tests that verify uncategorized jobs are skipped, version filtering via
selected_versions works, split platforms emit one Category per version, and
non-split platforms aggregate versions into a single category with the expected
description.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: dd693a95-7738-41a3-aa0d-e3e5656cadf1
📒 Files selected for processing (4)
.chai-bot/ci-status-jobs.yaml.chai-bot/hypershift_ci_daily_health_report.md.github/workflows/update-ci-job-registry.yamlhack/ci/update-job-registry.py
✅ Files skipped from review due to trivial changes (1)
- .chai-bot/ci-status-jobs.yaml
| BRANCH="automated/update-ci-job-registry" | ||
| git config user.name "github-actions[bot]" | ||
| git config user.email "41898282+github-actions[bot]@users.noreply.github.com" | ||
| git checkout -B "${BRANCH}" | ||
| git add .chai-bot/ci-status-jobs.yaml | ||
| git commit -m "chore(ci): update periodic job registry (release@${{ steps.generate.outputs.release_sha }})" | ||
| git push --force-with-lease origin "${BRANCH}" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
git push --force-with-lease behavior when local branch has no remote-tracking ref
💡 Result:
When you use git push --force-with-lease without explicitly specifying an expected value, Git relies on your local remote-tracking branch (e.g., refs/remotes/origin/branch-name) to determine the expected state of the remote branch [1][2]. If your local branch does not have a corresponding remote-tracking reference, git push --force-with-lease will fail [3]. This is because Git cannot verify the remote's current state against the expected value, and it defaults to rejecting the push rather than assuming it is safe to overwrite [1][3]. To successfully use --force-with-lease in this situation, you have two primary options: 1. Create or update the remote-tracking branch: You can fetch from the remote (e.g., git fetch origin), which will create the necessary tracking reference, allowing subsequent --force-with-lease commands to function normally [4]. 2. Explicitly specify the expected value: You can use the form git push --force-with-lease=:, where is the SHA-1 commit hash you expect the remote branch to currently point to [1][2]. If you intend to push a brand-new branch and want to ensure it does not already exist on the remote, you can use the empty string as the expectation [1][2][5]: git push --force-with-lease=branch-name: origin branch-name This tells Git that you expect the remote branch to be in a non-existent state (represented by the empty string or null SHA-1) [1][6]. If the branch already exists on the remote, the push will be rejected, protecting against accidental overwrites [1].
Citations:
- 1: https://cdn.kernel.org/pub/software/scm/git/docs/git-push.html
- 2: https://man7.org/linux/man-pages/man1/git-push.1.html
- 3: https://public-inbox.org/git/1374561800-938-4-git-send-email-gitster@pobox.com/
- 4: git/git@f17d642d3b
- 5: https://code.googlesource.com/git/+/HEAD/Documentation/git-push.adoc
- 6: http://public-inbox.org/git/20160726111911.cehaitav4enas7dm@john.keeping.me.uk/T/
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== update-ci-job-registry workflow ==="
nl -ba .github/workflows/update-ci-job-registry.yaml | sed -n '1,220p'
echo
echo "=== search for fetch / checkout refs in this workflow ==="
rg -n "fetch|checkout|force-with-lease|remote-tracking|origin/automated/update-ci-job-registry|git push" .github/workflows/update-ci-job-registry.yamlRepository: openshift/hypershift
Length of output: 238
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== update-ci-job-registry workflow ==="
cat -n .github/workflows/update-ci-job-registry.yaml | sed -n '1,220p'
echo
echo "=== search for fetch / checkout refs in this workflow ==="
rg -n "fetch|checkout|force-with-lease|remote-tracking|origin/automated/update-ci-job-registry|git push" .github/workflows/update-ci-job-registry.yamlRepository: openshift/hypershift
Length of output: 2923
🌐 Web query:
actions/checkout default fetch remote-tracking refs force-with-lease no remote-tracking branch behavior
💡 Result:
The actions/checkout action has specific default behaviors for fetching data that differ significantly from a standard local git clone [1][2]. Default Fetch Behavior: By default, actions/checkout performs a shallow fetch (fetch-depth: 1) [1][3]. It fetches only the single commit (the specific ref or SHA) that triggered the workflow, rather than fetching all history or all remote-tracking branches [1][2]. Because it is a shallow fetch, it does not populate the remote-tracking branches for other branches in the repository [1]. You can modify this behavior by setting fetch-depth: 0, which fetches all history and all branches and tags [1][3]. Interaction with force-with-lease: The git push --force-with-lease command is designed to prevent overwriting remote work by ensuring that the remote-tracking branch (e.g., origin/main) matches the value you expected it to have before the force push [4][5]. Because actions/checkout by default only fetches the single commit for the triggered event, it does not update your local remote-tracking branches to reflect the current state of the remote repository [1][2]. If you run a git push --force-with-lease in a workflow that used the default fetch-depth: 1: 1. Your local remote-tracking branch (e.g., origin/main) will be stale or non-existent [2]. 2. If the local remote-tracking branch is stale, --force-with-lease may be misled [6]. It checks against the state you have locally; if your local state is outdated, the command may perceive that you are up-to-date with the remote, even if someone else has pushed new commits in the interim [6]. To use --force-with-lease safely within a GitHub Action, you must ensure your local environment is aware of the actual remote state. This typically requires a deeper fetch (e.g., fetch-depth: 0 or at least fetching the specific branch you intend to update) so that your local remote-tracking branches are updated to the current state of the repository [1][3][6].
Citations:
- 1: https://github.com/actions/checkout/blob/main/README.md
- 2: https://github.com/actions/checkout/blob/master/adrs/0153-checkout-v2.md
- 3: https://github.com/actions/checkout/?tab=readme-ov-file
- 4: https://git-scm.com/docs/git-push
- 5: https://stackoverflow.com/questions/52823692/git-push-force-with-lease-vs-force
- 6: https://israynotarray.com/en/git/2026/02/26/git-push-force-with-lease-better-than-force/
🌐 Web query:
git push --force-with-lease no remote-tracking branch fails default expected value documentation
💡 Result:
When you run git push --force-with-lease without any arguments, Git uses the local remote-tracking branch (e.g., refs/remotes/origin/main) as the "expected value" for the remote ref [1][2][3]. If no such remote-tracking branch exists, or if it has not been updated to reflect the remote's current state, the push will fail because Git cannot confirm the remote ref's current value matches an expected baseline [3][4]. To successfully use --force-with-lease on a branch that lacks a corresponding remote-tracking branch, you must manually specify the expected value using the --force-with-lease=refname:expect syntax [2][3][5]. Key mechanisms to resolve this: 1. Manual Expectation: You can provide an explicit commit hash or ref as the expected value [2][3]. For example, --force-with-lease=main:some-commit-hash will only allow the push if the remote main branch currently points to some-commit-hash [2][5]. 2. Asserting Non-Existence: If you expect the branch to not exist on the remote yet, you can use an empty string as the expected value [2][5]. For example, --force-with-lease=feature-branch: will succeed only if the remote branch does not exist [2][3]. 3. Remote-Tracking Branches: Generally, Git's safety mechanism relies on these tracking branches as a heuristic [6]. If you do not have a tracking branch, you can create or update one by running git fetch [2][6]. In summary, the failure occurs because the "default" behavior of --force-with-lease is implicitly tied to existing remote-tracking refs [1][3]. When those are missing, Git lacks the necessary reference to perform the safety check, resulting in a rejected push [4].
Citations:
- 1: https://git-scm.com/docs/git-push
- 2: https://cdn.kernel.org/pub/software/scm/git/docs/git-push.html
- 3: https://man7.org/linux/man-pages/man1/git-push.1.html
- 4: github/gh-stack@44aac93
- 5: https://code.googlesource.com/git/+/refs/tags/v2.43.2/Documentation/git-push.txt
- 6: git/git@f17d642d3b
Fetch the branch before --force-with-lease. actions/checkout only fetches the triggering ref, and git checkout -B doesn’t create an origin/automated/update-ci-job-registry tracking ref here. That means this push can be rejected instead of safely leasing against the branch tip, so the update job can fail once the branch already exists. Fetch that branch first or pass an explicit expected SHA.
🧰 Tools
🪛 zizmor (1.26.1)
[info] 45-45: code injection via template expansion (template-injection): may expand into attacker-controllable code
(template-injection)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/update-ci-job-registry.yaml around lines 40 - 46, The push
in the update-ci-job-registry workflow is missing a fetched remote-tracking ref
for the automated/update-ci-job-registry branch, so --force-with-lease may fail
unexpectedly. Update the workflow step around git checkout -B and git push so it
first fetches the target branch from origin or otherwise supplies an explicit
expected SHA before pushing. Keep the fix localized to the branch update
sequence in the workflow job.
| **Per-category pass rate**: Count successful builds across all jobs in the category out of total testable builds (SUCCESS + FAILURE only; exclude ABORTED/ERROR). | ||
|
|
||
| **Health indicators**: | ||
| - 🟢 Pass rate ≥ 80% |
There was a problem hiding this comment.
I believe the pass rate for CR is much higher, like 90 or 95%. Did you consider using their health rates since we will need to start onboarding more HyperShift tests to CR?
There was a problem hiding this comment.
Good point — we'll start with 80% to get the daily summary running and iterate from there. Once we have enough data and the report is stable, we can raise the threshold to align with CR's bar.
There was a problem hiding this comment.
Follow-up on my own comment: I dug into the sippy source to understand how CR actually determines regressions, and it's not a fixed pass rate threshold like I was thinking.
CR uses Fisher's Exact Test comparing the current dev release against the last stable GA basis. A test goes red when its pass rate drops significantly relative to the basis, not when it crosses a fixed floor. The key defaults are:
- 95% confidence required for statistical significance
- 5% pity factor — pass rate must drop more than 5pp before Fisher is even evaluated
- 3 minimum failures in the sample
- >15pp drop = extreme regression, otherwise significant
So there isn't a "90 or 95% pass rate" to adopt from CR — a test at 82% can be green in CR if the basis was also ~82%.
These thresholds for aggregate periodic job health are a different signal than CR's per-test regression detection, so the 80%/40% split here seems reasonable for a daily monitoring bot. That said, the 40% red threshold feels very low — might be worth raising that.
I opened openshift/ci-docs#606 to document these CR thresholds on the ci-docs site since none of this was written down anywhere.
There was a problem hiding this comment.
Great deep-dive, thanks for documenting the CR methodology. Agree that 40% for red is generous — happy to raise it once we see how the first reports look.
|
The image is now returning HTTP 200, confirming this was a transient quay.io outage. I have all the evidence needed. Test Failure Analysis CompleteJob Information
Test Failure AnalysisErrorSummaryThe Root CauseThe root cause is a transient quay.io container registry outage that returned HTTP 503 (Service Unavailable) when the Konflux pipeline's Specifically:
This is not a code issue — PR #8918 only touches CI configuration files ( Recommendations
Evidence
|
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
.github/workflows/update-ci-job-registry.yaml (1)
20-20: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider pinning the
pyyamlversion.Unpinned dependency install in CI can introduce non-reproducible builds if a new pyyaml release changes behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/update-ci-job-registry.yaml at line 20, The CI workflow currently installs pyyaml without a fixed version, which can make the update-ci-job-registry job non-reproducible. Update the pip install step in the workflow to pin pyyaml to a specific version, keeping the change localized to the existing install command in the update-ci-job-registry workflow.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/update-ci-job-registry.yaml:
- Around line 38-56: The update-ci-job-registry workflow is interpolating
steps.generate.outputs.release_sha directly inside the shell script, which
creates an injection risk in the run block. Move that value into an env variable
for the relevant step and reference the env var in git checkout, commit message,
PR body, and branch name construction so update-ci-job-registry.sh logic uses
data rather than GitHub Actions text substitution. Keep using the existing
generate step and gh pr create flow, but avoid any direct ${{ ... }} expansions
inside the script body.
---
Nitpick comments:
In @.github/workflows/update-ci-job-registry.yaml:
- Line 20: The CI workflow currently installs pyyaml without a fixed version,
which can make the update-ci-job-registry job non-reproducible. Update the pip
install step in the workflow to pin pyyaml to a specific version, keeping the
change localized to the existing install command in the update-ci-job-registry
workflow.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: ba3bfc1a-57d6-4580-ad16-f9566f42550a
📒 Files selected for processing (2)
.github/workflows/update-ci-job-registry.yamlhack/ci/update-job-registry.py
🚧 Files skipped from review as they are similar to previous changes (1)
- hack/ci/update-job-registry.py
CNTRLPLANE-3766 Add job registry and report instructions under .chai-bot/ for a daily CI health report posted by chai-bot to #forum-ocp-hypershift. The job registry (.chai-bot/ci-status-jobs.yaml) tracks periodic Prow jobs across platform categories (AWS, Azure & KubeVirt, OpenStack, IBM/PowerVS, MCE Agent, HCM & Other). It is auto-generated nightly by hack/ci/update-job-registry.py via a GitHub Action that does a sparse checkout of openshift/release and categorizes jobs by platform keywords. The report instructions (.chai-bot/hypershift_ci_daily_health_report.md) tell chai-bot to compute per-category pass rates from the last 20 completed builds, compare last-10-vs-prior-10 for trend indicators, and post a worst-first scoreboard. Categories below 80% get threaded failure analysis. Categories below 50% trigger an hcp-itn incident proposal with a /meet command for synchronous triage via shadowbot. ABORTED and ERROR builds are excluded from pass rate computation to avoid conflating infrastructure issues with product failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
celebdor
force-pushed
the
chai-bot-ci-daily-health-report
branch
from
60e5028 to
46d3fc0
Compare
|
/override e2e-aks |
|
@celebdor: /override requires failed status contexts, check run or a prowjob name to operate on.
Only the following failed contexts/checkruns were expected:
If you are trying to override a checkrun that has a space in it, you must put a double quote on the context. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/override ci/prow/e2e-aks |
|
@celebdor: Overrode contexts on behalf of celebdor: ci/prow/e2e-aks, ci/prow/e2e-aws, ci/prow/e2e-aws-upgrade-hypershift-operator, ci/prow/e2e-azure-v2-self-managed, ci/prow/e2e-kubevirt-aws-ovn-reduced, ci/prow/e2e-v2-aws, ci/prow/e2e-v2-gke DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Add .chai-bot to the CEL path-exclusion regex so that PRs touching only bot instructions and job registry files do not trigger container image builds. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
celebdor
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@celebdor: This pull request references CNTRLPLANE-3766 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Write YAML to stdout by default so the script works in pipelines and with shell redirects. Progress messages go to stderr. Use --output to write directly to a file. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
/override ci/prow/e2e-aks |
|
/verified by celebdor |
openshift-ci-robot
added
the
verified
|
@celebdor: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@celebdor: Overrode contexts on behalf of celebdor: ci/prow/e2e-aks, ci/prow/e2e-aws, ci/prow/e2e-aws-upgrade-hypershift-operator, ci/prow/e2e-azure-v2-self-managed, ci/prow/e2e-kubevirt-aws-ovn-reduced, ci/prow/e2e-v2-aws, ci/prow/e2e-v2-gke DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bryan-cox, celebdor The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
@celebdor: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
6 participants
Summary
.chai-bot/ci-status-jobs.yaml— job registry with 71 periodic Prow jobs across 8 platform categories for OCP 5.0 and 4.23.chai-bot/hypershift_ci_daily_health_report.md— report instructions that chai-bot fetches at runtime to produce a daily CI health summary posted to #forum-ocp-hypershift (weekdays 07:00 UTC)Follows the pattern established by ROSA's CI daily health report in
openshift-online/rosa-e2e.A companion PR to
openshift-eng/ship-help-botwill add the scheduled task configuration pointing to these instructions.Jira: https://redhat.atlassian.net/browse/CNTRLPLANE-3766
Epic: https://redhat.atlassian.net/browse/CNTRLPLANE-3765
Test plan
🤖 Generated with Claude Code
Summary by CodeRabbit