chore(ci): Improve cluster creation and teardown for GHA E2E#21604
Open
msugakov wants to merge 6 commits into
Open
chore(ci): Improve cluster creation and teardown for GHA E2E#21604msugakov wants to merge 6 commits into
msugakov wants to merge 6 commits into
Conversation
so that we have something remaining in infra to investigate in case of failures. Also, don't swallow the error of the cluster deletion. Otherwise, how do we know that we can't delete clusters any more.
I could not easily fix the `actions/create-gke-cluster` action because it requires deeper thinking into pre/post hook. At the same time, I saw that all other tests don't use the action and create the cluster directly. After seeing that #21230 was created to enable use of spot VMs, it's clearly possible to use those through `provision_gke_cluster()` function with `GKE_SPOT` env variable. Thus, I consider the `actions/create-gke-cluster` action to be redundant and I got rid of it. By the way, this also should fix the cluster name uniqueness problem in `e2e-db-backup-restore-test`.
msugakov
added
e2e-gke-upgrade-tests
|
Skipping CI for Draft Pull Request. |
In the spirit of #21230
msugakov
force-pushed
the
misha/iterate-on-gha-e2e
branch
from
bbb373c to
cc63fd9
Compare
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (4)
📝 WalkthroughSummary by CodeRabbit
WalkthroughCI workflows for GKE-based end-to-end tests now gate teardown or deletion on successful cluster creation, add explicit step ids, set GKE_SPOT values, and inline GKE provisioning in the DB backup/restore job. ChangesGKE cluster provisioning and conditional teardown
Estimated code review effort: 2 (Simple) | ~12 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #21604 +/- ##
==========================================
- Coverage 50.37% 50.32% -0.06%
==========================================
Files 2844 2844
Lines 218476 218476
==========================================
- Hits 110068 109956 -112
- Misses 100426 100515 +89
- Partials 7982 8005 +23
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
🚀 Build Images ReadyImages are ready for commit 14097d7. To use with deploy scripts: export MAIN_IMAGE_TAG=4.12.x-414-g14097d7df4 |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/e2e-gke-upgrade-tests.yaml (1)
81-86: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winCluster deletion is gated on a step that also downloads artifacts, not just cluster wait.
The
wait-for-clusterstep (central: Lines 80-86, postgres: Lines 225-231, sensor: Lines 369-375) runs bothinfractl waitandinfractl artifacts ... --download-dir artifacts. Deletion (Lines 149, 293, 424) is gated on that same step's overall outcome. Ifinfractl waitsucceeds but the subsequentinfractl artifactsdownload fails for an unrelated reason (network blip, storage issue, etc.), the step outcome becomesfailure, and the cluster — which was successfully created and is healthy — will never be deleted, leaking infra/billing resources.Consider splitting the wait and artifact-download into separate steps, or capturing the wait's own exit code so deletion is gated purely on cluster availability rather than the combined step outcome.
♻️ Example split (applies similarly to postgres/sensor jobs)
- - name: Wait for the cluster and grab artifacts - id: wait-for-cluster - timeout-minutes: 25 - run: | - infractl wait "${CLUSTER_NAME}" - infractl artifacts "${CLUSTER_NAME}" --download-dir artifacts > /dev/null - echo "KUBECONFIG=$(pwd)/artifacts/kubeconfig" >> "$GITHUB_ENV" + - name: Wait for the cluster + id: wait-for-cluster + timeout-minutes: 25 + run: infractl wait "${CLUSTER_NAME}" + + - name: Grab artifacts + run: | + infractl artifacts "${CLUSTER_NAME}" --download-dir artifacts > /dev/null + echo "KUBECONFIG=$(pwd)/artifacts/kubeconfig" >> "$GITHUB_ENV"Also applies to: 149-149, 226-231, 293-293, 370-375, 424-424
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/e2e-gke-upgrade-tests.yaml around lines 81 - 86, The `wait-for-cluster` step is combining cluster readiness and artifact download, so later deletion gates on the whole step instead of just `infractl wait`. Split the `infractl wait` and `infractl artifacts ... --download-dir artifacts` work into separate steps in the workflow jobs, or otherwise preserve the wait result independently; then update the deletion guards to reference the cluster-wait step outcome for the `wait-for-cluster`/artifact sequence in each job (including the postgres and sensor variants).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.github/workflows/e2e-gke-upgrade-tests.yaml:
- Around line 81-86: The `wait-for-cluster` step is combining cluster readiness
and artifact download, so later deletion gates on the whole step instead of just
`infractl wait`. Split the `infractl wait` and `infractl artifacts ...
--download-dir artifacts` work into separate steps in the workflow jobs, or
otherwise preserve the wait result independently; then update the deletion
guards to reference the cluster-wait step outcome for the
`wait-for-cluster`/artifact sequence in each job (including the postgres and
sensor variants).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Central YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 911d8af7-51e7-4dd7-a662-e18fe8dd5edd
📒 Files selected for processing (5)
.github/actions/create-gke-cluster/action.yaml.github/workflows/e2e-byodb-test.yaml.github/workflows/e2e-db-backup-restore-test.yaml.github/workflows/e2e-gke-upgrade-tests.yaml.github/workflows/e2e-nongroovy-tests.yaml
💤 Files with no reviewable changes (1)
- .github/actions/create-gke-cluster/action.yaml
msugakov
changed the title
msugakov
changed the title
AlexVulaj
reviewed
Jul 8, 2026
in upgrade tests
AlexVulaj
approved these changes
Jul 8, 2026
there's no need to source the script to call just one function in it. The script allows passing the function and its arguments for such a use.
`continue-on-error: true` should show orange warning icon which is better than `|| true` or `|| echo` which results in the green one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
I suggest that this PR should be reviewed by commits. I provided descriptions in each commit's message.
User-facing documentation
Testing and quality
Automated testing
No change.
How I validated my change