ci: shard e2e into 4 legs + repair release publish#193
Merged
Conversation
📝 WalkthroughWalkthroughThis PR adds a new design document, ChangesE2E CI Sharding Plan
Estimated code review effort: 1 (Trivial) | ~5 minutes Related Issues: None specified Related PRs: None specified Suggested labels: documentation, ci Suggested reviewers: None specified Poem:
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
With the concurrent-runner limit raised to 20, revisit the matrix-sharding
idea that e2e-speedup-plan.md deferred ("Considered and skipped", item 2)
solely because of the old ~3-runner throttle.
Grounds the plan in measured step-level timings from a green PR run: both
e2e lanes are execution-dominated (full 636s exec / 4.6m overhead;
quickstart 823s exec / 4.2m overhead), and quickstart (~18m) — not full
(~15m) — is the real critical path. Proposes reshaping the 2-lane matrix
into ~6 balanced legs (label-filter split of the functional suite + break
up the sequential quickstart chain), each on its own runner + k3d cluster,
projecting the e2e gate from ~18m to ~8-10m.
Covers the details that must be handled: per-shard coverage merge via
Codecov flags, an e2e-complete aggregator so branch protection is matrix-
agnostic, keeping fail-fast:false, and why per-cluster isolation makes
sharding strictly safer than raising --procs on the singleton controller.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 0.29.1 release — the first to exercise the new sign/attest + retag pipeline — failed in both final publish jobs: - publish-manifest could not attach sbom.spdx.json or append notes: release-please publishes the GitHub release and GitHub now enforces immutable releases, which reject post-publish asset/body edits. Create the release as a draft, attach every asset while it is still a draft, and add a publish-release job that flips it to published only after both publish jobs succeed (a failure now leaves an unpublished draft, not a half-populated immutable release). - publish-helm's cosign chart signature failed with UNAUTHORIZED: `helm registry login` writes helm's own registry config, but cosign reads ~/.docker/config.json. Add an explicit `cosign login` before signing. Also refresh the README badges: - add the OpenSSF Best Practices badge (closes #189) - drop the retired Go Report Card badge (closes #183) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sunib
force-pushed
the
e2e-runner-sharding
branch
from
7fc34fc to
d0d257b
Compare
With up to 20 concurrent runners now available, split both e2e lanes to cut the wall-clock gate (was max(full ~15m, quickstart ~18m)). Per docs/design/e2e-ci-runner-sharding-plan.md this is Phases 1+2 at a 2-way split: - full -> full-manager (`manager`) + full-core (`!manager && !image-refresh`). The two filters are exact complements of the old `!image-refresh` full filter; a ginkgo --dry-run confirms 37 + 18 = 55 specs, so no spec is added, dropped, or double-run. Both shards run config-dir installs and collect e2e coverage; Codecov unions same-flag uploads, so merged coverage is unchanged. - quickstart -> quickstart-install (helm + manifest install validations on one cluster) + image-refresh (the local rebuild/reload/rollout chain on its own runner — the plan's likely-slowest slice). test-e2e gains E2E_LABEL_FILTER / E2E_REPORT_NAME (defaults preserve a plain `task test-e2e`); CI forwards them by env-name so the filter's `&&` never reaches the shell. e2e is not a required status check, so no branch-protection change is needed. Leg membership is a provisional first cut — rebalance from the e2e-ginkgo-reports-* per-spec timings once a green run produces them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sunib
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three things land on this branch, plus the design doc that motivated them.
Repair the release publish pipeline
The first real release (
0.29.1) failed in both final publish jobs:publish-releasejob that flips it to published only after both publish jobs succeed.cosignfailedUNAUTHORIZEDbecausehelm registry logindoesn't populate~/.docker/config.json. Fix: add acosign loginbefore signing.Shard the e2e suite across 4 runners
Cuts the wall-clock e2e gate now that up to 20 concurrent runners are available:
full→full-manager(manager) +full-core(!manager && !image-refresh) — exact-complement label filters (--dry-run: 37 + 18 = 55 specs = the old lane, nothing added/dropped). Both shards still collect coverage; Codecov unions thee2euploads, so merged coverage is unchanged.quickstart→quickstart-install(helm + manifest installs) +image-refresh(local rebuild chain, own runner).README badges
Add the OpenSSF Best Practices badge, drop the retired Go Report Card badge.
Also adds the design doc
docs/design/e2e-ci-runner-sharding-plan.md.Closes #189
Closes #183