ci(gitlab): streamline eic manifest DAG dependencies - #381
Conversation
Refactor GitLab CI DAG wiring so eic-manifest runs as a matrix aligned with eic and starts per-container as soon as its matching build shard completes. Restrict each manifest shard to its own metadata files and keep benchmark and deploy jobs dependent only on the specific manifest shard they require. Use YAML anchors to remove duplicated matrix and needs blocks so eic and eic-manifest stay synchronized and eic_ci/eic_xl manifest selectors are reused consistently. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Not ready to approve
The new matrix needs selector appears to be mis-specified (array-vs-scalar mismatch) and eic-manifest doesn’t currently mirror .nightly rules, both of which can break artifact matching / DAG satisfaction and trigger the new “No metadata files found” failure.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
This PR refactors the GitLab CI DAG around eic-manifest so manifest jobs can start as soon as their corresponding eic build shard finishes, reducing unnecessary waiting and duplication in the pipeline configuration.
Changes:
- Converted
eic-manifestinto aparallel:matrixjob aligned with the existingeicmatrix via a shared YAML anchor. - Added matrix-aware
needswiring so eacheic-manifestshard depends on its matchingeicshard and processes only its ownbuild-metadata-${BUILD_IMAGE}${ENV}-*.jsonfiles. - Updated downstream jobs to depend only on the specific manifest shard they require via anchored
needsentries.
File summaries
| File | Description |
|---|---|
.gitlab-ci.yml |
Reworks eic-manifest into a matrix job and rewires needs dependencies to be shard-specific using anchors/aliases. |
Review details
Suppressed comments (1)
.gitlab-ci.yml:488
- The
needs:parallel:matrixselector for the upstreameicjob uses single-element arrays (e.g.BUILD_IMAGE: ['…']). The upstreameicmatrix defines these variables as scalars, so this type mismatch can prevent GitLab from matching the correcteicshard and downloading its artifacts, which will make this shard hit the new "No metadata files found" error.
- BUILD_IMAGE: ['$[[ matrix.BUILD_IMAGE ]]']
ENV: ['$[[ matrix.ENV ]]']
BUILD_TYPE: ['$[[ matrix.BUILD_TYPE ]]']
BUILDER_IMAGE: ['$[[ matrix.BUILDER_IMAGE ]]']
RUNTIME_IMAGE: ['$[[ matrix.RUNTIME_IMAGE ]]']
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
| eic-manifest: | ||
| parallel: | ||
| matrix: *eic_matrix | ||
| extends: .docker | ||
| stage: eic-manifest |
This streamlines the GitLab pipeline DAG so jobs start as soon as their required artifacts are ready, instead of waiting on unrelated shards.
What changed
eic-manifestinto a matrix job aligned with theeicmatrix.needsfrom eacheic-manifestshard to its matchingeicshard.build-metadata-${BUILD_IMAGE}${ENV}-*.jsonso each shard handles only its own image metadata.needsso benchmark and deploy jobs depend only on the specific manifest shard they require.Duplication reduction and synchronization
eicmatrix and reused it ineic-manifest.needsselectors foreic_ciandeic_xlmanifest dependencies.Notes