Utilize clonerefs sparse checkout#5161
Open
Prucek wants to merge 4 commits intoopenshift:mainfrom
Open
Conversation
Contributor
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
📝 WalkthroughWalkthroughThis PR refactors cloning behavior from a skip-cloning approach to a repository-aware sparse-checkout strategy. Production code now determines which files to sparse-checkout based on repository and image configuration, with updated fixture files reflecting the new YAML output format. ChangesSparse Checkout Migration
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (2 warnings, 1 inconclusive)
✅ Passed checks (11 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Prucek The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
Prucek
changed the title
Prucek
changed the title
Prucek
changed the title
Member
Author
|
/label tide/merge-method-squash |
openshift-ci
Bot
added
the
tide/merge-method-squash
Replace skipCloning() with sparseCheckoutFiles() that computes the minimal set of files needed for image builds: .ci-operator.yaml (if from_repository is set) plus all Dockerfile paths from image configs. When sparse checkout files are available, set them on DecorationConfig instead of skipping cloning entirely. This allows prow to checkout only the files needed for image builds, significantly reducing clone time for large repositories. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a rehearsal job's primary ref (openshift/release) differs from the target repo, CompletePrimaryRefs propagates SparseCheckoutFiles from DecorationConfig to the extra ref (the original repo). However, the DecorationConfig.SparseCheckoutFiles remains set and prow applies it to the primary ref too, causing the release repo to be sparse-checked out — which breaks the clone with "unrelated histories" errors. Clear SparseCheckoutFiles from DecorationConfig after setting up the extra ref, since the files are already on the extra ref via CompletePrimaryRefs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When sparse checkout is active, clonerefs inside the src image build only checks out Dockerfiles, leaving the repo without source code. This breaks binary_build_commands (e.g. make all) and test steps that need the full source tree. Clear SparseCheckoutFiles from all refs before passing them to clonerefs in createBuild(), ensuring the src image always gets a full clone regardless of job-level sparse checkout settings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Periodic jobs use extra_refs[0] as their primary ref instead of a top-level Refs field. DecorationConfig.SparseCheckoutFiles was not being propagated to this ref, so periodic jobs never used sparse checkout even when configured. Copy SparseCheckoutFiles from DecorationConfig to the periodic's extra_refs[0] before appending it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prucek
force-pushed
the
sparse-checkout-fixes
branch
from
4bc1753 to
58a15fb
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
pkg/prowgen/jobbase.go (1)
95-104: Verify theopenshift/releaseconsumer before merging.This changes prowgen output from
skip_cloningtosparse_checkout_files. The linkedopenshift/releasefindings still show validation/generation code paths keyed onskip_cloning, so please runmake jobsand the release-side validators there to make sure this schema change does not break the downstream repo.As per coding guidelines, "
pkg/prowgen/**: Changes here require runningmake jobsin the openshift/release repo to verify no unexpected diffs appear".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/prowgen/jobbase.go` around lines 95 - 104, This change replaces setting DecorationConfig.SkipCloning with DecorationConfig.SparseCheckoutFiles; before merging, run the downstream verification: pull the openshift/release repo, run make jobs and the release-side validators to ensure no unexpected diffs or validation errors; if failures occur, either preserve backward compatibility by still setting b.base.UtilityConfig.DecorationConfig.SkipCloning when sparseCheckoutFiles is empty (in the same block that calls sparseCheckoutFiles) or update the release-side generation/validation to accept SparseCheckoutFiles instead of SkipCloning (check the places that consume SkipCloning in release validators and job generation).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@cmd/ci-operator-prowgen/testdata/zz_fixture_presubmit_TestFromCIOperatorConfigToProwYaml_Input_is_YAML_and_it_is_correctly_processed.yaml`:
- Around line 66-67: The fixture sets image-style sparse checkout for a
non-image unit presubmit: remove or change the sparse_checkout_files entry that
lists "Dockerfile" for the job with "--target=unit" so the unit presubmit is not
limited to image files; specifically locate the YAML block that contains
sparse_checkout_files and the job metadata referencing "--target=unit" and
either delete the sparse_checkout_files key or replace it with a general
checkout (or an appropriate non-image file list) so the fixture matches the
"image-build jobs only" intent.
---
Nitpick comments:
In `@pkg/prowgen/jobbase.go`:
- Around line 95-104: This change replaces setting DecorationConfig.SkipCloning
with DecorationConfig.SparseCheckoutFiles; before merging, run the downstream
verification: pull the openshift/release repo, run make jobs and the
release-side validators to ensure no unexpected diffs or validation errors; if
failures occur, either preserve backward compatibility by still setting
b.base.UtilityConfig.DecorationConfig.SkipCloning when sparseCheckoutFiles is
empty (in the same block that calls sparseCheckoutFiles) or update the
release-side generation/validation to accept SparseCheckoutFiles instead of
SkipCloning (check the places that consume SkipCloning in release validators and
job generation).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 78c043d6-fdce-416d-a3dc-28ccfbb28c15
📒 Files selected for processing (17)
cmd/ci-operator-prowgen/testdata/zz_fixture_postsubmit_TestFromCIOperatorConfigToProwYaml_Custom_test_timeout.yamlcmd/ci-operator-prowgen/testdata/zz_fixture_postsubmit_TestFromCIOperatorConfigToProwYaml_Input_is_YAML_and_it_is_correctly_processed.yamlcmd/ci-operator-prowgen/testdata/zz_fixture_postsubmit_TestFromCIOperatorConfigToProwYaml_Using_a_variant_config__one_test_and_images__one_existing_job._Expect_one_presubmit__pre_post_submit_images_jobs._Existing_job_should_not_be_changed..yamlcmd/ci-operator-prowgen/testdata/zz_fixture_postsubmit_TestFromCIOperatorConfigToProwYaml_one_test_and_images__no_previous_jobs._Expect_test_presubmit__pre_post_submit_images_jobs.yamlcmd/ci-operator-prowgen/testdata/zz_fixture_presubmit_TestFromCIOperatorConfigToProwYaml_Custom_test_timeout.yamlcmd/ci-operator-prowgen/testdata/zz_fixture_presubmit_TestFromCIOperatorConfigToProwYaml_Input_is_YAML_and_it_is_correctly_processed.yamlcmd/ci-operator-prowgen/testdata/zz_fixture_presubmit_TestFromCIOperatorConfigToProwYaml_Using_a_variant_config__one_test_and_images__one_existing_job._Expect_one_presubmit__pre_post_submit_images_jobs._Existing_job_should_not_be_changed..yamlcmd/ci-operator-prowgen/testdata/zz_fixture_presubmit_TestFromCIOperatorConfigToProwYaml_one_test_and_images__no_previous_jobs._Expect_test_presubmit__pre_post_submit_images_jobs.yamlpkg/prowgen/jobbase.gopkg/prowgen/prowgen.gopkg/prowgen/testdata/zz_fixture_TestGenerateJobs_Promotion_configuration_causes_promote_job_with_unique_targets.yamlpkg/prowgen/testdata/zz_fixture_TestGenerateJobs_promotion_postsubmit_and_periodic_.yamlpkg/prowgen/testdata/zz_fixture_TestNewProwJobBaseBuilderForTest_simple_container_based_test_with_timeout_and_no_decoration.yamlpkg/prowgen/testdata/zz_fixture_TestProwJobBaseBuilder_job_with_image_builds_in_of_openshift_release_main__does_not_have_no_builds__label.yamlpkg/prowgen/testdata/zz_fixture_TestProwJobBaseBuilder_private_job_with_cloning__including_podspec.yamlpkg/rehearse/jobs.gopkg/steps/source.go
🚧 Files skipped from review as they are similar to previous changes (3)
- pkg/prowgen/prowgen.go
- pkg/prowgen/testdata/zz_fixture_TestGenerateJobs_Promotion_configuration_causes_promote_job_with_unique_targets.yaml
- cmd/ci-operator-prowgen/testdata/zz_fixture_presubmit_TestFromCIOperatorConfigToProwYaml_Custom_test_timeout.yaml
Comment on lines
+66
to
+67
| sparse_checkout_files: | ||
| - Dockerfile |
There was a problem hiding this comment.
Non-image unit presubmit should not be sparse-checked out to only Dockerfile.
At Line 66-Line 67, this fixture sets image-style sparse checkout on a --target=unit job, which conflicts with the PR objective (“image-build jobs only”) and can encode incorrect expected output.
Proposed fixture correction
decoration_config:
- sparse_checkout_files:
- - Dockerfile
+ skip_cloning: true📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| sparse_checkout_files: | |
| - Dockerfile | |
| skip_cloning: true |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@cmd/ci-operator-prowgen/testdata/zz_fixture_presubmit_TestFromCIOperatorConfigToProwYaml_Input_is_YAML_and_it_is_correctly_processed.yaml`
around lines 66 - 67, The fixture sets image-style sparse checkout for a
non-image unit presubmit: remove or change the sparse_checkout_files entry that
lists "Dockerfile" for the job with "--target=unit" so the unit presubmit is not
limited to image files; specifically locate the YAML block that contains
sparse_checkout_files and the job metadata referencing "--target=unit" and
either delete the sparse_checkout_files key or replace it with a general
checkout (or an appropriate non-image file list) so the fixture matches the
"image-build jobs only" intent.
Member
Author
|
/test e2e |
openshift-ci
Bot
added
the
do-not-merge/hold
Contributor
|
@Prucek: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.ci-operator.yaml+ Dockerfile pathsDecorationConfig.SparseCheckoutFilesleaked to the release repo primary ref, causing "unrelated histories" errorsbinary_build_commandsSparseCheckoutFileswas not propagated fromDecorationConfigtoextra_refs[0]Context
The original sparse checkout feature (353a4a0) was reverted in #5145 due to multiple bugs.
Upstream prow PR kubernetes-sigs/prow#639 added
SparseCheckoutFilessupporton
Refs, which is consumed by clonerefs to limit the working tree.How it works
sparseCheckoutFiles()computes the file list fromReleaseBuildConfiguration:.ci-operator.yamliffrom_repositoryis setdockerfile_pathentries (respectingcontext_dir, skippingdockerfile_literaland externalref)DecorationConfig.SparseCheckoutFilesinstead ofskip_cloning: trueskip_cloning: true(no change)pkg/steps/source.go) always clears sparse checkout to get a full cloneDecorationConfigwhen the primary ref differs from the target repoDecorationConfigtoextra_refs[0]Other job types verified safe
ExtraRefsare replaced with fresh refs;PeriodicSpec()doesn't callCompletePrimaryRefsExtraRefsare rebuilt;CompletePrimaryRefsonly applies to the primary refTesting
🤖 Generated with Claude Code