fix(pkg): fallback at using git client when github API files limits is reached #146

FedeDP · 2024-05-08T12:14:59Z

Github API has an hard limit of 3000 files returned for push/pull request events. Workaround this when checking pre/post submit jobs to be triggered, by manually diffing using the git client.
See https://docs.github.com/en/rest/pulls/pulls?apiVersion=2022-11-28#list-pull-requests-files.

Porting of kubernetes/test-infra#30615 to the new repo.

k8s-ci-robot · 2024-05-08T12:15:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: FedeDP
Once this PR has been reviewed and has the lgtm label, please assign stevekuznetsov for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

cmd/status-reconciler/OWNERS
pkg/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2024-05-08T12:15:08Z

Welcome @FedeDP!

It looks like this is your first PR to kubernetes-sigs/prow 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/prow has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2024-05-08T12:15:08Z

Hi @FedeDP. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2024-05-08T12:15:14Z

✅ Deploy Preview for k8s-prow ready!

Name	Link
🔨 Latest commit	`cd46992`
🔍 Latest deploy log	https://app.netlify.com/sites/k8s-prow/deploys/6673d6376fe4320008c0f956
😎 Deploy Preview	https://deploy-preview-146--k8s-prow.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

FedeDP · 2024-05-08T12:17:20Z

pkg/config/jobs.go

-				changedFiles = append(changedFiles, change.Filename)
+
+			// Fallback to use gitClient since github API truncated the response
+			if len(changes) == github.ChangesFilesLimit && gc != nil {


As suggested here: kubernetes/test-infra#30615 (comment) this is now a fallback, that is only executed when:

gitclient is not nil

length of changes is exactly 3000

Of course, this means that if unluckily number of changes is exactly 3000, we would run the git client flow for nothing.

FedeDP · 2024-05-08T12:17:46Z

pkg/config/jobs.go

+			}
+
+			// in all failure cases, return truncated files
+			if len(changedFiles) == 0 {


If the gitclient approach fails, return the truncated list of files.

FedeDP · 2024-05-08T12:18:48Z

pkg/plugins/trigger/push.go

 			}
 		}
 		var changedFiles []string
-		for file := range changed {
-			changedFiles = append(changedFiles, file)
+		if len(changes) == github.ChangesFilesLimit && gc != nil {


As above for pull requests, only load the new code if exactly 3000 files changes are found.

FedeDP · 2024-05-08T12:19:03Z

pkg/plugins/trigger/push.go

+			}
+		}
+
+		if len(changedFiles) == 0 {


If the gitclient approach fails, return the truncated list of files just like before.

FedeDP · 2024-05-08T12:20:59Z

Note: i am still trying to understand how to test these changes; i am currently building all images and pushing them under my own dockerhub account; then i might switch our https://github.com/falcosecurity/test-infra repo to use my own prow images and see if triggering a job that failed before, works fine after.

FedeDP · 2024-05-08T12:21:04Z

/hold

…s reached. Github API has an hard limit of 3000 files returned for push/pull request events. Workaround this when checking pre/post submit jobs to be triggered, by manually diffing using the git client. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

FedeDP · 2024-05-09T09:29:02Z

pkg/plugins/trigger/push.go

 	"sigs.k8s.io/prow/pkg/github"
 	"sigs.k8s.io/prow/pkg/pjutil"
 )

-func listPushEventChanges(pe github.PushEvent) config.ChangedFilesProvider {
+func listPushEventChanges(gc git.ClientFactory, pe github.PushEvent) config.ChangedFilesProvider {
+	var changedFiles []string


This should be the only actual "breaking change", ie: cache changedFiles just like we do in pull-request.go, to avoid a git client diff call for each configured post submit when we are in the fallback case.

petr-muller · 2024-06-19T17:52:53Z

/ok-to-test

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

k8s-triage-robot · 2024-09-18T07:37:19Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

FedeDP · 2024-09-18T07:49:37Z

/remove-lifecycle stale

Sorry i still hadn't a chance to test this.

k8s-ci-robot requested review from cjwagner and droslean May 8, 2024 12:15

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 8, 2024

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 8, 2024

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 8, 2024

FedeDP commented May 8, 2024

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 8, 2024

FedeDP force-pushed the fix/trigger_github_api_files_limits branch from 4651916 to ad16d86 Compare May 9, 2024 09:27

FedeDP commented May 9, 2024

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 19, 2024

chore(pkg): fixed github spelling issues.

cd46992

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 18, 2024

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(pkg): fallback at using git client when github API files limits is reached #146

fix(pkg): fallback at using git client when github API files limits is reached #146

FedeDP commented May 8, 2024

k8s-ci-robot commented May 8, 2024

k8s-ci-robot commented May 8, 2024

k8s-ci-robot commented May 8, 2024

netlify bot commented May 8, 2024 •

edited

Loading

FedeDP May 8, 2024

FedeDP May 8, 2024

FedeDP May 8, 2024

FedeDP May 8, 2024 •

edited

Loading

FedeDP commented May 8, 2024 •

edited

Loading

FedeDP commented May 8, 2024

FedeDP May 9, 2024

petr-muller commented Jun 19, 2024

k8s-triage-robot commented Sep 18, 2024

FedeDP commented Sep 18, 2024 •

edited

Loading

fix(pkg): fallback at using git client when github API files limits is reached #146

Are you sure you want to change the base?

fix(pkg): fallback at using git client when github API files limits is reached #146

Conversation

FedeDP commented May 8, 2024

k8s-ci-robot commented May 8, 2024

k8s-ci-robot commented May 8, 2024

k8s-ci-robot commented May 8, 2024

netlify bot commented May 8, 2024 • edited Loading

✅ Deploy Preview for k8s-prow ready!

FedeDP May 8, 2024

Choose a reason for hiding this comment

FedeDP May 8, 2024

Choose a reason for hiding this comment

FedeDP May 8, 2024

Choose a reason for hiding this comment

FedeDP May 8, 2024 • edited Loading

Choose a reason for hiding this comment

FedeDP commented May 8, 2024 • edited Loading

FedeDP commented May 8, 2024

FedeDP May 9, 2024

Choose a reason for hiding this comment

petr-muller commented Jun 19, 2024

k8s-triage-robot commented Sep 18, 2024

FedeDP commented Sep 18, 2024 • edited Loading

netlify bot commented May 8, 2024 •

edited

Loading

FedeDP May 8, 2024 •

edited

Loading

FedeDP commented May 8, 2024 •

edited

Loading

FedeDP commented Sep 18, 2024 •

edited

Loading