Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flaking Test] integration TestStorageVersionMigrationWithCRD #123921

Closed
pacoxu opened this issue Mar 14, 2024 · 13 comments · Fixed by #123983, #124143 or #124877
Closed

[Flaking Test] integration TestStorageVersionMigrationWithCRD #123921

pacoxu opened this issue Mar 14, 2024 · 13 comments · Fixed by #123983, #124143 or #124877
Assignees
@nilekhc
Labels
kind/flake Categorizes issue or PR as related to a flaky test. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@pacoxu
Copy link
Member

pacoxu commented Mar 14, 2024

Which jobs are flaking?

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-integration-master/1767480303611285504

Which tests are flaking?

k8s.io/kubernetes/test/integration: storageversionmigrator TestStorageVersionMigrationWithCRD

Since when has it been flaking?

Added with #123344 03-09.

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#integration-master

Reason for failure (if possible)

E0312 10:10:59.700953  118731 watcher.go:471] failed to prepare current and previous objects: request to convert CR from an invalid group/version: stable.example.com/v2

Anything else we need to know?

/cc @nilekhc @deads2k @jpbetz

Relevant SIG(s)

/sig api-machinery
@pacoxu pacoxu added the kind/flake Categorizes issue or PR as related to a flaky test. label Mar 14, 2024
@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 14, 2024
@jiahuif
Copy link
Member

jiahuif commented Mar 14, 2024

/assign @nilekhc
as the feature owner. Thank you.
/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 14, 2024
@Vyom-Yadav
Copy link
Member

@jiahuif @nilekhc Can you please confirm what type of flake this is? Is it due to a bug, improper test, etc? I am assuming this won't block upcoming release cuts. Is that correct?

@aramase
Copy link
Member

aramase commented Mar 18, 2024 •
edited

Looking at the logs, the actual failure is here

 util.go:1008: Failed to create CR: client rate limiter Wait returned an error: context deadline exceeded

kubernetes/test/integration/storageversionmigrator/util.go

Lines 798 to 801 in a2a7090

crdUnstructured, err := svm.dynamicClient.Resource(crdResource).Namespace(defaultNamespace).Create(ctx, crdUnstructured, metav1.CreateOptions{})
if err != nil {
t.Fatalf("Failed to create CR: %v", err)
}

that's from

kubernetes/test/integration/storageversionmigrator/util.go

Lines 1002 to 1010 in a2a7090

err := wait.PollUntilContextTimeout(
ctx,
500*time.Millisecond,
wait.ForeverTestTimeout,
true,
func(ctx context.Context) (bool, error) {
triggerCR := svm.createCR(ctx, t, "triggercr", "v1")
svm.deleteCR(ctx, t, triggerCR.GetName(), "v1")
svmResource, err := svm.getSVM(ctx, t, crdSVMName)

which is failing after the 30s context deadline.

@aramase aramase mentioned this issue Mar 19, 2024
Merged
@pacoxu
Copy link
Member Author

pacoxu commented Mar 26, 2024

@aramase https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-integration-master/1772137045368508416 (03-24)failed after #123983 was merged(03-19)

@pacoxu
Copy link
Member Author

pacoxu commented Mar 26, 2024

/reopen

@k8s-ci-robot k8s-ci-robot reopened this Mar 26, 2024
@k8s-ci-robot
Copy link
Contributor

@pacoxu: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@pacoxu
Copy link
Member Author

pacoxu commented Apr 1, 2024

@jiahuif @nilekhc do you have any update or plan on this?

@aramase
Copy link
Member

aramase commented Apr 1, 2024

  1. Fix for TestStorageVersionMigrationWithCRD integration test failure #123983 addressed the first flake with context deadline.
  2. PR to address the new flakes: svm test: wait after updating CRD to avoid flakes #124143

@enj enj mentioned this issue Apr 1, 2024
Merged
@pacoxu
Copy link
Member Author

pacoxu commented Apr 2, 2024

  1. Fix for TestStorageVersionMigrationWithCRD integration test failure #123983 addressed the first flake with context deadline.
  2. PR to address the new flakes: svm test: wait after updating CRD to avoid flakes #124143

@aramase Thanks for the update. No flake until now.

@benluddy
Copy link
Contributor

benluddy commented May 3, 2024

@aramase @enj Just ran into a presubmit failure that looks similar, is this the same issue?

util.go:1013: Failed to create CR: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/121496/pull-kubernetes-integration/1786413739864494080#1:build-log.txt%3A129362

@benluddy benluddy mentioned this issue May 3, 2024
Merged
@aramase
Copy link
Member

aramase commented May 3, 2024 •
edited

@aramase @enj Just ran into a presubmit failure that looks similar, is this the same issue?

util.go:1013: Failed to create CR: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/121496/pull-kubernetes-integration/1786413739864494080#1:build-log.txt%3A129362

@nilekhc PTAL!

@aramase
Copy link
Member

aramase commented May 3, 2024

/reopen

@k8s-ci-robot k8s-ci-robot reopened this May 3, 2024
@k8s-ci-robot
Copy link
Contributor

@aramase: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@nilekhc nilekhc mentioned this issue May 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nilekhc nilekhc

Labels
kind/flake Categorizes issue or PR as related to a flaky test. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
CI Signal (SIG Release / Release Team)
Status: RESOLVED
Milestone
No milestone
7 participants
@benluddy @nilekhc @pacoxu @jiahuif @aramase @k8s-ci-robot @Vyom-Yadav