CSI E2E: retry csi-pod creation #68822
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
release-note-none
k8s-ci-robot
added
the
sig/testing
nikhita
reviewed
Sep 19, 2018
test/e2e/storage/csi_objects.go
Outdated
| var err error | ||
| ret, err = podClient.Create(pod) | ||
| return err | ||
| }).ShouldNot(HaveOccurred(), "Failed to create %q pod: %v", pod.GetName()) |
There was a problem hiding this comment.
I meant to remove the "%v". Gomega itself will dump the error. But well-spotted, thanks.
Further testing also showed that the default Expect() timeouts were too short. I've bumped that to 1m in the revised commit.
|
/kind bug |
k8s-ci-robot
added
kind/bug
pohly
force-pushed
the
csi-pod-creation
branch
2 times, most recently
from
0991e4f
to
b30f556
Compare
|
/retest |
msau42
reviewed
Sep 19, 2018
test/e2e/storage/csi_objects.go
Outdated
| // We could use a DaemonSet, but then the name of the csi-pod changes | ||
| // during each test run. It's simpler to just try for a while here. | ||
| var ret *v1.Pod | ||
| Eventually(func() error { |
There was a problem hiding this comment.
May be worth adding a CreatePodWithRetry utility function?
There was a problem hiding this comment.
Yes, that may be useful. I'll do that.
There was a problem hiding this comment.
Done. I ended up adding CreateEventually to pod.go's PodClient, which looked like the right place because it already had a Create (without retry).
pohly
changed the title
k8s-ci-robot
added
the
do-not-merge/work-in-progress
Normally the pod would get created via a DaemonSet controller, but during testing it is easier to create it directly. We just need to ignore errors (like 'No API token found for service account "csi-service-account"') and retry for a while. If the error persists, the error will still abort and report it eventually. This problem also occurs elsewhere, so an utility function in the framework for it seems justified. Fixes: kubernetes#68776
pohly
force-pushed
the
csi-pod-creation
branch
from
b30f556
to
6dbb07c
Compare
k8s-ci-robot
added
size/M
pohly
changed the title
k8s-ci-robot
removed
the
do-not-merge/work-in-progress
|
/retest |
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
/assign saad-ali |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: pohly, saad-ali The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
avalluri
pushed a commit
to intel/oim
that referenced
this pull request
Jan 30, 2019
As discussed upstream (kubernetes/kubernetes#67882), the 'No API token found for service account "csi-service-account"' error is normal and must be handled by trying to create the pod multiple times, either manually or via a DaemonSet. Here we simply loop with Gomega (same fix that is also proposed upstream in kubernetes/kubernetes#68822).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Normally the pod would get created via a DaemonSet controller, but
during testing it is easier to create it directly. We just need to
ignore errors (like 'No API token found for service account
"csi-service-account"') and retry for a while. If the error persists,
the error will still abort and report it eventually.
What this PR does / why we need it:
The CSI E2E can fail randomly due to a race condition.
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #68776
Release note:
/sig storage