gce: Reuse unsuccessfully provisioned volumes. #38702
Conversation
jsafrane
added
the
sig/storage
k8s-ci-robot
added
the
cncf-cla: yes
|
This change is |
k8s-github-robot
added
size/XS
jsafrane
added
release-note-none
jsafrane
force-pushed
the
gce-provisioning-existing
branch
2 times, most recently
from
bec64d2
to
6582f38
Compare
|
Jenkins CRI GCE Node e2e failed for commit bec64d23c63d5f852c7600bfb27d0844d8e67cba. Full PR test history. The magic incantation to run this job again is |
| return gce.waitForZoneOp(createOp, zone) | ||
| err = gce.waitForZoneOp(createOp, zone) | ||
| if isGCEError(err, "alreadyExists") { | ||
| glog.V(2).Infof("GCE PD %q already exists, reuising", name) |
There was a problem hiding this comment.
Maybe Warningf? Also, reuising -> reusing
There was a problem hiding this comment.
fixed both the level and typo
GCE PD names generated by Kubernetes are guaranteed to be unique - they contain name of the cluster and UID of the PVC that is behind it. Presence of a GCE PD that has the same name as we want to provision indicates that previous provisioning did not go well and most probably the controller manager process was restarted in the meantime. Kubernetes should reuse this volume and not provision a new one.
jsafrane
force-pushed
the
gce-provisioning-existing
branch
from
6582f38
to
ba6ab90
Compare
|
[APPROVALNOTIFIER] Needs approval from an approver in each of these OWNERS Files: We suggest the following people: |
|
/approval |
jingxu97
added
the
lgtm
that did not work, please /approve |
|
I approve it by myself :-) |
jsafrane
added
the
approved
|
removing saads review since changes were made and its not showing the comment. |
|
@k8s-bot kops aws e2e test this |
|
@jsafrane: The following test(s) failed:
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
Automatic merge from submit-queue (batch tested with PRs 38702, 41810, 41778, 41858, 41872) |
Automatic merge from submit-queue (batch tested with PRs 38505, 41785, 46315) Fix provisioned GCE PD not being reused if already exists @jsafrane PTAL This is another attempt at #38702 . We have observed that `gce.service.Disks.Insert(gce.projectID, zone, diskToCreate).Do()` instantly gets an error response of alreadyExists, so we must check for it. I am not sure if we still need to check for the error after `waitForZoneOp`; I think that if there is an alreadyExists error, the `Do()` above will always respond with it instantly. But because I'm not sure, and to be safe, I will leave it.
Automatic merge from submit-queue (batch tested with PRs 38505, 41785, 46315) Fix provisioned GCE PD not being reused if already exists @jsafrane PTAL This is another attempt at kubernetes/kubernetes#38702 . We have observed that `gce.service.Disks.Insert(gce.projectID, zone, diskToCreate).Do()` instantly gets an error response of alreadyExists, so we must check for it. I am not sure if we still need to check for the error after `waitForZoneOp`; I think that if there is an alreadyExists error, the `Do()` above will always respond with it instantly. But because I'm not sure, and to be safe, I will leave it.
GCE PD names generated by Kubernetes are guaranteed to be unique - they
contain name of the cluster and UID of the PVC that is behind it.
Presence of a GCE PD that has the same name as we want to provision
indicates that previous provisioning did not go well and most probably
the controller manager process was restarted in the meantime.
Kubernetes should reuse this volume and not provision a new one.
Fixes #38681