Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add some more logging so that we can debug PD delete errors. #10001

Merged
merged 1 commit into from Jun 18, 2015

Conversation

brendandburns
Copy link
Contributor

@brendandburns
Copy link
Contributor Author

@ixdy too.

@k8s-bot
Copy link

k8s-bot commented Jun 18, 2015

GCE e2e build/test passed for commit 0bf936d.

@@ -214,7 +214,12 @@ func deletePD(pdName string) error {
zone := testContext.CloudConfig.Zone

// TODO: make this hit the compute API directly.
return exec.Command("gcloud", "compute", "--project="+testContext.CloudConfig.ProjectID, "disks", "delete", "--zone="+zone, pdName).Run()
cmd := exec.Command("gcloud", "compute", "--project="+testContext.CloudConfig.ProjectID, "disks", "delete", "--zone="+zone, pdName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any interest in stdout vs stderr?

If this is flaky, doing some parsing and retrying on transient errors might be a good thing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

combined output gets both. I want the data first, then we can see what to do with it.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure that the error is that the PD is still in use (i.e. attached). In that case, retrying the deletion probably won't help (although retrying the detachment probably will). As per @brendandburns, let's see what data this this unearths before deciding. I'm fairly certain that what we'll want here in the end will be explicit confirmation that the detachment has completed (not just that the detach API call has succeeded), and also retries on the PD deletion.
As an aside, I'm not 100% convinced that this test should actually be made to fail if only the PD deletion fails. That seems like a GCE rather than a K8S issue. But I could be convinced either way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, +1, I think this test should probably not fail if GCE delete via gcloud fails, esp. if the disk is no longer attached.

@ghost
Copy link

ghost commented Jun 18, 2015

Sorry, I just saw this now. LGTM.

@ghost ghost added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 18, 2015
@ghost ghost self-assigned this Jun 18, 2015
satnam6502 added a commit that referenced this pull request Jun 18, 2015
Add some more logging so that we can debug PD delete errors.
@satnam6502 satnam6502 merged commit 538d127 into kubernetes:master Jun 18, 2015
@ghost
Copy link

ghost commented Jun 19, 2015

Yes.

#9994 (comment)

I believe that the disk cannot be deleted because it is still attached to
the instance.

Q

On Thu, Jun 18, 2015 at 9:45 PM, Satnam Singh notifications@github.com
wrote:

Any idea why this test failed?
http://kubekins.dls.corp.google.com/job/kubernetes-e2e-gce/7046/testReport/junit/(root)/Kubernetes%20e2e%20suite/Pod_Disks_should_schedule_a_pod_w__a_readonly_PD_on_two_hosts__then_remove_both_/


Reply to this email directly or view it on GitHub
#10001 (comment)
.

@brendandburns brendandburns unassigned ghost Aug 12, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lgtm "Looks good to me", indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants