Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wait for pod not running or gone in storage tests #113135

Merged
merged 1 commit into from Oct 18, 2022
Merged

Wait for pod not running or gone in storage tests #113135

merged 1 commit into from Oct 18, 2022

Conversation

soltysh
Copy link
Contributor

@soltysh soltysh commented Oct 18, 2022

What type of PR is this?

The storage test [sig-storage] Multi-AZ Cluster Volumes should schedule pods in the same zones as statically provisioned PVs from

kubernetes/test/e2e/storage/ubernetes_lite_volumes.go

Line 53 in 83415e5

ginkgo.It("should schedule pods in the same zones as statically provisioned PVs", func() {
relies during cleanup on a for loop checking all the pods to be gone see

kubernetes/test/e2e/storage/ubernetes_lite_volumes.go

Line 96 in 83415e5

e2epod.WaitForPodNoLongerRunningInNamespace(c, config.pod.Name, ns)
, which assumes either NotFound or finished.

During refactoring in #109704 that condition has changed, previously WaitForPodNoLongerRunningInNamespace worked for both NotFound and Completed pod, after - it only checks Completed pods. This PR introduces another method which works for both conditions.

Found in OpenShift CI (https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/27416/pull-ci-openshift-origin-master-e2e-gcp-ovn/1582239480008413184) where this test consistently fails, since we run more packed cluster where PodGC will kick in sooner removing the pods.

/kind cleanup
/kind failing-test

Special notes for your reviewer:

/assign @jsafrane
/cc @aojea @tallclair

Does this PR introduce a user-facing change?

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 18, 2022
@k8s-ci-robot k8s-ci-robot added kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 18, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added area/e2e-test-framework Issues or PRs related to refactoring the kubernetes e2e test framework area/test approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. sig/storage Categorizes an issue or PR as relevant to SIG Storage. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/testing Categorizes an issue or PR as relevant to SIG Testing. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 18, 2022
@soltysh
Copy link
Contributor Author

soltysh commented Oct 18, 2022

/triage accepted
/priority important-longterm

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Oct 18, 2022
@jsafrane
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 18, 2022
@openshift-ci-robot openshift-ci-robot mentioned this pull request Oct 18, 2022
Merged
@dims
Copy link
Member

dims commented Oct 18, 2022

/retest

@aojea
Copy link
Member

aojea commented Oct 18, 2022

cc @pohly

aojea
aojea reviewed Oct 18, 2022
View reviewed changes
test/e2e/framework/pod/wait.go
@@ -421,6 +421,25 @@ func WaitForPodNameUnschedulableInNamespace(c clientset.Interface, podName, name
})
}

// WaitTimeoutForPodNoLongerRunningOrNotFoundInNamespace waits default amount of time (defaultPodDeletionTimeout)
// for the specified pod to stop running or disappear. Returns an error if timeout occurs first.
func WaitTimeoutForPodNoLongerRunningOrNotFoundInNamespace(c clientset.Interface, podName, namespace string) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this not the same as WaitForPodNotFoundInNamespace ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L545

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WaitForPodNotFoundInNamespace waits explicitly for NotFound, whereas this is either NotFound or Completed (ie. WaitTimeoutForPodNoLongerRunningInNamespace) so you can think of it as a combination of the two. That's how the code in storage was relying on it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the test that uses this, it seems like it's called after the pod is deleted? Wouldn't it be better to just wait for NotFound in that case?

@k8s-ci-robot k8s-ci-robot merged commit 8186ebd into kubernetes:master Oct 18, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.26 milestone Oct 18, 2022
tallclair
tallclair reviewed Oct 18, 2022
View reviewed changes
Copy link
Member

@tallclair tallclair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is already merged, but left some questions about whether the original behavior was correct.

test/e2e/framework/pod/wait.go
@@ -421,6 +421,25 @@ func WaitForPodNameUnschedulableInNamespace(c clientset.Interface, podName, name
})
}

// WaitTimeoutForPodNoLongerRunningOrNotFoundInNamespace waits default amount of time (defaultPodDeletionTimeout)
// for the specified pod to stop running or disappear. Returns an error if timeout occurs first.
func WaitTimeoutForPodNoLongerRunningOrNotFoundInNamespace(c clientset.Interface, podName, namespace string) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the test that uses this, it seems like it's called after the pod is deleted? Wouldn't it be better to just wait for NotFound in that case?

test/e2e/storage/ubernetes_lite_volumes.go
Comment on lines 97 to 98
e2epv.PVPVCCleanup(c, ns, config.pv, config.pvc)
err = e2epv.DeletePVSource(config.pvSource)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will these succeed if the pod still exists (but isn't running)?

test/e2e/storage/ubernetes_lite_volumes.go
@@ -93,7 +93,7 @@ func PodsUseStaticPVsOrFail(f *framework.Framework, podCount int, image string)
e2epod.DeletePodOrFail(c, ns, config.pod.Name)
}
for _, config := range configs {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this could be run in parallel to speed it up.

@soltysh
Copy link
Contributor Author

soltysh commented Oct 18, 2022

I'll sync with storage folks and get them updated in followups, your points are totally valid @tallclair

@soltysh soltysh deleted the fix_storage_timeouts branch October 18, 2022 20:13
@soltysh soltysh mentioned this pull request Oct 19, 2022
Merged
@outscale-mdr outscale-mdr mentioned this pull request Feb 16, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aojea aojea aojea left review comments

@tallclair tallclair tallclair left review comments

Assignees

@jsafrane jsafrane

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/e2e-test-framework Issues or PRs related to refactoring the kubernetes e2e test framework area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. release-note-none Denotes a PR that doesn't merit a release note. sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.26
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@soltysh @k8s-ci-robot @jsafrane @dims @aojea @tallclair