New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestUpdateNodeWithMultiplePods will fail if we run all test in its package (pkg/controller/node/scheduler/) #55671

Closed
NickrenREN opened this Issue Nov 14, 2017 · 5 comments

Comments

Projects
None yet
3 participants
@NickrenREN
Member

NickrenREN commented Nov 14, 2017

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:
When we run single test for TestUpdateNodeWithMultiplePods , it will always succeed, but if we run all test in its package, it will fail.

What you expected to happen:
Always succeed

How to reproduce it (as minimally and precisely as possible):
run tests using make test WHAT=./pkg/controller/node/scheduler/

root@nickren-14:/home/nickren/GoProjects/src/k8s.io/kubernetes# make test WHAT=./pkg/controller/node/scheduler/
Running tests for APIVersion: v1,admissionregistration.k8s.io/v1alpha1,admission.k8s.io/v1alpha1,apps/v1beta1,apps/v1beta2,apps/v1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,
+++ [1114 15:45:12] Running tests without code coverage
--- FAIL: TestUpdateNodeWithMultiplePods (2.50s)
        taint_controller_test.go:486: Starting testcase "Pods with different toleration times are evicted appropriately"
        taint_controller_test.go:512: Sleeping for 499.999781ms
        taint_controller_test.go:521: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:538: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:512: Sleeping for 549.098983ms
        taint_controller_test.go:532: Failed to deleted pod pod2 after 0
        taint_controller_test.go:486: Starting testcase "Evict all pods not maching all taints instantly"
        taint_controller_test.go:512: Sleeping for 499.999786ms
        taint_controller_test.go:521: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:521: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:521: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:538: Found not-delete action with verb list. Ignoring.
FAIL
FAIL    k8s.io/kubernetes/pkg/controller/node/scheduler 36.237s
make: *** [test] Error 1

/sig node

@NickrenREN

This comment has been minimized.

Show comment
Hide comment
@NickrenREN

NickrenREN Nov 14, 2017

Member

I debug for this, and find it is time issue.
First test case, pod2 will be evicted after 1 second, but fakeClientset.Actions() is nil

root@nickren-14:/home/nickren/GoProjects/src/k8s.io/kubernetes# make test WHAT=./pkg/controller/node/scheduler/
Running tests for APIVersion: v1,admissionregistration.k8s.io/v1alpha1,admission.k8s.io/v1alpha1,apps/v1beta1,apps/v1beta2,apps/v1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,
+++ [1114 15:43:41] Running tests without code coverage
--- FAIL: TestUpdateNodeWithMultiplePods (2.47s)
        taint_controller_test.go:486: Starting testcase "Pods with different toleration times are evicted appropriately"
        taint_controller_test.go:512: Sleeping for 499.999795ms
        taint_controller_test.go:518:  actions: [{{ list { v1 pods} } /v1, Kind=Pod  { spec.nodeName=node1}} {{default delete { v1 pods} } pod1}]
        taint_controller_test.go:522: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:539: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:512: Sleeping for 579.717115ms
        taint_controller_test.go:518:  actions: []
        taint_controller_test.go:533: Failed to deleted pod pod2 after 0
        taint_controller_test.go:486: Starting testcase "Evict all pods not maching all taints instantly"
        taint_controller_test.go:512: Sleeping for 499.999746ms
        taint_controller_test.go:518:  actions: [{{ list { v1 pods} } /v1, Kind=Pod  { spec.nodeName=node1}} {{default delete { v1 pods} } pod3} {{default delete { v1 pods} } pod1} {{default delete { v1 pods} } pod2}]
        taint_controller_test.go:522: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:518:  actions: [{{ list { v1 pods} } /v1, Kind=Pod  { spec.nodeName=node1}} {{default delete { v1 pods} } pod3} {{default delete { v1 pods} } pod1} {{default delete { v1 pods} } pod2}]
        taint_controller_test.go:522: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:518:  actions: [{{ list { v1 pods} } /v1, Kind=Pod  { spec.nodeName=node1}} {{default delete { v1 pods} } pod3} {{default delete { v1 pods} } pod1} {{default delete { v1 pods} } pod2}]
        taint_controller_test.go:522: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:539: Found not-delete action with verb list. Ignoring.
FAIL
FAIL    k8s.io/kubernetes/pkg/controller/node/scheduler 36.244s
make: *** [test] Error 1
Member

NickrenREN commented Nov 14, 2017

I debug for this, and find it is time issue.
First test case, pod2 will be evicted after 1 second, but fakeClientset.Actions() is nil

root@nickren-14:/home/nickren/GoProjects/src/k8s.io/kubernetes# make test WHAT=./pkg/controller/node/scheduler/
Running tests for APIVersion: v1,admissionregistration.k8s.io/v1alpha1,admission.k8s.io/v1alpha1,apps/v1beta1,apps/v1beta2,apps/v1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,
+++ [1114 15:43:41] Running tests without code coverage
--- FAIL: TestUpdateNodeWithMultiplePods (2.47s)
        taint_controller_test.go:486: Starting testcase "Pods with different toleration times are evicted appropriately"
        taint_controller_test.go:512: Sleeping for 499.999795ms
        taint_controller_test.go:518:  actions: [{{ list { v1 pods} } /v1, Kind=Pod  { spec.nodeName=node1}} {{default delete { v1 pods} } pod1}]
        taint_controller_test.go:522: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:539: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:512: Sleeping for 579.717115ms
        taint_controller_test.go:518:  actions: []
        taint_controller_test.go:533: Failed to deleted pod pod2 after 0
        taint_controller_test.go:486: Starting testcase "Evict all pods not maching all taints instantly"
        taint_controller_test.go:512: Sleeping for 499.999746ms
        taint_controller_test.go:518:  actions: [{{ list { v1 pods} } /v1, Kind=Pod  { spec.nodeName=node1}} {{default delete { v1 pods} } pod3} {{default delete { v1 pods} } pod1} {{default delete { v1 pods} } pod2}]
        taint_controller_test.go:522: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:518:  actions: [{{ list { v1 pods} } /v1, Kind=Pod  { spec.nodeName=node1}} {{default delete { v1 pods} } pod3} {{default delete { v1 pods} } pod1} {{default delete { v1 pods} } pod2}]
        taint_controller_test.go:522: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:518:  actions: [{{ list { v1 pods} } /v1, Kind=Pod  { spec.nodeName=node1}} {{default delete { v1 pods} } pod3} {{default delete { v1 pods} } pod1} {{default delete { v1 pods} } pod2}]
        taint_controller_test.go:522: Found not-delete action with verb list. Ignoring.
        taint_controller_test.go:539: Found not-delete action with verb list. Ignoring.
FAIL
FAIL    k8s.io/kubernetes/pkg/controller/node/scheduler 36.244s
make: *** [test] Error 1
@NickrenREN

This comment has been minimized.

Show comment
Hide comment
@NickrenREN

NickrenREN Nov 14, 2017

Member

/assign

Member

NickrenREN commented Nov 14, 2017

/assign

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Feb 12, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot commented Feb 12, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Mar 14, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

fejta-bot commented Mar 14, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Apr 13, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot commented Apr 13, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment