New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix 'Schedulercache is corrupted' error #55262

Merged
merged 1 commit into from Nov 8, 2017

Conversation

@liggitt
Member

liggitt commented Nov 7, 2017

Fixes #50916

If an Assume()ed pod is Add()ed with a different nodeName, the podStates view of the pod is not corrected to reflect the actual nodeName. On the next Update(), the scheduler observes the mismatch and process exits.

Fixed 'Schedulercache is corrupted' error in kube-scheduler
@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Nov 7, 2017

Member

/retest

Member

liggitt commented Nov 7, 2017

/retest

@timothysc

This comment has been minimized.

Show comment
Hide comment
@timothysc
Member

timothysc commented Nov 7, 2017

/assign @wojtek-t

@timothysc timothysc modified the milestone: v1.9 Nov 7, 2017

@timothysc timothysc added this to the v1.8 milestone Nov 7, 2017

@timothysc

/lgtm - minor comment.

Show outdated Hide outdated plugin/pkg/scheduler/schedulercache/cache_test.go Outdated
@k8s-cherrypick-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-cherrypick-bot

k8s-cherrypick-bot Nov 7, 2017

Removing label cherrypick-candidate because no release milestone was set. This is an invalid state and thus this PR is not being considered for cherry-pick to any release branch. Please add an appropriate release milestone and then re-add the label.

k8s-cherrypick-bot commented Nov 7, 2017

Removing label cherrypick-candidate because no release milestone was set. This is an invalid state and thus this PR is not being considered for cherry-pick to any release branch. Please add an appropriate release milestone and then re-add the label.

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Nov 7, 2017

Member

updated to remove unrelated ttl bit from unit test

Member

liggitt commented Nov 7, 2017

updated to remove unrelated ttl bit from unit test

@cblecker

This comment has been minimized.

Show comment
Hide comment
@cblecker

cblecker Nov 7, 2017

Member

CRD Flake
/test pull-kubernetes-unit

Member

cblecker commented Nov 7, 2017

CRD Flake
/test pull-kubernetes-unit

@bsalamat

Your change looks good to me, I just wonder if Scheduler resource accounting remains valid in such scenario where a pod is assumed to a different node that it is bound to.

Show outdated Hide outdated plugin/pkg/scheduler/schedulercache/cache_test.go Outdated
@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Nov 7, 2017

Member

fixed gofmt error and test message, re-tagging

Member

liggitt commented Nov 7, 2017

fixed gofmt error and test message, re-tagging

@k8s-merge-robot k8s-merge-robot removed the lgtm label Nov 7, 2017

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Nov 7, 2017

Member

I just wonder if Scheduler resource accounting remains valid in such scenario where a pod is assumed to a different node that it is bound to.

It was just the podStates' version of the pod that wasn't getting updated. The removePod(currState.pod)/addPod(pod) dance corrects the per-node accounting already (and the unit test I added demonstrates the per-node accounting is corrected):

case ok && cache.assumedPods[key]:
if currState.pod.Spec.NodeName != pod.Spec.NodeName {
// The pod was added to a different node than it was assumed to.
glog.Warningf("Pod %v assumed to a different node than added to.", key)
// Clean this up.
cache.removePod(currState.pod)
cache.addPod(pod)
}

Member

liggitt commented Nov 7, 2017

I just wonder if Scheduler resource accounting remains valid in such scenario where a pod is assumed to a different node that it is bound to.

It was just the podStates' version of the pod that wasn't getting updated. The removePod(currState.pod)/addPod(pod) dance corrects the per-node accounting already (and the unit test I added demonstrates the per-node accounting is corrected):

case ok && cache.assumedPods[key]:
if currState.pod.Spec.NodeName != pod.Spec.NodeName {
// The pod was added to a different node than it was assumed to.
glog.Warningf("Pod %v assumed to a different node than added to.", key)
// Clean this up.
cache.removePod(currState.pod)
cache.addPod(pod)
}

@liggitt liggitt added the lgtm label Nov 7, 2017

@cblecker

This comment has been minimized.

Show comment
Hide comment
@cblecker

cblecker Nov 7, 2017

Member

@liggitt looks like there's still a gofmt issue on ./plugin/pkg/scheduler/schedulercache/cache_test.go

Member

cblecker commented Nov 7, 2017

@liggitt looks like there's still a gofmt issue on ./plugin/pkg/scheduler/schedulercache/cache_test.go

@bsalamat

This comment has been minimized.

Show comment
Hide comment
@bsalamat

bsalamat Nov 7, 2017

Contributor

Thanks, @liggitt!

/lgtm

Contributor

bsalamat commented Nov 7, 2017

Thanks, @liggitt!

/lgtm

@k8s-merge-robot k8s-merge-robot removed the lgtm label Nov 7, 2017

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Nov 7, 2017

Member

actually fixed gofmt error

Member

liggitt commented Nov 7, 2017

actually fixed gofmt error

@cblecker

This comment has been minimized.

Show comment
Hide comment
@cblecker

cblecker Nov 7, 2017

Member

reapplying LGTM
/lgtm

Member

cblecker commented Nov 7, 2017

reapplying LGTM
/lgtm

@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Nov 7, 2017

Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, cblecker, liggitt, timothysc

Associated issue: 50916

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

Contributor

k8s-merge-robot commented Nov 7, 2017

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, cblecker, liggitt, timothysc

Associated issue: 50916

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@cblecker

This comment has been minimized.

Show comment
Hide comment
@cblecker

cblecker Nov 7, 2017

Member

uhhh..

W1107 23:53:54.719] Run: ('bash', '-c', 'cd kubernetes && ./hack/jenkins/test-dockerized.sh')
W1107 23:53:54.722] bash: line 0: cd: kubernetes: No such file or directory

That's an odd unit test failure. Let's retry.
/test pull-kubernetes-unit

Member

cblecker commented Nov 7, 2017

uhhh..

W1107 23:53:54.719] Run: ('bash', '-c', 'cd kubernetes && ./hack/jenkins/test-dockerized.sh')
W1107 23:53:54.722] bash: line 0: cd: kubernetes: No such file or directory

That's an odd unit test failure. Let's retry.
/test pull-kubernetes-unit

@cblecker

This comment has been minimized.

Show comment
Hide comment
@cblecker

cblecker Nov 8, 2017

Member

unit test failing: #55276

Member

cblecker commented Nov 8, 2017

unit test failing: #55276

@cblecker

This comment has been minimized.

Show comment
Hide comment
@cblecker

cblecker Nov 8, 2017

Member

fingers crossed
/test pull-kubernetes-unit

Member

cblecker commented Nov 8, 2017

fingers crossed
/test pull-kubernetes-unit

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Nov 8, 2017

Member

TestCRD flake
/test pull-kubernetes-unit

Member

liggitt commented Nov 8, 2017

TestCRD flake
/test pull-kubernetes-unit

jpbetz added a commit that referenced this pull request Nov 8, 2017

Merge pull request #55266 from liggitt/automated-cherry-pick-of-#55262-…
…upstream-release-1.8

Automated cherry pick of #55262
@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Nov 8, 2017

Contributor

/test all [submit-queue is verifying that this PR is safe to merge]

Contributor

k8s-merge-robot commented Nov 8, 2017

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Nov 8, 2017

Contributor

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

Contributor

k8s-merge-robot commented Nov 8, 2017

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-merge-robot k8s-merge-robot merged commit 33f873d into kubernetes:master Nov 8, 2017

11 of 12 checks passed

Submit Queue Required Github CI test is not green: pull-kubernetes-e2e-gce
Details
cla/linuxfoundation liggitt authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-kops-aws Jenkins job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce Jenkins job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-unit Jenkins job succeeded.
Details
pull-kubernetes-verify Jenkins job succeeded.
Details

@liggitt liggitt deleted the liggitt:schedulercache branch Nov 9, 2017

k8s-merge-robot added a commit that referenced this pull request Nov 14, 2017

Merge pull request #55267 from liggitt/automated-cherry-pick-of-#55262-…
…upstream-release-1.7

Automatic merge from submit-queue.

Automated cherry pick of #55262

Cherry pick of #55262 on release-1.7.

#55262: Fix 'Schedulercache is corrupted' error
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment