Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix 'Schedulercache is corrupted' error #55262

Merged
merged 1 commit into from Nov 8, 2017

Conversation

@liggitt
Copy link
Member

commented Nov 7, 2017

Fixes #50916

If an Assume()ed pod is Add()ed with a different nodeName, the podStates view of the pod is not corrected to reflect the actual nodeName. On the next Update(), the scheduler observes the mismatch and process exits.

Fixed 'Schedulercache is corrupted' error in kube-scheduler
@liggitt

This comment has been minimized.

@liggitt

This comment has been minimized.

Copy link
Member Author

commented Nov 7, 2017

/retest

@timothysc

This comment has been minimized.

Copy link
Member

commented Nov 7, 2017

/assign @wojtek-t

@timothysc timothysc modified the milestone: v1.9 Nov 7, 2017

@timothysc timothysc added this to the v1.8 milestone Nov 7, 2017

@timothysc
Copy link
Member

left a comment

/lgtm - minor comment.

t.Fatalf("AddPod failed: %v", err)
}
}
cache.cleanupAssumedPods(now.Add(2 * ttl))

This comment has been minimized.

Copy link
@timothysc

timothysc Nov 7, 2017

Member

Why do you up the ttl?

This comment has been minimized.

Copy link
@liggitt

liggitt Nov 7, 2017

Author Member

this is simulating passing of time to ensure the added pod doesn't expire. was copy/paste from previous testcase, though, and not really relevant to this one, so I can remove it

@k8s-cherrypick-bot

This comment has been minimized.

Copy link

commented Nov 7, 2017

Removing label cherrypick-candidate because no release milestone was set. This is an invalid state and thus this PR is not being considered for cherry-pick to any release branch. Please add an appropriate release milestone and then re-add the label.

@liggitt

This comment has been minimized.

Copy link
Member Author

commented Nov 7, 2017

updated to remove unrelated ttl bit from unit test

@cblecker

This comment has been minimized.

Copy link
Member

commented Nov 7, 2017

CRD Flake
/test pull-kubernetes-unit

@bsalamat
Copy link
Member

left a comment

Your change looks good to me, I just wonder if Scheduler resource accounting remains valid in such scenario where a pod is assumed to a different node that it is bound to.

}
for _, podToUpdate := range tt.podsToUpdate {
if err := cache.UpdatePod(podToUpdate[0], podToUpdate[1]); err != nil {
t.Fatalf("AddPod failed: %v", err)

This comment has been minimized.

Copy link
@bsalamat

bsalamat Nov 7, 2017

Member

s/AddPod/UpdatePod/

@liggitt liggitt force-pushed the liggitt:schedulercache branch from a9b08fa to 5048838 Nov 7, 2017

@liggitt

This comment has been minimized.

Copy link
Member Author

commented Nov 7, 2017

fixed gofmt error and test message, re-tagging

@k8s-github-robot k8s-github-robot removed the lgtm label Nov 7, 2017

@liggitt

This comment has been minimized.

Copy link
Member Author

commented Nov 7, 2017

I just wonder if Scheduler resource accounting remains valid in such scenario where a pod is assumed to a different node that it is bound to.

It was just the podStates' version of the pod that wasn't getting updated. The removePod(currState.pod)/addPod(pod) dance corrects the per-node accounting already (and the unit test I added demonstrates the per-node accounting is corrected):

case ok && cache.assumedPods[key]:
if currState.pod.Spec.NodeName != pod.Spec.NodeName {
// The pod was added to a different node than it was assumed to.
glog.Warningf("Pod %v assumed to a different node than added to.", key)
// Clean this up.
cache.removePod(currState.pod)
cache.addPod(pod)
}

@liggitt liggitt added the lgtm label Nov 7, 2017

@cblecker

This comment has been minimized.

Copy link
Member

commented Nov 7, 2017

@liggitt looks like there's still a gofmt issue on ./plugin/pkg/scheduler/schedulercache/cache_test.go

@bsalamat

This comment has been minimized.

Copy link
Member

commented Nov 7, 2017

Thanks, @liggitt!

/lgtm

@liggitt liggitt force-pushed the liggitt:schedulercache branch from 5048838 to a366e6c Nov 7, 2017

@k8s-github-robot k8s-github-robot removed the lgtm label Nov 7, 2017

@liggitt

This comment has been minimized.

Copy link
Member Author

commented Nov 7, 2017

actually fixed gofmt error

@cblecker

This comment has been minimized.

Copy link
Member

commented Nov 7, 2017

reapplying LGTM
/lgtm

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

commented Nov 7, 2017

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, cblecker, liggitt, timothysc

Associated issue: 50916

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@cblecker

This comment has been minimized.

Copy link
Member

commented Nov 7, 2017

uhhh..

W1107 23:53:54.719] Run: ('bash', '-c', 'cd kubernetes && ./hack/jenkins/test-dockerized.sh')
W1107 23:53:54.722] bash: line 0: cd: kubernetes: No such file or directory

That's an odd unit test failure. Let's retry.
/test pull-kubernetes-unit

@cblecker

This comment has been minimized.

Copy link
Member

commented Nov 8, 2017

unit test failing: #55276

@cblecker

This comment has been minimized.

Copy link
Member

commented Nov 8, 2017

fingers crossed
/test pull-kubernetes-unit

@liggitt

This comment has been minimized.

Copy link
Member Author

commented Nov 8, 2017

TestCRD flake
/test pull-kubernetes-unit

jpbetz added a commit that referenced this pull request Nov 8, 2017

Merge pull request #55266 from liggitt/automated-cherry-pick-of-#5526…
…2-upstream-release-1.8

Automated cherry pick of #55262
@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

commented Nov 8, 2017

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

commented Nov 8, 2017

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-github-robot k8s-github-robot merged commit 33f873d into kubernetes:master Nov 8, 2017

11 of 12 checks passed

Submit Queue Required Github CI test is not green: pull-kubernetes-e2e-gce
Details
cla/linuxfoundation liggitt authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-kops-aws Jenkins job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce Jenkins job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-unit Jenkins job succeeded.
Details
pull-kubernetes-verify Jenkins job succeeded.
Details

@liggitt liggitt deleted the liggitt:schedulercache branch Nov 9, 2017

k8s-github-robot pushed a commit that referenced this pull request Nov 14, 2017

Kubernetes Submit Queue
Merge pull request #55267 from liggitt/automated-cherry-pick-of-#5526…
…2-upstream-release-1.7

Automatic merge from submit-queue.

Automated cherry pick of #55262

Cherry pick of #55262 on release-1.7.

#55262: Fix 'Schedulercache is corrupted' error
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.