Write a guide to writing a new scheduler

[davidopp]

Once we make it possible for people to write new schedulers (#11793), we should write some guidelines/best practices for things people should do when they write a new scheduler. For example

• update the rescheduler
• back off trying to schedule a pod onto a node if kubelet keeps rejecting the pod from that node
• use our standard client code (like pkg/client/cache)
• use the library of kubelet predicates (Move scheduler fit predicates into a library #12744)
• https://github.com/kubernetes/kubernetes/pull/17197/files#r44845537
• etc.

We should probably also write a guide to how to write a new controller, which might have some overlap with a guide on how to write a new scheduler. We already have a very thorough guide to adding new API resources
https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api_changes.md

@lavalamp

[HaiyangDING]

+1 for this doc.

[lavalamp]

Scheduler is an "initializer", in that it sets a field under .spec after an object has been created. Controllers, by contrast, act on other objects, potentially updating the status periodically.

Both share use of watches--possibly over multiple resource types--they're basically the same other than the action(s) they perform.

So I agree that they should probably be covered in the same doc. Our current controllers are possibly slightly more advanced than our scheduler.

[davidopp]

@jdef I just remembered this issue; it's related to your suggestion at the scheduling SIG meeting to have something like a conformance test for new schedulers. We should probably just use this issue.

Just for posterity, the three things we discussed at the meeting were

• must use the predicate library that Kubernetes uses (Move scheduler fit predicates into a library #12744)
• must use standard set of non-predicate criteria Move scheduler fit predicates into a library #12744 (comment)
  (e.g. only consider pods with empty nodeName, don't consider pods marked unschedulable, etc.)
• must respect scheduler name annotation
• must not change nodeName if it is already written

[jdef]

Thanks for passing this along, David, as well as for summarizing the
discussion points from the SIG meeting.

On Wed, Dec 9, 2015 at 1:57 AM, David Oppenheimer notifications@github.com
wrote:

@jdef https://github.com/jdef I just remembered this issue; it's
related to your suggestion at the scheduling SIG meeting to have something
like a conformance test for new schedulers. We should probably just use
this issue.

Just for posterity, the three things we discussed at the meeting were

• must use the predicate library that Kubernetes uses (Move scheduler fit predicates into a library #12744
  Move scheduler fit predicates into a library #12744)
• must use standard set of non-predicate criteria Move scheduler fit predicates into a library #12744 (comment)
  Move scheduler fit predicates into a library #12744 (comment)
  (e.g. only consider pods with empty nodeName, don't consider pods marked
  unschedulable, etc.)
• must respect scheduler name annotation

—
Reply to this email directly or view it on GitHub
#17208 (comment)
.

[jdef]

/cc @timstclair I believe that you wanted to be copied on this

[timstclair]

@timothysc

[timothysc]

I'm guessing we're going to want to write an sdk library + documentation. Writing docs against the api itself for a scheduler will be error prone due to the constant subtle shifts in semantics.

[timothysc]

@davidopp are there work items we can complete as part of the 1.6 docs effort?

[davidopp]

I think our time would be better spent documenting how the default scheduler works. People who want to write their own scheduler will probably just copy the default scheduler and replace whatever parts they want to replace. So what they need is to understand how the default scheduler is structured, moreso than a "guide" to writing a new one.

[Yancey1989]

@davidopp +1

[spzala]

@davidopp hi David, since this seems a pure doc issue, should we close it here and open one against kubernetes.github.io to make it better visible to Doc team? - just a thought. Thanks!

[timothysc]

@spzala I'm ok with it. Could you open a docs issue and cross link to this one please.

[spzala]

@timothysc very sorry, totally missed your comment earlier, and yes absolutely. Thanks!!

[davidopp]

sure

[spzala]

@davidopp @timothysc Hi, I have created this doc issue kubernetes/website#4517 - hope it makes sense? Thanks!

[spzala]

@davidopp @timothysc Hi, just a gentle reminder :-) that per our earlier discussion can you close this issue as doc issue is opened to cover it now - kubernetes/website#4517 ? Thanks!

[spzala]

@davidopp Thanks !!