Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce topology into the runtimeClass API #75744

Merged
merged 9 commits into from May 31, 2019

Conversation

@yastij
Copy link
Member

commented Mar 26, 2019

What type of PR is this?

/kind api-change
/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes: Fixes #72413

Special notes for your reviewer:

/assign @tallclair

Does this PR introduce a user-facing change?:

NONE
@yastij

This comment has been minimized.

Copy link
Member Author

commented Mar 26, 2019

/sig scheduling
/sig node
/priority important-soon

@yastij yastij force-pushed the yastij:runtimeclass-scheduling-api branch 2 times, most recently from 212e4e5 to 352d9d5 Mar 26, 2019

@yastij yastij changed the title introduce topology into the runtimeClass API WIP: introduce topology into the runtimeClass API Mar 26, 2019

@yastij yastij force-pushed the yastij:runtimeclass-scheduling-api branch from 352d9d5 to d253246 Mar 26, 2019

@k82cn

This comment has been minimized.

Copy link
Member

commented Mar 27, 2019

any KEP or discussion about that?

@yastij

This comment has been minimized.

Copy link
Member Author

commented Mar 27, 2019

Ref kubernetes/enhancements#909

This is mostly a placeholder/wip as this is still on reviewv

@tallclair

This comment has been minimized.

Copy link
Member

commented Apr 4, 2019

Please update this to match the KEP (NodeSelectorTerm -> NodeSelector)

@yastij yastij force-pushed the yastij:runtimeclass-scheduling-api branch from d253246 to 8f5e8fd Apr 4, 2019

@k8s-ci-robot k8s-ci-robot added size/M and removed size/S labels Apr 4, 2019

@yastij yastij force-pushed the yastij:runtimeclass-scheduling-api branch 3 times, most recently from 84b7afc to bd2b7aa Apr 4, 2019

@k8s-ci-robot k8s-ci-robot added the size/L label Apr 19, 2019

@yastij yastij force-pushed the yastij:runtimeclass-scheduling-api branch from e00cf99 to 02288bc Apr 19, 2019

@yastij yastij force-pushed the yastij:runtimeclass-scheduling-api branch from b6685a9 to 3285db8 Apr 20, 2019

@yastij

This comment has been minimized.

Copy link
Member Author

commented Apr 20, 2019

/retest

@liggitt liggitt added this to Assigned in API Reviews Apr 20, 2019

tallclair and others added some commits Apr 24, 2019

@saad-ali
Copy link
Member

left a comment

Meeting Notes From the API review meeting today with @thockin

  • Why not something more generic? Maybe like a node capabilities struct (maybe on node resource), where pod requests node capability of "runtime windows?
    • Not considered in this context. Considered SchedulingPolicy context.
    • Don't want everything complicated (bespoke features) getting dumped on scheduler. In this case Scheduler will have new predicate for RuntimeClass.
  • RuntimeClass comes with list of tolerations. Why is that not sufficient for this scheduling?
    • Taint all windows nodes as windows? Because you could have overlap. Could have nodes run kata, gvisor, and some overlap that runC. Need tolerations for union.
  • Why is Node Selector not good enough?
    • If every pod had RuntimeClass and you have labels set up you don't need taints and tolerations. But you could make same argument for general taints and tolerations (if labels are set up on everything).
    • Taints and tolerations let you set default instead of just match or not match.
  • Why is it insufficient to label pod with node selector that says OS==Windows?
    • Original motivator in case of gvisor nodes -- don't want non-gvisor pods (non sandboxed) -- taint helps repel default pods.
  • If we had defaulting could get rid of tolerations?
    • Yes, all *Class types should have default. Worth looking in to how StorageClass does it.
  • Talking about adding scheduler logic to add selector to intersect with node selector?
    • Yes. Let us have a nice error message -- if you can't schedule due to what's on pod vs what's on runtime class.
  • Back to original question: why not demand make this more generic. Node capabilities, and make RuntimeClass in to a node capabilities. Worth thinking about.
    • Concept of error message is nice, but if we make it general, we don't need to do this again.
  • What is time line?
    • Hoping to get it in to 1.15 as beta. But we could call it alpha and put it behind a feature gate.
    • General solution may be not that much more work. Let's set up a follow up discussion with SIG-Node @dchen1107.
  • Why label selector on nodes -- instead of first class thing on nodes?
    • Using labels means smaller change. Requires you to put selector in RuntimeClass so you have to agree on what label is on node.
  • How does label get on node?
    • Part of node setup or config (whoever is responsible for configuring node does it).
    • Another reason, regular gvisor vs debug gvisor runtime. Add new runtime class, but don't want to update labels, want to reuse the same labels. "runc overhead" is the more interesting example.

Open questions:

  1. Generalize or not? Capability or not?
  2. Why use word topology -- it has lots of baggage?
    • Not tied to it, copied from StorageClass. Stay away from that word unless you mean physical structure of things, which this isn't representing.
@tallclair

This comment has been minimized.

Copy link
Member

commented May 22, 2019

Updated based on API review feedback, according to kubernetes/enhancements#1069

tallclair added some commits May 22, 2019

@tallclair tallclair force-pushed the yastij:runtimeclass-scheduling-api branch from 4d5a0cc to 2e38485 May 22, 2019

@thockin
Copy link
Member

left a comment

Thanks!

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm label May 31, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented May 31, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: thockin, yastij

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 6273a7a into kubernetes:master May 31, 2019

20 checks passed

cla/linuxfoundation tallclair authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-csi-serial Skipped.
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gce-storage-slow Skipped.
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details

@liggitt liggitt moved this from Assigned to Completed, 1.16 in API Reviews Jun 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.