KEP-3990: PodTopologySpread DoNotSchedule-to-ScheduleAnyway fallback mode

[sanposhiho]

• One-line PR description: PodTopologySpread DoNotSchedule-to-ScheduleAnyway fallback mode

• Issue link: PodTopologySpread DoNotSchedule-to-ScheduleAnyway fallback mode #3990

• Other comments:

[sanposhiho]

/assign @alculquicondor @MaciekPytel
/sig scheduling autoscaling

[sanposhiho]

It's a design similar to what we discussed in the issue. But, I switched this to fallbackCriteria for the reason described here. Open for discussion about which design we should choose.

[sanposhiho]

@alculquicondor What do you think?

[sanposhiho]

@wojtek-t I assigned it to you as other sig-scheduling KEPs do.
But, please reassign it to other people if needed. 🙏

[wojtek-t]

Just quickly skimmed through it - I will reviewer deeper when you will have buy-in from the SIG.

[wojtek-t]

Just quickly skimmed through it - I will reviewer deeper when you will have buy-in from the SIG.

[wojtek-t]

I don't understand it - can you clarify?

[sanposhiho]

For example, let's say PodA has a required Pod Topology Spread with fallbackCriteria: ScaleUpFailed. When PodA's first scheduling, all Nodes went through Pod Topology Spread filter plugin, but they were rejected by other plugins (e.g., the resource fit filter plugin because of insufficient resources.), and PodA got unschedulable state as a result. Then, the cluster autoscaler tried to create a new Node for PodA but it couldn't make it because of stockout.

In this case, Pod Topology Spread doesn't need to fallback actually because it said OK to all Nodes in the previous scheduling cycle -- meaning Pod Topology Spread isn't the cause of unschedulable in the past scheduling cycle and the fallback won't make PodA schedulable.
But, the problem is that Pod Topology Spread cannot see why this Pod was rejected in the previous scheduling cycle. It only understands that PodA was rejected by someone in the previous scheduling, and the cluster autoscaler couldn't create a new Node.

[a7i]

nit:

Suggested change

	title: Pod Topology Spread DoNotSchedule to SchedulingAnyway fallback mode
	title: Pod Topology Spread DoNotSchedule to ScheduleAnyway fallback mode

[alculquicondor]

/cc @mwielgus

[a7i]

Could it not simply default to FailedScheduling. As far as I know, ScaleUpFailed is a cluster-autoscaler specific event so for those who use something else (e.g. karpenter), then this will not work.

One challenge with FailedScheduling is that's what cluster-autoscaler reacts on so there could be a race condition in cluster-autoscaler scaling up vs. switching to fallback mode.

[sanposhiho]

This part only means that ScaleUpFailed is a new value that will be added to fallbackCriteria. Yes, I took this name from the cluster autoscaler's event name, but it should work with any cluster autoscaler if they update TriggeredScaleUp Pod condition, which is also proposed in this KEP.

[alculquicondor]

We had too much to bite in this cycle. This will have to wait for the next release :(

Try to get some input from SIG Autoscaling in one of their meetings.

[sanposhiho]

Yes..

Try to get some input from SIG Autoscaling in one of their meetings.

Sure.

[ellistarn]

Curious if there's an analogous use case for:

• Preferential node affinity
• Preferential pod affinity/antiaffinity

Is there any world where a user may want to have more control over how preferences are handled with autoscalers?

[sanposhiho]

if there's an analogous use case for:

Yes, might be. We would wait for the actual usecases from users before implementing the similar stuff in them though, we should make this design to be easy to follow in other scheduling constants

[ellistarn]

This makes me think that we might want to model fallback criteria at the pod level.

preferentialFallback:
  - PodAffinity
  - AntiAffinity
  - TopologySpreadConstraints

It may be difficult to get the granularity right, though. At another extreme (as you propose), you'd add a preference policy for each preferred term across the pod spec.

[sanposhiho]

That's an interesting idea.
Because when people want to use fallback, they may want to do the same in all required scheduling constraints the Pod has. Also, we may want to give preferentialFallback the way to "fallback in all".

So, maybe it'd be like this:

preferentialFallback:
  schedulingConstraints: 
  - "*" # do fallback in all scheduling constraints.
  # You can also specify individual scheduling constraints name, which supports fallback.
  # (But, note that this KEP's scope is only TopologySpread.)
  fallBackCriteria:
  - ScaleUpFailed

Let me include this idea in the alternative section for now. Like the idea of introducing enums, we can gather feedback from more people about the design choice among them.

[ellistarn]

There are some benefits to reusing this an existing field with new enum values. I'm a touch concerned with how complicated topologyspreadconstraints are getting, with two new honor fields (with different defaults).

I interact with a bunch of customers who are learning kubernetes scheduling for the first time when onboarding to Karpenter, and communicating the complexity and gotchas of topology is always a huge hurdle to get over.

[sanposhiho]

Obviously, I agree that we need to make it easy to understand as much as possible. But, also introducing a new field doesn't always result in confusion.
For this particular KEP, I think FallBackCriteria is not much more complicated than introducing new enum values in WhenUnsatisfiable. Then, currently I'm leaning towards FallBackCriteria for the ease to introduce it into other scheduling constraints, as described.
But, maybe I'm biased since I know the scheduler/CA, I know everything that this KEP wants to do, etc. 😅
So, I'd like to keep this discussion open to gather more feedback from reviewers.

[knelasevero]

I agree that topologyspreadconstraints are difficult to explain to clients (been there), but I think this added field is optional complexity. It is also the other way around, right? It is a complex solution to a complex setup, but the complex setup came first. Clients that need to hear this are already dealing with specific cases that call for all of this, and in theory are easier to talk to. Also agree that I don't see it as easier to explain when considering existing field with new enum values, so I guess that if it is hard to explain, it would be hard either way.

[jsoref]

apparently i wrote this a while ago...

[jsoref]

Suggested change

	It can contain two values: `ScaleUpFailed` to fallback when the cluster autoscaler fails to create new Node for Pods,
	It can contain two values: `ScaleUpFailed` to fall back when the cluster autoscaler fails to create new Node for Pods,

[jsoref]

Suggested change

	and `PreemptionFailed` to fallback when the preemption doesn't help to make Pods schedulable.
	and `PreemptionFailed` to fall back when the preemption doesn't help to make Pods schedulable.

[jsoref]

Suggested change

	and has some low-priority Pods to make space (often called overprovisional Pods, balloon Pods, etc).
	and has some low-priority Pods to make space (often called overprovisional Pods, balloon Pods, etc.).

[jsoref]

Suggested change

	so that Pod Topology Spread can decide to fallback only when the Pod was rejected by Pod Topology Spread.
	so that Pod Topology Spread can decide to fall back only when the Pod was rejected by Pod Topology Spread.

[jsoref]

Suggested change

	// FallbackCriterion represents when the scheduler fallbacks from the required scheduling constraint to the preferred one.
	// FallbackCriterion represents when the scheduler falls back from the required scheduling constraint to the preferred one.

[jsoref]

Suggested change

	// FallbackCriteria is the list of criteria that the scheduler decides when to fallback from DoNotSchedule to ScheduleAnyway.
	// It's valid to set only when WhenUnsatisfiable is DoNotSchedule.
	// If multiple criteria are in this list, the scheduler fallbacks when ALL criteria in `FallbackCriterion` are satisfied.
	// It's an optional field. The default value is nil, meaning the scheduler never fallbacks.
	// FallbackCriteria is the list of criteria that the scheduler decides when to fall back from DoNotSchedule to ScheduleAnyway.
	// It's valid to set only when WhenUnsatisfiable is DoNotSchedule.
	// If multiple criteria are in this list, the scheduler falls back when ALL criteria in `FallbackCriterion` are satisfied.
	// It's an optional field. The default value is nil, meaning the scheduler never falls back.

[sanposhiho]

Describing the current situation here:

I brought this KEP into SIG/Autoscaling weekly meeting a couple of months ago, and concluded that we had to consider how CA set TriggeredScaleUp: false reliably.

[maciekpytel] - One comment, in general, being unable to trigger scale-up is one failure scenario. However CA can also trigger a scaleup but fail to get capacity, for cloudprovider reasons etc.

• We’d need to reliably set the status in these conditions, not sure if the CA is able to do this today.
  Could end up in a situation where the CA keeps trying other node groups
• Will talk internally with other CA maintainers and come back with someone best able to commit to feedback on this.

Weekly meeting note: https://docs.google.com/document/d/1RvhQAEIrVLHbyNnuaT99-6u9ZUMp7BfkPupT2LAZK7w/edit#bookmark=id.oxf7zvbyd9ya

Not sure if we'll make it in this release though, I'll spend some time on this KEP in this release cycle as well - I'll go around CA's implementation to understand scenarios there myself, and elaborate the KEP around there. Thanks for several feedback so far, I'll bring the fixes towards all at the same time.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sanposhiho
Once this PR has been reviewed and has the lgtm label, please ask for approval from alculquicondor, wojtek-t. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS
• keps/sig-scheduling/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@sanposhiho: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-enhancements-verify	ea3fc17	link	true	/test pull-enhancements-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[sanposhiho]

/remove-lifecycle stale