Support Stop/Resume of KNative Services

[cl-gavan]

/area API
/area autoscale
/area networking
/kind spec

Describe the feature

We (Cloudera) are building an Application Serving platform on top of KNative, and our customers have asked for a feature to stop Applications at the end of their work day, and be able to resume the same Applications the next work day to save on compute costs overnight. While we could build this into our own platform directly, we think we might bring value into the KNative community by contributing this feature upstream.

The feature would work similarly to stop/resume of endpoints in KServe:

1. We propose adding a new annotation: "serving.knative.dev/stopped". When this is set to true on a KNative Service, KNative would stop routing traffic to that service, and allow it to scale down to 0. Any traffic to this service would return an error. (e.g., http.StatusServiceUnavailable)
2. A new "stopped" condition would need to be added to revision status.
3. Autoscaler would check for the new annotation and skip scaling decisions entirely if it is set to true: Don't scale up from zero, don't process metrics.
4. Networking/Route Controller would exclude stopped revisions from routing decisions. Don't include them in traffic splits even if they have traffic assigned.

We are willing to invest engineering effort into building and maintaining this feature, and would primarily need help with KNative architectural guidance and code reviews.

Also, as a side note, KServe has two deployment modes, and serverless mode (KNative mode) does not have the stop/resume endpoints feature. By adding this feature to KNative, we would also be adding value to KServe downstream as well.

[dprotaso]

I'm confused why they don't want to scale their workloads to 0? Can you elaborate on that?

[cl-gavan]

They do use scale to 0 currently. There are 2 reasons why stop/resume is useful on top of scale to 0:

1. Some customers wish to shut down services entirely regardless of traffic: e.g. for holidays or weekends.
2. Some customers are requesting using an autoscaler (KEDA, HPA) that can scale by CPU/Memory. These autoscalers do not support scale to 0, and so stop/resume would allow them to shutdown resources when they know there will be little to no workload.

[dprotaso]

Some customers wish to shut down services entirely regardless of traffic: e.g. for holidays or weekends.

Do you have an example why they want to do this?

Some customers are requesting using an autoscaler (KEDA, HPA) that can scale by CPU/Memory. These autoscalers do not support scale to 0, and so stop/resume would allow them to shutdown resources when they know there will be little to no workload.

I'm a bit confused by this comment since you're referring to non-knative autoscalers and then refer to the start/stop feature you want to add to Knative.

[dprotaso]

Any traffic to this service would return an error. (e.g., http.StatusServiceUnavailable)

Thinking about this more I don't think this generalizes well. Different workloads might want different semantics for their stopped workloads. eg. one returns a json response, another returns an HTML page if it's user facing etc.

I'm wondering if it's simpler to just create a special revision for each workload and when the service is paused we simply route traffic to that paused revision.

eg.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hello
  namespace: default
spec: 
  template: {...}
  traffic:
    - latestRevision: true
      percent: 0
    - revisionName: hello-paused-revision
      percent: 100

[dprotaso]

Alternatively Knative lets your have an independent route resource traffic split over different configurations.

serving/pkg/apis/serving/v1/route_types.go

Line 86 in 9e5c532

ConfigurationName string `json:"configurationName,omitempty"`

One shared configuration could be used for paused workloads. If you're exposing Knative Services to end-users though you probably woudn't want to do this as it's more advanced way of operating workloads

[dprotaso]

What I honestly think would be simpler is to just delete the Knative Service and recreate it when you want to serve traffic again.

[cl-gavan]

I don't think deleting/recreating is a viable option. We have enterprise customers that want to be able to:

• Rollback revisions
• Preserve configurations
• Maintain a history of metrics

A revision handling paused workloads makes sense, although we would need to make sure it doesn't get GC'd.

For a simpler solution, could we also temporarily set min/maxScale to 0, then restore min/maxScale on resume? It's not very graceful in handling traffic while the service is stopped, but it should work, right?

[dprotaso]

A revision handling paused workloads makes sense, although we would need to make sure it doesn't get GC'd.

We have a revision annotation for that "serving.knative.dev/no-gc":"true"

For a simpler solution, could we also temporarily set min/maxScale to 0, then restore min/maxScale on resume? It's not very graceful in handling traffic while the service is stopped, but it should work, right?

In the last release we added support for manually scaling the revision via annotations - #16186

So what you're suggesting could work but I would test out your scenario. The pods still exit gracefully so it shouldn't drop any traffic that's occurring - with the exception of some long lived connections. We wait TimeoutSeconds for the pod to exit (by setting terminatingGracePeriod).