Add complex scaling logic for custom formula & external scaling via grpc server #4583

gauron99 · 2023-05-29T12:08:03Z

UPDATE

Ive separated 2 functions in this PR into 2 and am keeping this as draft only for references as references

new feat: formula here
new feat: grpc servers as external calculations here

Summary

adds new structure in SO that describes how to modify/alter fetched metrics. User can describe new target to scale on within this structure. IF this is defined, new composite-scaler-metric is created and passed on to HPA (hpa.go). HPA will ask for only this one metric and internally all external metrics will be fetched and used in scale_handler.go.
2 new features within this struct are formula and external calculations (user-defined grpc server).

formula -> define formula with trigger names to modify metrics how ever you want using https://github.com/antonmedv/expr. Value returned from this will be attached to composite-scaler metric and returned as a final metric.
external calc -> user can defined grpc server and call it via url. New .proto file was created to match this. KEDA (as client) will try to connect to this server and pass its metrics via method Calculate.

one can chain multiple external calcs together when they define multiple. (its an array of grpc servers)

Order of execution: Fetch all external metrics from triggers -> external calcs execute in order (from top to bottom) as given in SO -> apply formula (you can use last name of grpc server if you want to manipulate its returned metric further)

Checklist

implement timeout & fallback for EC (e2e test)
create more test scenarios ( unit tests fallback)
create more test scenarios ( unit tests scale_handler)
admission webhook checkers
security for grpc connection
Changelog has been updated and is aligned with our changelog requirements
A PR is done to update the documentation on (docu-link)
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #3567
Fixes #2440

controllers/keda/hpa.go

pkg/externalscaling/client.go

pkg/scaling/scale_handler.go

carlreid · 2023-07-04T06:35:29Z

Hey, first of all, not to push you at all regarding the work on this PR. I am wondering what kind of priority these changes have, so I can see if I should wait for this to be released to use AND logic in KEDA, or look into some other potential solutions in the short term

gauron99 · 2023-07-04T15:58:34Z

@carlreid hello, Im working on this now daily. All functions are implemented, working on tests and some other specifics now to comply with PR standards. Im expecting to finish in few days then add few days for reviews etc.

carlreid · 2023-07-04T16:15:58Z

Hey @gauron99, thanks for the update! Then it makes sense for me to pause working on changes my side and use the great work you're doing here 👍

pkg/scaling/scale_handler.go

gauron99 · 2023-07-19T07:58:13Z

/run-e2e external_scaling*

gauron99

Sorry for this big PR, shouldve separated grpc servers into its own PR. Ive made some self-reviews and what i thought were important points

gauron99 · 2023-07-19T11:20:04Z

apis/keda/v1alpha1/scaledobject_types.go

+
+// ComplexScalingLogic describes advanced scaling logic options like formula
+// and gRPC server for external calculations
+type ComplexScalingLogic struct {


There was a point from @JorTurFer kedacore/keda-docs#1189 (comment) that this structure could use a better name - something like modifiers in yaml file. In such case i think it'd be good to change this one as well

gauron99 · 2023-07-19T11:20:31Z

apis/keda/v1alpha1/scaledobject_types.go

+// that KEDA can connect to with collected metrics and modify them. Each server
+// has a timeout and tls certification. If certDir is left empty, it will
+// connect with insecure.NewCredentials()
+type ExternalCalculation struct {


this struct as well

gauron99 · 2023-07-19T11:25:08Z

apis/keda/v1alpha1/scaledobject_types.go

@@ -141,6 +166,10 @@ type ScaledObjectStatus struct {
 	// +optional
 	ResourceMetricNames []string `json:"resourceMetricNames,omitempty"`
 	// +optional
+	CompositeScalerName string `json:"compositeScalerName,omitempty"`
+	// +optional
+	ExternalCalculationHealth map[string]HealthStatus `json:"externalCalculationHealth,omitempty"`


i created a new structure for grpc servers to have their own health status. Option number 2 could be to add them to already existing health status and prefix them with something like external-calculator or modifier depending on name of struct (see above). Or not prefix it at all and just keep it how it is but all with one health struct

gauron99 · 2023-07-19T11:28:08Z

apis/keda/v1alpha1/scaledobject_webhook.go

+
+// ValidateComplexScalingLogic validates all combinations of given arguments
+// and their values
+func ValidateComplexScalingLogic(so *ScaledObject, specs []autoscalingv2.MetricSpec) (float64, autoscalingv2.MetricTargetType, error) {


im not so sure these validation funcs should be in apis/keda/v1alpha/scaledobject_webhook.go but wasnt sure where to put them. there is dependency cycle if its in hpa.go (where the other call to this function is) because it imports kedav1alpha1

gauron99 · 2023-07-19T11:32:44Z

pkg/externalscaling/api/externalCalculation.proto

+
+message Response {
+	MetricsList list = 1;
+	string error = 2;


response possibly doesnt need error string because generated Calculate method is implemented with error return value by default

gauron99 · 2023-07-19T11:35:56Z

pkg/fallback/fallback.go

+const externalCalculatorStr string = "externalcalculator"
+
+// TODO: gauron99 - possible refactor this if trying to unify status updates & fallback functionality
+func isFallbackEnabled(scaledObject *kedav1alpha1.ScaledObject, metricSpec v2.MetricSpec, determiner string) bool {


The fallback functionality is also used to update status of health of metrics. I think this could be separated and refactored. I talked briefly with @zroubalik about this. Health status is updated multiple times (for each metric separate). I would propose locally changing the status via variable and only at the end of GetScaledObjectMetrics() in scale_handler.go it would be updated in SO in cluster. While its being manipulated with it would use the local variable. (If this is acceptable Id create an issue and like to implement this separately because this pr is too big already)

gauron99 · 2023-07-19T11:45:43Z

pkg/scaling/scale_handler.go

@@ -401,43 +451,55 @@ func (h *scaleHandler) ClearScalersCache(ctx context.Context, scalableObject int

 // GetScaledObjectMetrics returns metrics for specified metric name for a ScaledObject identified by its name and namespace.
 // It could either query the metric value directly from the scaler or from a cache, that's being stored for the scaler.
-func (h *scaleHandler) GetScaledObjectMetrics(ctx context.Context, scaledObjectName, scaledObjectNamespace, metricName string) (*external_metrics.ExternalMetricValueList, error) {
+func (h *scaleHandler) GetScaledObjectMetrics(ctx context.Context, scaledObjectName, scaledObjectNamespace, metricsName string) (*external_metrics.ExternalMetricValueList, error) {


this is renamed because when matched metric is found its assigned - metricName := spec.External.Metric.Name.