Adding 1->n and n->1 scaling for Queue scaler #47

Aarthisk · 2019-03-21T22:34:54Z

Adding HPA scaling for Queues.
Changes include

Creating a HPA when a scaled object is synced
Deleting a HPA when a scaled object is deleted
Additional logic to not scale to zero immediately
Scaler interface changes to get the new metric definitions and metrics
Metric adapter that implements Metrics API for HPA to query and scale on external metrics
/deploy has the deployment script for the metrics adapter

…e each other.

yaron2 · 2019-03-21T22:37:33Z

Excited to see this!

yaron2 · 2019-03-21T22:44:22Z

Once #41 is merged, will you need to update based on the cool-down period work?

ahmelsayed · 2019-03-22T19:25:25Z

btw, I can't build this PR. I thought the CI error was a test error, but looking at it again, the CI is also failing to build. The error is


vendor/github.com/emicklei/go-restful/container.go:17:2: cannot find package "github.com/emicklei/go-restful/log" in any of:
        /home/ahmed/go/src/github.com/Azure/Kore/vendor/github.com/emicklei/go-restful/log (vendor tree)
        /usr/lib/go-1.10/src/github.com/emicklei/go-restful/log (from $GOROOT)
        /home/ahmed/go/src/github.com/emicklei/go-restful/log (from $GOPATH)
vendor/k8s.io/apiserver/pkg/server/options/audit.go:38:2: cannot find package "k8s.io/apiserver/plugin/pkg/audit/log" in any of:
        /home/ahmed/go/src/github.com/Azure/Kore/vendor/k8s.io/apiserver/plugin/pkg/audit/log (vendor tree)
        /usr/lib/go-1.10/src/k8s.io/apiserver/plugin/pkg/audit/log (from $GOROOT)
        /home/ahmed/go/src/k8s.io/apiserver/plugin/pkg/audit/log (from $GOPATH)

I tried running dep ensure and it's been running for about 10 minutes now.....

ahmelsayed · 2019-03-22T19:25:55Z

also there is a status.png file that I don't think you meant to commit

.vscode/launch.json

cmd/apiserver.local.config/certificates/apiserver.crt

cmd/main.go

pkg/controller/controller.go

pkg/handler/scale_handler.go

ahmelsayed · 2019-03-22T19:06:24Z

pkg/handler/scale_handler.go

+	newhpa := &v2beta1.HorizontalPodAutoscaler{Spec: *newHpaSpec, ObjectMeta: *hpaObjectSpec}
+	newhpa, err = h.kubeClient.AutoscalingV2beta1().HorizontalPodAutoscalers(scaledObjectNamespace).Create(newhpa)
+	if errors.IsAlreadyExists(err) {
+		log.Warnf("HPA with namespace %s and name %s already exists", scaledObjectNamespace, hpaName)


I wonder if we need to have an "upgrade" mechanism once people update kore's version, so that we can change the HPA scaling logic on kore updates

possibly- I can see us updating the max replicas when the scaled object overrides it for example.

pkg/handler/scale_handler.go

pkg/provider/provider.go

Aarthisk · 2019-03-22T20:44:30Z

also there is a status.png file that I don't think you meant to commit

I did not. Removed.

Aarthisk · 2019-03-22T20:54:53Z

also there is a status.png file that I don't think you meant to commit

Yes, removed.

Aarthisk · 2019-03-22T20:57:11Z

removed

Aarthisk · 2019-03-22T22:39:26Z

Once #41 is merged, will you need to update based on the cool-down period work?

This is now removed.

ahmelsayed · 2019-03-25T23:38:56Z

pkg/handler/scale_handler.go

@@ -39,9 +43,100 @@ func NewScaleHandler(koreClient clientset.Interface, kubeClient kubernetes.Inter

 // WatchScaledObjectWithContext enqueues the ScaledObject into the work queue
 func (h *ScaleHandler) WatchScaledObjectWithContext(ctx context.Context, scaledObject *kore_v1alpha1.ScaledObject) {
+	h.createHPAForNewScaledObject(ctx, scaledObject)


This can be called asynchronously as well i think.

Also keep in mind that this is called on every scaledobject update. That can be frequent

It could be called asynchronously but I don't think it should.
The WatchScaledObjectWithContext itself is executed on a separate goroutine from the get go, no?

pkg/handler/scale_handler.go

pkg/adapter/adapter.go

yaron2 · 2019-03-26T20:13:23Z

pkg/handler/scale_handler.go

@@ -39,9 +43,100 @@ func NewScaleHandler(koreClient clientset.Interface, kubeClient kubernetes.Inter

 // WatchScaledObjectWithContext enqueues the ScaledObject into the work queue
 func (h *ScaleHandler) WatchScaledObjectWithContext(ctx context.Context, scaledObject *kore_v1alpha1.ScaledObject) {
+	h.createHPAForNewScaledObject(ctx, scaledObject)


It could be called asynchronously but I don't think it should.
The WatchScaledObjectWithContext itself is executed on a separate goroutine from the get go, no?

pkg/handler/scale_handler.go

pkg/scalers/azure_queue_scaler.go

yaron2 · 2019-03-26T20:23:07Z

Review submitted.

… getScalers

…rthisk-work

pkg/adapter/adapter.go

yaron2 · 2019-03-27T16:43:30Z

Please resolve conflicts.

…work # Conflicts: # .circleci/config.yml # pkg/scalers/azure_queue.go # pkg/scalers/azure_queue_scaler.go

yaron2 · 2019-03-27T17:39:08Z

LGTM

Adding 1->n and n->1 scaling for Queue scaler Former-commit-id: 5e4e37e

Adding 1->n and n->1 scaling for Queue scaler Former-commit-id: 0e2b157

Aarthisk added 8 commits March 15, 2019 14:46

HPA scaling

8ca161a

adding cluster admin access to HPA service account

d05fd86

delete HPA on scaled object removal.

6b2753f

removing extraneous copy -pastes.

eb1c5b5

dealing with race when scaledecision and HPA replicacount can overrid…

848585c

…e each other.

changed namespace to Kore

94bb0f1

Comments and changing to use new docker file + https for Q scaling.

62626f5

removing unused files.

71e4bca

Aarthisk requested review from ahmelsayed and yaron2 March 21, 2019 22:34

ahmelsayed requested changes Mar 22, 2019

View reviewed changes

Aarthisk closed this Mar 22, 2019

Aarthisk reopened this Mar 22, 2019

Aarthisk and others added 9 commits March 22, 2019 21:05

code review comments

0479850

adding cluster admin access to HPA service account

316df65

code review comments and refactoring.

36fb9f1

Adding the new adapter file.

b9e9f0e

Missing vendor files

c48fcae

toml and lock changes

d354902

adding more vendor files

28554a2

comment out make test

f03159a

Merge branch 'master' into aarthisk-work

89a6155

ahmelsayed reviewed Mar 25, 2019

View reviewed changes

pkg/handler/scale_handler.go Outdated Show resolved Hide resolved

ahmelsayed reviewed Mar 25, 2019

View reviewed changes

pkg/handler/scale_handler.go Outdated Show resolved Hide resolved

yaron2 suggested changes Mar 26, 2019

View reviewed changes

yaron2 mentioned this pull request Mar 26, 2019

Use Operator SDK #54

Closed

Aarthisk added 2 commits March 26, 2019 15:14

Refactoring adapter creation, Yaron's other comments and implementing…

9b45d71

… getScalers

Merge branch 'aarthisk-work' of https://github.com/Azure/Kore into aa…

7c0870b

…rthisk-work

Aarthisk requested review from ahmelsayed and yaron2 March 26, 2019 22:26

yaron2 reviewed Mar 26, 2019

View reviewed changes

pkg/adapter/adapter.go Show resolved Hide resolved

Merge branch 'master' of https://github.com/Azure/Kore into aarthisk-…

bcbcbb8

…work # Conflicts: # .circleci/config.yml # pkg/scalers/azure_queue.go # pkg/scalers/azure_queue_scaler.go

yaron2 approved these changes Mar 27, 2019

View reviewed changes

yaron2 merged commit 5e4e37e into master Mar 27, 2019

yaron2 deleted the aarthisk-work branch March 27, 2019 17:41

patnaikshekhar pushed a commit to patnaikshekhar/keda that referenced this pull request Jul 17, 2019

Merge pull request kedacore#47 from Azure/aarthisk-work

2c3dedf

Adding 1->n and n->1 scaling for Queue scaler Former-commit-id: 5e4e37e

Aarthisk pushed a commit that referenced this pull request Aug 12, 2019

Merge pull request #47 from Azure/aarthisk-work

2b0593a

Adding 1->n and n->1 scaling for Queue scaler Former-commit-id: 0e2b157

preflightsiren pushed a commit to preflightsiren/keda that referenced this pull request Nov 7, 2021

removing func install (kedacore#47)

8ec878f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding 1->n and n->1 scaling for Queue scaler #47

Adding 1->n and n->1 scaling for Queue scaler #47

Aarthisk commented Mar 21, 2019

yaron2 commented Mar 21, 2019

yaron2 commented Mar 21, 2019 •

edited

ahmelsayed commented Mar 22, 2019

ahmelsayed commented Mar 22, 2019

ahmelsayed Mar 22, 2019

Aarthisk Mar 26, 2019

Aarthisk commented Mar 22, 2019

Aarthisk commented Mar 22, 2019

Aarthisk commented Mar 22, 2019

Aarthisk commented Mar 22, 2019

ahmelsayed Mar 25, 2019

ahmelsayed Mar 25, 2019

yaron2 Mar 26, 2019

yaron2 Mar 26, 2019

yaron2 commented Mar 26, 2019

yaron2 commented Mar 27, 2019

yaron2 commented Mar 27, 2019

Adding 1->n and n->1 scaling for Queue scaler #47

Adding 1->n and n->1 scaling for Queue scaler #47

Conversation

Aarthisk commented Mar 21, 2019

yaron2 commented Mar 21, 2019

yaron2 commented Mar 21, 2019 • edited

ahmelsayed commented Mar 22, 2019

ahmelsayed commented Mar 22, 2019

ahmelsayed Mar 22, 2019

Choose a reason for hiding this comment

Aarthisk Mar 26, 2019

Choose a reason for hiding this comment

Aarthisk commented Mar 22, 2019

Aarthisk commented Mar 22, 2019

Aarthisk commented Mar 22, 2019

Aarthisk commented Mar 22, 2019

ahmelsayed Mar 25, 2019

Choose a reason for hiding this comment

ahmelsayed Mar 25, 2019

Choose a reason for hiding this comment

yaron2 Mar 26, 2019

Choose a reason for hiding this comment

yaron2 Mar 26, 2019

Choose a reason for hiding this comment

yaron2 commented Mar 26, 2019

yaron2 commented Mar 27, 2019

yaron2 commented Mar 27, 2019

yaron2 commented Mar 21, 2019 •

edited