Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ScaledObject with argo Rollout scaleTargetRef fails to reconcile #4344

Closed
golanpa opened this issue Mar 9, 2023 · 11 comments · Fixed by #5351
Closed

ScaledObject with argo Rollout scaleTargetRef fails to reconcile #4344

golanpa opened this issue Mar 9, 2023 · 11 comments · Fixed by #5351
Labels
feature-request All issues for new features that have not been committed to help wanted Looking for support from community stale All issues that are marked as stale due to inactivity

Comments

@golanpa
Copy link

golanpa commented Mar 9, 2023

Report

ScaledObject with Argo Rollout scaleTargetRef fails to reconcile in GKE and HPA resource is not created.

I am using Keda 2.9.3 (installed via https://kedacore.github.io/charts, version: 2.9.4).

Expected Behavior

HPA resource should be created for the ScaledObject.

Actual Behavior

Reconciliation fails with the following error and HPA resource is not created.

If a Deployment ref is used then the reconciliation succeeds, e.g:

scaleTargetRef
    apiVersion: apps/v1
    kind: Deployment
    name: archive-creator

Steps to Reproduce the Problem

Having an Argo Rollout resource named archive-creator use the following Keda resources:

ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: archive-creator-subscription
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    name: archive-creator
  minReplicaCount: 2
  maxReplicaCount: 200
  pollingInterval: 30
  cooldownPeriod: 180
  fallback:
    failureThreshold: 3
    replicas: 1
  triggers:
    - type: gcp-pubsub
      authenticationRef:
        name: keda-clustertrigger-auth-gcp-credentials
        kind: ClusterTriggerAuthentication
      metadata:
        subscriptionName: "projects/project/—–subscriptions/archive-creator"
        mode: "SubscriptionSize"
        metricType: "AverageValue"
        value: "16"

ClusterTriggerAuthentication:

apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: keda-clustertrigger-auth-gcp-credentials
spec:
  podIdentity:
    provider: gcp

Logs from KEDA operator

2023-03-08T14:53:47Z	ERROR	Failed to ensure HPA is correctly created for ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "67d2f6a9-9a9a-4ef4-b9fe-fa585da0c206", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}

KEDA Version

2.9.3

Kubernetes Version

None

Platform

Google Cloud

Scaler Details

gcp-pubsub

Anything else?

No response
@golanpa golanpa added the bug Something isn't working label Mar 9, 2023
@golanpa
Copy link
Author

golanpa commented Mar 9, 2023

some additional information. The failure is happening when the Argo Rollout is defined with a workloadRef:

workloadRef:
     apiVersion: apps/v1
     kind: Deployment
     name: archive-creator

There is no issue if the Argo Rollout defines a pod template within the Spec.
See for more information about workloadRef

@JorTurFer
Copy link
Member

Hello,
Could you share KEDA Operator logs?

@zroubalik
Copy link
Member

In the error message I can see: "error parsing PubSub metadata: GoogleApplicationCredentials not found" please check the scaler config

@golanpa
Copy link
Author

golanpa commented Mar 14, 2023

@zroubalik, i use the above ClusterTriggerAuthentication configuration. It works when I use Deployment, when I use Rollout with embedded pod spec. The error occurs only when I use Rollout with workloadRef.

Here is the problematic Rollout:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: archive-creator
  namespace: production
  labels:
    app: archive-creator
  annotations:
    notifications.argoproj.io/subscribe.on-rollout-step-completed.slack: wixmp-argo-rollouts-notifications
    notifications.argoproj.io/subscribe.on-rollout-completed.slack: wixmp-argo-rollouts-notifications
    notifications.argoproj.io/subscribe.on-rollout-aborted.slack: wixmp-argo-rollouts-notifications
spec:
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: archive-creator
  **workloadRef:
    apiVersion: apps/v1
    kind: Deployment
    name: archive-creator**
  strategy:
    canary:
      maxSurge: "25%"
      maxUnavailable: 0
      steps:
        - setWeight: 100
        - pause: { duration: 3m }
        - analysis:
            templates:
              - templateName: archive-creator-success-rate
                clusterScope: true
            args:
              - name: app
                value: archive-creator
              - name: namespace
                value: production

@golanpa
Copy link
Author

golanpa commented Mar 14, 2023

@JorTurFer , here are more logs:

2023-03-14T07:10:37Z	INFO	Creating a new HPA	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "ede9c889-660c-4af1-9304-a1a3d14146af", "HPA.Namespace": "production", "HPA.Name": "keda-hpa-archive-creator-subscription"}
2023-03-14T07:10:37Z	INFO	Initializing Scaling logic according to ScaledObject Specification	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "ede9c889-660c-4af1-9304-a1a3d14146af"}
2023-03-14T07:10:37Z	ERROR	scalehandler	error resolving auth params	{"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/workspace/pkg/scaling/scale_handler.go:543
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/workspace/pkg/scaling/scale_handler.go:269
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:190
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startPushScalers
	/workspace/pkg/scaling/scale_handler.go:313
2023-03-14T07:10:37Z	ERROR	scalehandler	Error getting scalers	{"type": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription", "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startPushScalers
	/workspace/pkg/scaling/scale_handler.go:315
2023-03-14T07:10:37Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "cb49de5f-ab9f-4f16-b4a6-4c3a516d5ec9"}
2023-03-14T07:10:37Z	ERROR	scalehandler	error resolving auth params	{"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/workspace/pkg/scaling/scale_handler.go:543
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/workspace/pkg/scaling/scale_handler.go:269
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:190
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:345
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:162
2023-03-14T07:10:37Z	ERROR	scalehandler	Error getting scalers	{"object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:347
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:162
E0314 07:10:37.818060       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"archive-creator-subscription.174c378c8741b21e", GenerateName:"", Namespace:"production", SelfLink:"", UID:"", ResourceVersion:"71837", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ScaledObject", Namespace:"production", Name:"archive-creator-subscription", UID:"4976322f-917f-44c7-b518-48542c1d5c6f", APIVersion:"keda.sh/v1alpha1", ResourceVersion:"34125421", FieldPath:""}, Reason:"KEDAScalerFailed", Message:"error parsing PubSub metadata: GoogleApplicationCredentials not found", Source:v1.EventSource{Component:"keda-operator", Host:""}, FirstTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 0, time.Local), LastTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 813298441, time.Local), Count:2, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "archive-creator-subscription.174c378c8741b21e" is forbidden: User "system:serviceaccount:kedacore:keda-operator" cannot patch resource "events" in API group "" in the namespace "production"' (will not retry!)
2023-03-14T07:10:37Z	ERROR	scalehandler	error resolving auth params	{"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/workspace/pkg/scaling/scale_handler.go:543
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/workspace/pkg/scaling/scale_handler.go:269
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:190
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:200
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z	ERROR	Error getting scalers	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "cb49de5f-ab9f-4f16-b4a6-4c3a516d5ec9", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:202
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z	ERROR	Failed to create new HPA resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "cb49de5f-ab9f-4f16-b4a6-4c3a516d5ec9", "HPA.Namespace": "production", "HPA.Name": "keda-hpa-archive-creator-subscription", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:154
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z	ERROR	Failed to check HPA for possible update	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "cb49de5f-ab9f-4f16-b4a6-4c3a516d5ec9", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:429
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z	ERROR	Failed to ensure HPA is correctly created for ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "cb49de5f-ab9f-4f16-b4a6-4c3a516d5ec9", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:179
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
E0314 07:10:37.822198       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"archive-creator-subscription.174c378c8741b21e", GenerateName:"", Namespace:"production", SelfLink:"", UID:"", ResourceVersion:"71837", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ScaledObject", Namespace:"production", Name:"archive-creator-subscription", UID:"4976322f-917f-44c7-b518-48542c1d5c6f", APIVersion:"keda.sh/v1alpha1", ResourceVersion:"34125421", FieldPath:""}, Reason:"KEDAScalerFailed", Message:"error parsing PubSub metadata: GoogleApplicationCredentials not found", Source:v1.EventSource{Component:"keda-operator", Host:""}, FirstTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 0, time.Local), LastTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 819924869, time.Local), Count:3, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "archive-creator-subscription.174c378c8741b21e" is forbidden: User "system:serviceaccount:kedacore:keda-operator" cannot patch resource "events" in API group "" in the namespace "production"' (will not retry!)
2023-03-14T07:10:37Z	ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "cb49de5f-ab9f-4f16-b4a6-4c3a516d5ec9", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "9778841f-dec2-4332-9321-aeeb94e1fa6c"}
2023-03-14T07:10:37Z	ERROR	scalehandler	error resolving auth params	{"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/workspace/pkg/scaling/scale_handler.go:543
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/workspace/pkg/scaling/scale_handler.go:269
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:190
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:200
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z	ERROR	Error getting scalers	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "9778841f-dec2-4332-9321-aeeb94e1fa6c", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:202
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z	ERROR	Failed to create new HPA resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "9778841f-dec2-4332-9321-aeeb94e1fa6c", "HPA.Namespace": "production", "HPA.Name": "keda-hpa-archive-creator-subscription", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:154
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z	ERROR	Failed to check HPA for possible update	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "9778841f-dec2-4332-9321-aeeb94e1fa6c", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:429
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z	ERROR	Failed to ensure HPA is correctly created for ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "9778841f-dec2-4332-9321-aeeb94e1fa6c", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:179
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
E0314 07:10:37.852262       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"archive-creator-subscription.174c378c8741b21e", GenerateName:"", Namespace:"production", SelfLink:"", UID:"", ResourceVersion:"71837", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ScaledObject", Namespace:"production", Name:"archive-creator-subscription", UID:"4976322f-917f-44c7-b518-48542c1d5c6f", APIVersion:"keda.sh/v1alpha1", ResourceVersion:"34126099", FieldPath:""}, Reason:"KEDAScalerFailed", Message:"error parsing PubSub metadata: GoogleApplicationCredentials not found", Source:v1.EventSource{Component:"keda-operator", Host:""}, FirstTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 0, time.Local), LastTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 849263448, time.Local), Count:4, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "archive-creator-subscription.174c378c8741b21e" is forbidden: User "system:serviceaccount:kedacore:keda-operator" cannot patch resource "events" in API group "" in the namespace "production"' (will not retry!)
E0314 07:10:37.854537       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"archive-creator-subscription.174c378c881519bb", GenerateName:"", Namespace:"production", SelfLink:"", UID:"", ResourceVersion:"71838", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ScaledObject", Namespace:"production", Name:"archive-creator-subscription", UID:"4976322f-917f-44c7-b518-48542c1d5c6f", APIVersion:"keda.sh/v1alpha1", ResourceVersion:"34126099", FieldPath:""}, Reason:"ScaledObjectCheckFailed", Message:"Failed to ensure HPA is correctly created for ScaledObject", Source:v1.EventSource{Component:"keda-operator", Host:""}, FirstTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 0, time.Local), LastTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 849420970, time.Local), Count:2, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "archive-creator-subscription.174c378c881519bb" is forbidden: User "system:serviceaccount:kedacore:keda-operator" cannot patch resource "events" in API group "" in the namespace "production"' (will not retry!)
2023-03-14T07:10:37Z	ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "9778841f-dec2-4332-9321-aeeb94e1fa6c", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "c9d93f9d-f876-4bfb-a065-8b614f758c8d"}
2023-03-14T07:10:37Z	ERROR	scalehandler	error resolving auth params	{"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/workspace/pkg/scaling/scale_handler.go:543
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/workspace/pkg/scaling/scale_handler.go:269
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:190
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:200
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z	ERROR	Error getting scalers	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "c9d93f9d-f876-4bfb-a065-8b614f758c8d", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}

@zroubalik
Copy link
Member

zroubalik commented Mar 14, 2023 •
edited
Loading

@golanpa gotcha, the problem is that workloadRef is not supported at the moment, you need to use Rollout with podSpec

@golanpa
Copy link
Author

golanpa commented Mar 14, 2023

@zroubalik, thanks for the insight. Is there a request for that? should we transition this "bug" to a feature request?

@zroubalik zroubalik added feature-request All issues for new features that have not been committed to help wanted Looking for support from community and removed bug Something isn't working labels Mar 14, 2023
@zroubalik
Copy link
Member

@golanpa did that. Are you willing to contribute this?

@golanpa
Copy link
Author

golanpa commented Mar 15, 2023

@zroubalik , i will try to take a look at that

@sbdtu5498 sbdtu5498 removed their assignment Jun 17, 2023
@stale
Copy link

stale bot commented Aug 18, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale All issues that are marked as stale due to inactivity label Aug 18, 2023
@stale
Copy link

stale bot commented Sep 3, 2023

This issue has been automatically closed due to inactivity.

@stale stale bot closed this as completed Sep 3, 2023
@JorTurFer JorTurFer mentioned this issue Jan 7, 2024
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature-request All issues for new features that have not been committed to help wanted Looking for support from community stale All issues that are marked as stale due to inactivity
Projects
Roadmap - KEDA Core
Status: Ready To Ship
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix pod identity ignored when scaled target is CRD wozniakjan/keda
4 participants
@zroubalik @golanpa @JorTurFer @sbdtu5498