KEDA tries to resolve secrets from restricted namespace

[caedo960]

Report

I have KEDA up and running using KEDA_RESTRICT_SECRET_ACCESS environment variable which restricts secret access to only KEDA namespace. I have a rather simple ScaleObject (below) which has a target deployment and scale trigger using prometheus as a source. There is no TriggerAuthentication or any other reference to secrets in ScaledObject.

The target deployment (test-deployment) has some environment variables from secrets:

envFrom:
  - secretRef:
      name: test-deployment-89c6h5f565

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: test-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment 
    name: test-deployment
  pollingInterval: 30
  minReplicaCount: 2
  maxReplicaCount: 5
  advanced:
    restoreToOriginalReplicaCount: false
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 900
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.prometheus:9090/
      query: max by(test_tasks_queue)(test_tasks_pending_blocked{service="test-deployment", test_tasks_queue="default"})
      threshold: '100'
      activationThreshold: '0'

When I apply the ScaledObject to the cluster it fails to create the HPA resource for ScaledObject because it cannot find the secret what the deployment uses:

keda-operator  error resolving secrets for ScaleTarget: error reading secret ref &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:test-deployment-89c6h5f565,},Optional:nil,} on namespace default: secret "test-deployment-89c6h5f565" not found

keda-operator  Failed to ensure HPA is correctly created for ScaledObject

However if I allow KEDA to access secrets from all namespaces the ScaledObject and HPA are created succesfully and everything works fine.

Expected Behavior

KEDA should not need to access secrets from the namespace where the scale target is located if there is no reference to a secret in the ScaledObject or TriggerAuthentication is not used.

Actual Behavior

KEDA tries to list/watch/get secrets from the scale target namespace even though usage of secret(s) is not required from ScaledObject.

Steps to Reproduce the Problem

1. KEDA is installed using helm and the access to secrets is limited using the following parameters in values.yaml

permissions:
  metricServer:
    restrict:
      secret: true
  operator:
    restrict:
      secret: true

2. ScaledObject (described in the report section) is applied to the cluster
3. Creation of HPA resource for ScaledObject fails

Logs from KEDA operator

"level":"error","ts":"2023-05-03T10:57:16Z","msg":"Failed to ensure HPA is correctly created for ScaledObject","controller":"scaledobject","controllerGroup":"keda.sh","controllerKind":"ScaledObject","ScaledObject":{"name":"test-scaledobject","namespace":"default"},"namespace":"default","name":"test-scaledobject","reconcil │
eID":"7fc01087-cd26-479e-b31c-567dba2736e1","error":"error resolving secrets for ScaleTarget: error reading secret ref &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:test-deployment-89c6h5f565,},Optional:nil,} on namespace default: secret \"test-deployment-89c6h5f565\" not found","stacktrace":"github.com/kedacore/keda/v2/contro │
llers/keda.(*ScaledObjectReconciler).Reconcile\n\t/workspace/controllers/keda/scaledobject_controller.go:178\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconci │
leHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).St │
art.func2.2\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}
                                                                                                                                                                                                                                                             
{"level":"error","ts":"2023-05-03T10:57:16Z","msg":"Reconciler error","controller":"scaledobject","controllerGroup":"keda.sh","controllerKind":"ScaledObject","ScaledObject":{"name":"test-scaledobject","namespace":"default"},"namespace":"default","name":"test-scaledobject","reconcileID":"7fc01087-cd26-479e-b31c-567dba2736e1 │
","error":"error resolving secrets for ScaleTarget: error reading secret ref &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:test-deployment-89c6h5f565,},Optional:nil,} on namespace default: secret \"test-deployment-89c6h5f565\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconci │
leHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).St │
art.func2.2\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}

KEDA Version

2.10.0

Kubernetes Version

1.24

Platform

Google Cloud

Scaler Details

Prometheus

Anything else?

No response

[JorTurFer]

I think that we should skip the check here if the secrets are restricted to a namespace different from the SO namespace

keda/pkg/scaling/resolver/scale_resolvers.go

Lines 305 to 317 in c143d42

	} else if source.SecretRef != nil {
	secretsMap, err := resolveSecretMap(ctx, client, logger, source.SecretRef, namespace, secretsLister)
	switch {
	case err == nil:
	for k, v := range secretsMap {
	resolved[k] = v
	}
	case source.SecretRef.Optional != nil && *source.SecretRef.Optional:
	// ignore error when Secret is marked as optional
	continue
	default:
	return nil, fmt.Errorf("error reading secret ref %s on namespace %s: %w", source.SecretRef, namespace, err)
	}

[caedo960]

I think that we should skip the check here if the secrets are restricted to a namespace different from the SO namespace

keda/pkg/scaling/resolver/scale_resolvers.go

Lines 305 to 317 in c143d42

	} else if source.SecretRef != nil {
	secretsMap, err := resolveSecretMap(ctx, client, logger, source.SecretRef, namespace, secretsLister)
	switch {
	case err == nil:
	for k, v := range secretsMap {
	resolved[k] = v
	}
	case source.SecretRef.Optional != nil && *source.SecretRef.Optional:
	// ignore error when Secret is marked as optional
	continue
	default:
	return nil, fmt.Errorf("error reading secret ref %s on namespace %s: %w", source.SecretRef, namespace, err)
	}

Is it possible to implement your proposal?

[JorTurFer]

Hey,
Sure, are you willing to open a PR with the required changes?

[tobotg]

@JorTurFer I quickly looked into this issue, and I'm wondering if it could be sufficient to use optional field on the secretRef

envFrom:
  - secretRef:
      name: test-deployment-89c6h5f565
      optional: true

to ignore error when the secret is not found, instead of making code changes.

[JorTurFer]

I'm wondering if it could be sufficient to use optional field on the secretRef

Could be (I'm not sure), but it requires changes in how the workload is deployed and KEDA should be able to handle this security requirement internally, not asking for secrets outside the allowed namespaces. IMO the code should be changed at the end