Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to 2.10 broke deployments triggered by the same trigger type (rabbitmq) #4683

Closed
lesio999 opened this issue Jun 12, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@lesio999
Copy link

lesio999 commented Jun 12, 2023

Report

I have two separate deployment - api and svc, both communicating to each other via rabbitmq.
For both dedicated so and ta resources are created.
scaleObject are using dedicated names and scaleTargetRef, but the same trigger type rabbitmq, with the same metadata
metadata:
mode: QueueLength
value: '5'
queueName: que.request

Prior to updates to 2.10 in 2.6, that configuration worked without any issues, scaling both deployments.
With 2.10, that stopper working.
If only one deployment is enabled to use KEDA (only one so is in namespace, it works just fine).
When both deployments are enable, we see warnings with reason: AmbiguousSelector and FailedComputeMetricsReplicas
pods by selector one-ofour-label=xxxx are controlled by multiple HPAs: [a-namespace/keda-hpa-svc a-namespace/keda-hpa-api
No errors and no scaling.

Expected Behavior

the configuration to work as in 2.6 release

Actual Behavior

Warnings with reason: AmbiguousSelector and FailedComputeMetricsReplicas
pods by selector one-ofour-label=xxxx are controlled by multiple HPAs: [a-namespace/keda-hpa-svc a-namespace/keda-hpa-api
No errors and no scaling.

Logs from KEDA operator

example

KEDA Version

2.10.0

Kubernetes Version

1.26

Platform

Azure

Scaler Details

triggers:: -type: rabbitmq metadata: mode: QueueLength value: '5' queueName: que.request

Anything else?

No response

@lesio999 lesio999 added the bug Something isn't working label Jun 12, 2023
@JorTurFer
Copy link
Member

Hello,
That problem is not related with KEDA, it's a change that Kubernetes introduced in v1.26 for preventing multiple HPA scaling the same workloads: kubernetes/kubernetes#112011

Do you have more than 1 ScaledObject for the same workload? Or a ScaledObject and another external HPA? Other option could be a misconfiguration in labels/labelSelector at deployment level.

The PR I have linked explains both cases. I close this issue because it's not an issue in KEDA, but feel free to reopen it if you still think that there is something related with KEDA

@lesio999
Copy link
Author

Currently we do not have any other hpa resources besides the ones (2) created by keda out of so. I'll look at the labels/selectors to workaround the dups. if I need details/help, I'll open this ticket maybe with different type (not as a bug)

@JorTurFer
Copy link
Member

You can ask here again 😄 I just closed it for not having to enter for checking that it's something in k8s and not a bug, but we will try to help in any case

@lesio999
Copy link
Author

eliminating label's dups as workaround, resolved the issue. We still consider it as a bug in kubernetes and will be watching kubernetes/kubernetes#116898.

thanks for help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Ready To Ship
Development

No branches or pull requests

2 participants