ScaledObject status Unknown

[silasfrigo]

Report

Our scaledObject status shows unknow

n

And keda logs shows the following:

{
   "ScaledObject.Namespace":"channels-integrations",
   "ScaledObject.Name":"ch-int-mercadolivre-orders-inbound-scaledobject",
   "error":"error getting scaler for trigger #0: error creating kafka client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"
}

Expected Behavior

ScaledObject to show status and complete hpa creation.

Actual Behavior

Status stays in unknown.

Steps to Reproduce the Problem

1. Keda version 2.4
2. Create a TriggerAuthentication and a ScaledObject file
3. Deploy and apply changes.

Logs from KEDA operator

{
   "ScaledObject.Namespace":"channels-integrations",
   "ScaledObject.Name":"ch-int-mercadolivre-orders-inbound-scaledobject",
   "error":"error getting scaler for trigger #0: error creating kafka client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"
}

KEDA Version

2.4.0

Kubernetes Version

No response

Platform

Other

Scaler Details

Kafka

Anything else?

We are using confluent cloud.

Keda operator logs:

{
   "ScaledObject.Namespace":"channels-integrations",
   "ScaledObject.Name":"ch-int-mercadolivre-orders-inbound-scaledobject",
   "error":"error getting scaler for trigger #0: error creating kafka client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"
}

And ScaledObject status show 'Unknown' state.

keda version 2.4

apiVersion: v1
kind: Secret
metadata:
  name: keda-kafka-secrets
  namespace: my-namespace
data:
  sasl: "plaintext as b64"
  username: "username as b64"
  password: "password as b64"
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-kafka-credential
  namespace: my-namespace
spec:
  secretTargetRef:
  - parameter: sasl
    name: keda-kafka-secrets
    key: sasl
  - parameter: username
    name: keda-kafka-secrets
    key: username
  - parameter: password
    name: keda-kafka-secrets
    key: password
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-scaled-object-name
  namespace: my-namespace
spec:
  scaleTargetRef:
    name: my-deployment-name
  cooldownPeriod: 30
  minReplicaCount: 0
  maxReplicaCount: 3
  pollingInterval: 10
  triggers:
  - type: kafka
    metadata:
      metricName: my-metric-name
      namespace: my-namespace
      bootstrapServers: our-broker-address.confluent.cloud:9092
      consumerGroup: consumer_group
      topic: my_topic
      lagThreshold: "10"
      offsetResetPolicy: earliest
    authenticationRef:
      name: keda-trigger-auth-kafka-credential

[odilonjk]

Hi!

It might be relevant to say: our applications running on this k8s cluster communicate with the kafka brokers without any problem. So it's not a network problem.

[odilonjk]

BTW, the account that we're using to test Keda auto-scale, has full access on Kafka cluster, so it also doesn't seem to be a permission problem.

[zroubalik]

What Kafka Broker version are you trying to connect to? We are using https://github.com/Shopify/sarama Go client for connection. You might want to tweak the client version: https://keda.sh/docs/2.4/scalers/apache-kafka/#trigger-specification

[odilonjk]

Hi @zroubalik
We already tried to tweak the kafka version, however without success.

We also don't know exactly how version is it running on Kafka, because it's on Confluent Cloud and they don't let the customers know what version is it (usually it's the latest version, as they support told me).

@silasfrigo also made a downgrade to Keda 1.5 and it worked as expected. But the problem appeared on Keda 2.0 and 2.4.

[silasfrigo]

Hi @zroubalik. What bothers me is that in version 1.x we did not have to provide kafka brokers version. Why is that necessary now? And if it really is necessary, why is the field optional, with a default value? Just trying to figure out if this can really be our problem.
Thank you!

[zroubalik]

The version is not necessary, it is only if you would like to change it. Majority of users should be able to connect without tweaking it.

That's strange, are you able to build a simple app (consumer) with the Sarama client and try the connection to the Kafka Broker, to see whether the problem is there?

Btw in KEDA v1, there's older version of Sarama client, so that might be the problem, your Broker might be some older version as well 🤷‍♂️ have you tried to set the same Sarama version that's been used in v1?

Hard to tell.

[odilonjk]

I don't think the broker is running an old version, based on the Confluent support answer:

The version of Kafka running in Confluent Cloud is not published, because we deploy updates frequently, and the actual version may not align exactly with a release version. You can generally assume that your Cloud cluster(s) are running the latest available version or newer (currently Apache Kafka 2.8.0/Confluent 6.2.0). Clients and brokers are backward and forward compatible with each other, so the actual version typically shouldn't matter. For additional explanation, see here: https://docs.confluent.io/cloud/current/faq.html#what-version-of-ak-does-ccloud-use.

Can it be a problem in the way the SASL credentials are being build?

Our application can communicate directly with the broker without any problem, however our apps are running on confluent-kafka-go.

[silasfrigo]

Hello @zroubalik. We tested using Sarama and something weird happened. Using exactly the same configs we use on KEDA 1.5, we cannot send the message, and in order to make the message go, we had to do the following:

        config := sarama.NewConfig()
	config.ClientID = "sarama-cc-test"
	config.Producer.RequiredAcks = sarama.WaitForAll
	config.Producer.Retry.Max = 5
	config.Producer.Return.Successes = true
	config.Net.SASL.Enable = true
	config.Net.SASL.Mechanism = "PLAIN"
	config.Net.SASL.User = "username"
	config.Net.SASL.Password = "password"
	config.Net.TLS.Enable = true

We had to set config.Net.TLS.Enable as True, even without having it on our kafka and without passing its config to Sarama.
Without TLS being enable, we could not send the message, we got the same error "...is your cluster reachable?"

[silasfrigo]

For anyone that faces this issue in the future, i found this Pull Request that solved our issue: #1288

Closing the issue for now.

Thank you @zroubalik for all the information, and @odilonjk for all the help you gave in the investigation process.