Keda GCP PubSub Triggering Unnecessary ScaledJobs After Ack

[emirsaidh]

Report

I have long-running ScaledJobs, lasting approximately 10 minutes. The trigger configuration appears as follows, scaling one job for each message in the PubSub queue. However, upon successful processing and sending an acknowledgment (ack) to PubSub, Keda checks the queue every 30 seconds. Due to PubSub's non-real-time behavior, it updates the UnackedMessageNumber almost after 2 minutes. Consequently, Keda scales new jobs every 30 seconds (3-4 times).

• type: gcp-pubsub
  metadata:
  mode: "SubscriptionSize" # Optional - Default is SubscriptionSize - SubscriptionSize or OldestUnackedMessageAge
  value: "1.0"
  subscriptionName: my-sub

Expected Behavior

• Upon receiving a message from PubSub, Keda should scale one job to process the message.
• After processing and acknowledging the message, Keda should wait for the next message before scaling another job.

Actual Behavior

• Keda checks the PubSub queue every 30 seconds, irrespective of the message processing time.
• Due to the delayed update of UnackedMessageNumber in PubSub (approximately after 2 minutes), Keda incorrectly scales new jobs every 30 seconds, resulting in unnecessary job scaling (3-4 times).

Steps to Reproduce the Problem

1- Set up a PubSub subscription.
2- Configure Keda to scale jobs based on SubscriptionSize.
3- Ensure that the ScaledJobs have a runtime of few minutes.
4- Start sending messages to the PubSub queue
5- Monitor the behavior of Keda as it scales jobs based on the messages received.
6- Observe that Keda incorrectly scales new jobs every 30 seconds, even though the ack message sent to PubSub.

Logs from KEDA operator

example

KEDA Version

2.11.2

Kubernetes Version

1.27

Platform

Other

Scaler Details

GCP PubSub

Anything else?

No response

[JorTurFer]

Hello @emirsaidh ,
Thanks for reporting! The problem here is that pub/sub still reports the jobs some time after your ACK the message and KEDA doesn't check the messages at all, so we cannot check if the message is the same or not. In this case, I'd suggest 2 options to mitigate the gap:

• Increase the pollingInterval to 2 minutes to mitigate the impact of this (but take into account that it will apply also for adding jobs)
• Changing your job's code to exit if there isn't any message in the queue. With this approach, you could finish and remove those jobs which have been created because of pub/sub propagation lag

I know that both aren't the best option, but the problem is that the backend (stackdriver api) still reports the message

[emirsaidh]

Unfortunately both solutions does not work for us but thanks a lot for the response and your time @JorTurFer, I understand that is not about Keda but Pub/Sub.