Redis 7.0 Streams - use Lag instead of pending

[Jouse82]

Existing Redis Streams scaler uses the Xpending which may be completely false information. XPending only shows the number of read but unacked messages and therefore may not reflect the queue status.

Redis 7.0 has introduced a new metric "Lag", which shows the amount of unread messages for a Stream and Group.
XINFO stream group

This should be the made available for the Redis streams scaler to use instead of the Xpending.

[JorTurFer]

Hey @Jouse82
This sounds totally legit. My only concern is if this change can be done keeping the backward compatibility in any way or we will need to create (another) redis scaler 🤔

BTW, are you willing to contribute with this?

[tomkerkhove]

We should have the capability to configure which approach to use given this is only in Redis 7 as well

[KomTak001]

backward compatibility

Prerequisites: The threshold for stream and consumer group is obtained from yaml.

If XINFO GROUPS [streameName] has lag

threashold <= lag

Then scale

If there is no lag
Get last-delivered-id and get the entry list with, count the number,

XRANGE [streameName] [last-derivered-id for specific group] + COUNT [threashold]

and,

[Number] = threashold

Then scale

I think this will maintain backward compatibility to some extent, how about this?

[JorTurFer]

I'm not an expert on Redis but it sounds good

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[sergio-ferreira-jscrambler]

backward compatibility

Prerequisites: The threshold for stream and consumer group is obtained from yaml.

If XINFO GROUPS [streameName] has lag

threashold <= lag

Then scale

If there is no lag Get last-delivered-id and get the entry list with, count the number,

XRANGE [streameName] [last-derivered-id for specific group] + COUNT [threashold]

and,

[Number] = threashold

Then scale

I think this will maintain backward compatibility to some extent, how about this?

I'm worried of the performance hit this brings with the XRANGE command, which iterates all entries from the last-delivered-id to the last entry (+).

My team is using Redis 6 in production and since we don't have the lag value directly from Redis, we're using a script which does pretty much the same as your code to get the lag. It iterates with XRANGE since the last-delivered-id to the last message. However, we noticed that under high loads, and depending on the COUNT value, this iteration will block Redis for a few seconds - we've had test cases where this blocked Redis to up to 15 seconds

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[krishnadurai]

@JorTurFer can you please check the PR addressing this feature by my colleague @mikelam-us addressing this issue? We are both willing to maintain the Redis implementations:

#4592

[zroubalik]

Thanks @krishnadurai & @mikelam-us !

[zroubalik]

Implented in 2.11

[elpablete]

Is this available to use on "redis-streams" docs 2.13 ? I only see it in the docs for "redis-cluster-streams" docs 2.13

[JorTurFer]

Based on e2e tests I'd say that yes and it's not properly updated docs

[elpablete]

Can I help updating the docs?

[JorTurFer]

Thanks for the proposal! ❤️
I did it yesterday : kedacore/keda-docs#1304