Can redis key space notifications be pushed to the redis stream instead of pub/sub channel?

[jharishabh7]

We have a requirement that we need to get a notification on changes to a Redis data structure. Based on my research I found out that I can use Redis key space notifications for doing the same. However, Redis key space notifications send the events to Redis pub/sub channel which is fire and forget i.e once the clients lose the connections all the events till the connection is up again are lost.

Redis streams solve this problem. Also, I want to use consumer group feature of Redis streams. So is there any way that Redis key space notifications can be pushed to Redis streams instead of Redis pub/sub channel?

I would also like to contribute if this feature does not exist already.

[itamarhaber]

Hello @jharishabh7

Only modules can do this at the time being.

xref: https://stackoverflow.com/questions/54127210/can-redis-key-space-notifications-be-pushed-to-the-redis-stream-instead-of-pub-s

[jharishabh7]

Thank you @itamarhaber for the confirmation.

[gkorland]

@jharishabh7 can you describe the use case? which key space notifications would you like to be pushed to streams?

[jharishabh7]

@gkorland - Well the use case is that we want to get notifications, if status of a Data structure in redis changes(in our case if something has been pushed to a sorted set). So we thought we can use key space notifications for this purpose.

But key space notification internally use pub/sub channels of Redis which are fire and forget. So in case our client disconnects we will lose some notifications. Also we want to use something like a consumer group where only one of the client gets an event if we have multiple instances for scalability purposes.

So we thought it would be nice if key space notifications pushed the events to a Redis stream instead of a pub/sub channel. As we want the channel to be reliable and scalable.

And that is why I was wondering if this is possible to do in redis. And @itamarhaber said it is not possible to do it without building a module for it.

So we are now exploring blocked pop to solve our use case. But I think it would be a nice add-on to Redis functionality.

[antirez]

I think we'll end having such support soon, because it is trivial and it solves a huge problem with lost messages in keyspace notifications...

[jharishabh7]

Thank you so much @antirez . That will be awesome!

[antirez]

Thanks for validating the usefulness of this. It's quite some time that I think about it. Another "internal" use case I've for streams is to publish (not by default, only if enabled) server metrics every second to some stream. Memory usage, clients, iops-per-sec, ... Of course the stream will be capped but it's simple to retain a few days without much memory usage.

[jharishabh7]

Yes, that would be awesome too. Let me know if I can contribute in any way and if you are open for receiving some PR's.

[wang2bo2]

Thanks for validating the usefulness of this. It's quite some time that I think about it. Another "internal" use case I've for streams is to publish (not by default, only if enabled) server metrics every second to some stream. Memory usage, clients, iops-per-sec, ... Of course the stream will be capped but it's simple to retain a few days without much memory usage.

I’m working on a serverless stream consumer dispatcher. It would be very helpful if we could listen to the key space in a reliable way. In my case, that would allow users to create arbitrary streams and got consumer groups attached to it as soon as possible.

[maguec]

My use case here would be to watch a stream for key expiration events then to use gears to listen to that stream and then remove that document from a redis search index and/or redis graph

[antirez]

Just to make you aware. Right now the blocker on this is: what to do with Cluster? Users cannot specify a single key name in the cluster, the local instance may not map with such slot for instance. Also, do we want unified or non unified events in that case (per single instance or a single key in some instance)?

So far I'm thinking about two very different solutions:

1. Let's specify a single key name for the cluster as well: nodes will route the events to a single master that is responsible for such key, that will do all the XADDs. So we have a single unified view.
2. Allow Cluster to use database "1" (SELECT 1 basically) as a local, non sharded database. Use this private database for such tasks, that is, to push local events in that case, but in the future for other stuff.

It's a big choice. There is this other possibility:

3. Don't support that for Cluster for now and bring the feature ASAP for the standalone Redis mode.

[itamarhaber]

WRT 1 - that would potentially create a very hot master shard - smells off.
WRT 2 - a shard-local DB sounds like a good idea for a lot of things, and I prefer that approach for collecting keyspace notifications (and perhaps other types of notifications that may be upcoming). It also gives "atomicity" and as long as the shard is available, so are its events streams.

Another, perhaps half-baked, possibility:

3.5 Have the cluster connect to another, external, Redis database (single or cluster, proxy or not, HA or not, whatevs) for that purpose. In the case of a single instance, it can use 'local' and perhaps some configurable DB number.

[andrascz]

Is this still on the table? We would like to get __keyevent@0__:expired notifications for gathering some statistics about some expired keys (expired sessions). Our first thinking was putting this into the related service. Unfortunately relying on pubsub does not play nice with multiple runnig instance of the service. A stream would mean we could use a consumer group so only one instance would get an expiration event.

[itamarhaber]

Hello @andrascz

Is this still on the table?

I think it is still on the table, but no action has been taken in this direction so far. That said, it is possible to achieve this with modules, for example RedisGears. Here's a function that acts as an expiry trigger in a cluster (or standalone) and adds the key names to a stream:

EDIT: this is just a quick poc snippet to show the principle - the real solution should be slightly more complexified TBD
EDIT 2: A full(er) example is at https://oss.redislabs.com/redisgears/1.0/examples.html#keyspace-notification-processing

$ cat /tmp/expired-stream.py
streamkey = 'expired-stream'
gb = GearsBuilder()
gb.repartition(lambda x: streamkey)
gb.foreach(lambda x: execute('XADD', streamkey, '*', 'key', x['key']))
gb.register(prefix='*', eventTypes=['expired'])

$ cat /tmp/expired-stream.py | redis-cli -p 30001 -x  RG.PYEXECUTE
OK
$ redis-cli -c -p 30001
127.0.0.1:30001> KEYS *
(empty array)
127.0.0.1:30001> SETEX foo 1 bar
-> Redirected to slot [12182] located at 127.0.0.1:30003
OK
127.0.0.1:30003> SETEX baz 2 qux
-> Redirected to slot [4813] located at 127.0.0.1:30001
OK
127.0.0.1:30001> keys *
1) "expired-stream"
127.0.0.1:30001> XRANGE expired-stream 0 +
1) 1) "1588081672407-0"
   2) 1) "key"
      2) "foo"
2) 1) "1588081696056-0"
   2) 1) "key"
      2) "baz"
127.0.0.1:30001>

[andrascz]

Thank you for the pointer and example. This is even more powerful than the notification as in our use case we would generate a new element in a Redis List for every expired notification message if the key matches a prefix. I can accomplish this with only RedisGears by adding some extra logic in the executed Python code. So no need to read a stream from our services.

[ligoo]

@itamarhaber That's a great solution, thanks for the post. I tried it but something is going on that I can't understand.
When I'm listening to messages with something like XREAD-GROUP GROUP mygroup NAME myname STREAMS expired-stream > I can listen to new incoming messages when I simply XADD expired-stream * hello world. But when a key actually expire with SET toto titi ex 5 the script is running fine but it doesn't trigger the XREAD-GROUP thing as a new incoming message until some other client read the stream with something like XRANGE expired-stream 0 +.
What am I missing ?

[itamarhaber]

@ligoo rest assured there's little if anything that you're missing :)

That's an effect of a known issue - #6936 - and to work around it (for now), you need to use RG's 'sync' mode. I don't want to turn this to a discussion about the above, but feel free to use the RG repo for that purpose.

[soloestoy]

Here is a draft design:

1. add meta database in Redis, and use special db name to access meta database, for example select -1.
2. when a keyspace notification event triggered, add the event to a stream in meta database, it's similar with a special XADD:
   key foo expired -> select -1; xadd __keyevent@0__:expired * key foo
   Then we can use the streams commands to read the meta key.
3. for cluster, we allow using select -1 to access meta database, and accessing key in meta database doesn't need to redirect, it means different nodes could have same meta keys, just like notification channel.

The above is easy to implement, but there are still some problem I didn't figure it out(it's about replica and persistance, the key point is if the keys in meta database belong to real data or not? notify pubsub channel doesn't belong to real data) :

1. does the keys in meta database need to be persistance(both RDB and AOF)?
   • if not, the history of notification will be lost after restart and load data from disk.
   • if needs, the AOF file may grow too fast, since every notify XADD needs a SELECT -1 before it, and the next write command also needs a new SELECT.
   • and if needs, the RDB file would contain meta keys too, that will affect full resync, see below.
2. does the keys in meta database need to be replicated to replicas?
   • if not, the history of notification will be lost after failover.
   • if needs, there are some scenarios may lead to inconsistency:
     • the write commands replicated to replica would trigger notify on replicas, and then add events to replicas' notify stream, then the meta key in replica is larger than master's(to fix it, we can disable notify when the client is master).
3. allow or not allow XREADGROUP command on meta database in replica? since it's a write command. Or instead of X like commands, implement other commands to access meta key? maybe too complex.
4. FLUSHALL would flush meta database or not?

[oranagra]

to share one concern:
i feel it would be a bit hackish to use normal commands (SELECT and XRAD) on some hard coded key name.
i.e. we have some major feature of redis (keyspace notifications), implicitly creating "keys" and expecting the user to access them with normal commands and special key names. (in contrast for example to the SLOWLOG, which has its own command.

On alternative is maybe to put this in the hands of the user in some way, i.e. the user explicitly calls some command like KEYSPACE TRACK <key name>, if we do that, the user the one who redirected it to a specific key.
on the other hand, this poses some issue with cluster.

anyway, just sharing one more concern, maybe someone will be able to come up with a winning idea.

[yossigo]

This has been the subject of long sessions involving @oranagra @YaacovHazan @MeirShpilraien @guybe7 @itamarhaber @inbaryuval and others. I'm posting a brief summary of these discussions here for reference.

The bottom line is this gets more complex when we consider cluster environment and various scenarios, so very strong arguments about the value of this feature are needed to drive any further work.

Rationale - Why do it?

Redis already supports keyspace notifications to make it possible to track changes and operations on the server side.

There are many use cases that can benefit from this mechanism, including:

• Write through cache scenarios
• Change Data Capture (where Redis is the source)
• Other application patterns that are based on event sourcing

The main limitation of the existing keyspace notifications mechanism is that it offers very few guarantees. Notifications are sent once to connected clients and are not stored anywhere, so a client that drops a connection may lose any arbitrary number of notifications.

Reliable KSN (RKSN)

The goal is to define a keyspace notifications mechanism which is reliable. The properties of this solution are:

• Notifications are not lost, even if not immediately delivered
• Clients use a client-side cursor mechanism to be able to track new notifications
• Notifications are delivered in a total order to corresponds to the order of operations on the server

RKSN storage and access

The RKSN properties are identical to what Streams provide, so we considered the option of simply using stream keys, which can be accessed by clients using existing stream commands.

However, using regular stream keys creates a conflict with Redis Cluster as the key name must match the locally assigned hash slots. Redis could override that in the case of a notifications stream key, but this would also impact cluster-aware clients that would need to access the notifications stream key on all nodes regardless of hash slots topology.

Another alternative we considered was using a dedicated pseudo-database and SELECT to store those keys. However, this has been rejected as SELECT and multiple databases are not supported by Redis Cluster.

The final option was to store reliable notifications as special meta-data that lives outside the keyspace. The interface still resembles streams, but there is a dedicated (e.g., KREAD) command to access this data.

Replication and persistence

RKSN data needs to be persisted into RDB, for several reasons:

• A master may restart and still need to serve past notifications to clients
• A replica may perform full-sync, get promoted, and need to serve past notifications

The stream of notification includes additional meta-data, such as timestamp or stream entry ID that provide more information about the operation and support a client-side cursor mechanism.

Because of this, notifications must also be propagated locally to AOF and explicitly replicated to replicas. On the replica side, RKSN should be inhibited when processing commands on the replication link.

Redis Cluster

Slot-level ordering

In a cluster environment there is no concept of total order between operations that take place on different nodes. Because hash slots may also migrate between nodes, RKSN can effectively guarantee total ordering of notifications only for a specific hash slot.

Because of this, notifications should include an explicit hash slot identifier so clients can easily distinguish between ordered and unordered events.

RKSN Aggregation

Cluster aware clients need to read RKSN from all cluster nodes, as even for a single key there's no way to guarantee that the full RKSN history is available on a single node.

Clients will be able to aggregate, order, and if necessary de-duplicate notifications received from different nodes.

Migration

The cluster key migration mechanism needs to be enhanced to also support migration of RKSN entries. This operation does not need to be atomic as notifications may live on both source and target nodes during migration, but once a slot has been fully migrated we must guarantee the importing node has the full notification history.

Configuration

RKSN have a more significant impact on resources (memory in particular) and should have several configuration options to control that:

• Limit on number of stored notifications / memory used / etc.
• Notify-time filtering similar to the existing notify-keyspace-events configuration
• Read-time filters in the KREAD command.

Open Issues

FLUSHALL

Should the FLUSHALL command also flush all RKSN, or do we need a dedicated command that explicitly flushes the RKSN history?

Writable Replicas

We currently assume that replicas propagate received notifications and never generate local notifications on commands received on the replication link.

Should writable replicas generate and propagate notifications for commands received from local users?

[hpatro]

@yossigo Thanks for the detailed explanation. I would like to work on this. I see few alternatives mentioned above. Is there any recommended approach already or should I put up a proposal ?

[YourTechBud]

What's the current status of this? I would like to enable CDC like usecases for my current redis cluster

[oranagra]

the current status is summed up in the last big comment.
there are no plans to support this in the near future.

[hpatro]

@YourTechBud Could you explain your usecase a bit more?

Currently the notification you would receive on SET operation are the below two. This doesn't have the complete data (value part) to build CDC usecase. How do you plan to tackle it ?

1) "pmessage"
2) "*"
3) "__keyspace@0__:A"
4) "set"

1) "pmessage"
2) "*"
3) "__keyevent@0__:set"
4) "A"

[christiancarpinelli]

any news on this?