Proposal: Keyspace Notifications for Kvrocks

[Aetherance]

1. Background and goals

Applications reading from a read-only Kvrocks replica keep a local cache to cut latency and reduce round-trips to the primary. Keeping that cache fresh requires a notification when a key changes. Kvrocks has no such mechanism today, which blocks applications that depend on Redis Keyspace Notifications from migrating.

This design implements the minimal Redis-compatible Keyspace Notifications surface needed by #2915:

• notify-keyspace-events configuration.
• Events published to __keyspace@<db>__:<key> with payload = event name.
• Events published to __keyevent@<db>__:<event> with payload = key.
• Consumption through the existing SUBSCRIBE / PSUBSCRIBE.
• SET and DEL notifications on both primary and replica.

The MVP is intentionally narrow. It does not infer Redis command semantics from RocksDB storage mutations alone: those records cannot distinguish SET from APPEND, SETRANGE, or INCR, nor a genuine DEL command from lazy expiry or internal delete-and-recreate flows. Instead, it reuses the existing Redis-type LogData carried in the same write batch to distinguish the supported set and del cases.

2. Architecture

Redis emits keyspace events at the command execution point, and replicas re-emit them by replaying the replicated command stream. Kvrocks replicates the RocksDB WAL (WriteBatch), not Redis commands, so replicas need explicit command context when a storage mutation is ambiguous.

The design therefore extends the existing redis::WriteBatchLogData context in the same WriteBatch as the data mutation:

flowchart TD
    subgraph primary["Primary / standalone"]
        cmd["Command layer<br/>SET / DEL succeeds"]
        logdata["Create existing LogData<br/>for replicas"]
        local_event["Keep command-derived<br/>notify fact"]
        write["db_->Write(batch) succeeds"]
        publish_primary["After write succeeds<br/>publish local fact"]
        notify_primary["NotifyKeyspaceEvent"]
    end

    subgraph replica["Replica"]
        recv["Receive replicated WriteBatch"]
        apply["Apply batch succeeds"]
        decode_replica["Extract notify events<br/>during batch iteration"]
        notify_replica["NotifyKeyspaceEvent"]
    end

    pub["PublishMessage<br/>keyspace / keyevent"]
    clients["Local SUBSCRIBE / PSUBSCRIBE subscribers"]

    cmd --> logdata --> write --> publish_primary --> notify_primary
    cmd --> local_event --> publish_primary
    logdata -. "replicated with WriteBatch" .-> recv
    recv --> apply --> decode_replica --> notify_replica
    notify_primary --> pub
    notify_replica --> pub
    pub --> clients

Loading

This keeps the WAL as the replication carrier without guessing event names from column-family writes. The primary already has command execution context, so it keeps the local notification fact and publishes it only after db_->Write succeeds. The replica publishes after it applies the same batch and reconstructs the notification from the replicated LogData plus metadata records. With the same local notify-keyspace-events setting, both roles publish the same event names and payloads.

Pub/Sub fan-out is reused unchanged: Server::PublishMessage already handles exact-channel and pattern subscriptions and is a pure in-memory operation, so it works on read-only replicas.

3. Design decisions

3.1 Config: notify-keyspace-events

Add the notify-keyspace-events config. The default is "" (disabled). The MVP parser accepts the Kvrocks-supported subset of Redis flags K E A g $ l s h z t, rejects unsupported or unknown characters, and supports runtime updates through CONFIG SET.

Only g (del) and $ (set) currently produce notifications. The remaining data-type flags are accepted because Kvrocks has those storage types, but they have no MVP emitter yet. Internally these flags are parsed into a new notification bitmask, separate from the existing storage RedisType enum:

Bit	Flag	Meaning
1 << 0	K	publish keyspace channels
1 << 1	E	publish keyevent channels
1 << 2	g	generic events; MVP emits del
1 << 3	$	string events; MVP emits set
1 << 4	l	accepted, no MVP emitter
1 << 5	s	accepted, no MVP emitter
1 << 6	h	accepted, no MVP emitter
1 << 7	z	accepted, no MVP emitter
1 << 8	t	accepted, no MVP emitter

In this MVP, A expands only to the accepted Kvrocks data-class subset g$lshzt. Redis-only or out-of-scope flags such as a, d, x, e, m, n, o, and c are intentionally not accepted until their event semantics are implemented. Only $ and g produce events, so the common KEA setting enables both channel forms and lets set / del pass the filter.

The other classes enabled by A are accepted but inert for now. If later releases add notification extraction for list/hash/etc., existing KEA deployments will naturally start receiving those events; that matches Redis semantics, but should be called out in release notes.

3.2 Reusing existing LogData

Kvrocks already writes redis::WriteBatchLogData into many write batches with WriteBatch::PutLogData(). The current source model is:

class WriteBatchLogData {
  RedisType type_;
  std::vector<std::string> args_;
};

It is encoded as the numeric Redis type followed by optional arguments. Existing batch iterators already separate server-level log data from Redis-type log data before decoding. This MVP keeps that boundary and only extends Redis-type WriteBatchLogData; it does not add a new top-level log-data kind.

The MVP reuses that mechanism and slightly extends the optional arguments with notification command context, for example by adding Redis command codes such as kRedisCmdSet and kRedisCmdDel. New command codes must be appended without renumbering existing RedisCommand values. It does not add an independent kse1 record and does not put raw key names into LogData arguments. The affected namespace and key are still decoded from the existing metadata column-family PutCF / DeleteCF records with ExtractNamespaceKey, which preserves raw key bytes and avoids adding one log record per key. Primary-subkey and stream records use InternalKey elsewhere and are ignored by the del extractor.

A new or extended replica-side notification extractor is stateful in the same way as the existing WriteBatchExtractor: LogData establishes the current Redis type and optional command context, and subsequent metadata-column-family records can produce notifications when both the LogData context and the column-family operation match a supported event.

• set: requires a Redis string LogData context explicitly marked as a supported SET command path, then a metadata PutCF whose decoded metadata type is kRedisString.
• del: requires a LogData context explicitly marked as the supported DEL command path, then a metadata DeleteCF. It must not fire for primary-subkey deletes such as HDEL, SREM, ZREM, or list element removals.

The LogData must precede the records it describes, matching current write-batch patterns. If the context is missing, malformed, a server log, or unrelated to key notifications, the extractor skips notification emission for those records.

Decode errors are non-fatal in the notification pass. A malformed or unknown LogData command context is logged and skipped for notification purposes; it must not fail an already committed write or block replication.

LogData command context records semantic facts, not publication decisions. They are written only when a command has definitely performed a supported mutation. Each node still applies its own notify-keyspace-events filter when publishing.

LogData write policy:

• If the primary's notify-keyspace-events enables no supported class bit (g / $), supported commands may skip adding notification command context to avoid default-off overhead. The skip keys only on class bits, never on the K / E channel selectors, so a primary with a class bit but no selector still carries the context for replicas to publish.
• If the primary skips notification context, replicas cannot publish notifications for that write even if their local config is later enabled (the operational consequence for replica caches is stated as a guarantee in §5).
• When enabled, each supported batch adds only a small command-context extension to existing Redis LogData; the affected keys come from the normal write records already present in the WAL and replication stream.

Configuration is thus checked twice — at LogData-context write time on the primary and at publish time on each node; a CONFIG SET that disables notifications between the two still ships the context to replicas but suppresses the primary's local publish.

3.3 Command paths

Only commands with unambiguous MVP semantics add notification command context:

• SET: mark the batch as a supported set producer only when the literal SET command actually writes the key. Conditional variants such as SET key value NX must not add notification context when the condition fails. Shared string helpers must not infer set from kRedisString alone, because other string commands also write string metadata.
• DEL: mark the batch as a supported del producer only when the literal DEL command deletes at least one key. The extractor emits one del event per metadata DeleteCF, each carrying that single key. Missing keys do not produce write records or notifications. UNLINK currently shares CommandDel and MDel, so the implementation must distinguish the command name or command attributes and avoid enabling UNLINK unless it is explicitly added to the MVP scope.

Every other write path (the out-of-scope list in §4) adds no notification command context in this MVP. This is deliberate: emitting no event is safer than emitting a Redis event name that does not match the command semantics.

3.4 Event source and trigger points

Primary and replica publish at different points, and only the replica reconstructs notifications from LogData:

• The primary command path keeps a local notification fact for a successful literal SET / DEL, then calls Server::NotifyKeyspaceEvent only after db_->Write succeeds. It does not infer primary notifications from LogData or column-family records.
• The replica applies the replicated batch first, then extracts notifications from LogData and metadata records while iterating that same batch, reusing the path already present around parseWriteBatch.

Supported command paths pass notification command context to the type/storage operation that creates the existing WriteBatchLogData for replicas, but those type/storage operations do not publish. On the primary, publication is a single after-write step from the retained local fact. This keeps failed writes silent and prevents a single-node write from publishing once from the command path and once from the batch path.

Publication is bound only to live write/apply paths. During restart, RocksDB replays WAL internally inside DB::Open; that recovery does not invoke the notification path, so historical LogData is not republished.

Decode errors are non-fatal in the notification handler. Malformed or unsupported notification context is logged and skipped; keyspace notifications must never abort a committed write or stall replication.

3.5 Event emitter

Server::NotifyKeyspaceEvent applies the local config filter and reuses the existing Pub/Sub path to publish the two Redis-compatible channel forms:

For a successful SET foo bar in the default namespace, KEA publishes:

__keyspace@0__:foo  -> "set"
__keyevent@0__:set  -> "foo"

For a successful DEL foo, it publishes:

__keyspace@0__:foo  -> "del"
__keyevent@0__:del  -> "foo"

3.6 <db> / namespace mapping

For Redis compatibility, the default namespace maps to db 0, so standard subscriptions such as PSUBSCRIBE __keyevent@0__:* work out of the box.

Non-default Kvrocks namespaces cannot be inserted into <db> directly. A namespace literally named "0" would collide with Redis db 0 channels and could leak events across tenants. The mapping must therefore be collision-free.

MapNamespaceToKeyspaceDB(ns) is defined as:

• default namespace -> "0";
• non-default namespace -> "ns:" + PercentEncode(ns), where percent encoding escapes every byte outside [A-Za-z0-9_.-] and also escapes %.

Subscribers to __keyevent@0__:* therefore only see the default namespace. Subscribers that opt into non-default namespaces use the encoded form, for example __keyevent@ns:tenantA__:*. The payload remains only the key name, so namespace identity comes from the channel.

The key component in __keyspace@<db>__:<key> and the payload in __keyevent@<db>__:<event> are the raw user key bytes, matching Redis behavior. Keys are not percent-encoded, even if they contain :, whitespace, or binary bytes.

4. Event scope

Supported MVP events:

Command	Event class	Event	Emitted when
SET	$	set	the command writes the key
DEL	g	del	the key is actually deleted

Out of scope:

• UNLINK, APPEND, SETRANGE, GETSET, INCR, and other string commands.
• Hash/list/set/zset/stream element updates.
• EXPIRE, PEXPIRE, expired, and lazy/compaction expiry.
• FLUSHDB / FLUSHALL and DeleteRangeCF notifications.
• rename_from, rename_to, new, keymiss, evicted, and module events.

FLUSHDB is explicitly a limitation for cache invalidation: this MVP does not emit per-key or aggregate flush notifications. Applications that rely on flush visibility need a later design for aggregate flushdb / flushall events.

5. Semantics and guarantees

• Best effort only. Notifications are Pub/Sub messages: they are not persisted, acknowledged, replayed, or retried.
• Failed writes publish nothing. LogData context in an uncommitted batch has no visible effect because publication only runs after successful db_->Write or replica apply.
• A write commits before its local notification publishes; a crash in that window can make local subscribers miss the event. Replica notifications publish only after apply, so they can lag the primary, and a replica crash between apply and publish loses the event — restart recovery does not replay it, which also prevents duplicate notifications after crashes.
• notify-keyspace-events is node-local, and the primary's setting gates notification LogData context (§3.2). A primary with no supported class bit writes no notification context, so its replicas publish nothing no matter how they are configured. For the Receive notification for SET and DEL events on replicas (minimal implementation of notify-keyspace-events) #2915 replica-cache use case operators must therefore enable a supported class bit (for example KEA) on the primary — even if it has no local subscribers — and set the same value on every subscribed replica.
• Rolling upgrades rely on older nodes tolerating the slightly extended WriteBatchLogData arguments. Older nodes may ignore the notification context and not publish, but they must still apply and replicate the write batch.
• There is no cross-node de-duplication: a client subscribed to both primary and replica may process the same logical mutation twice.
• For one affected key, keyspace is published before keyevent on one node. No ordering is promised across different affected keys — multi-key commands like DEL k1 k2 k3, concurrent writes, or events on different nodes. Extraction follows write-batch iteration order, but clients must not depend on cross-key ordering.
• The event stream is a cache-invalidation hint, not a reliable CDC mechanism.

6. Tests

Add focused coverage, grouped by concern.

Configuration:

• Flag parsing, rejection of unsupported or invalid flags, and CONFIG SET runtime changes.
• Config bitmask: accepted flags map to distinct bits separate from the existing RedisType enum, and g / $ make del / set pass the filter for their event classes.
• A expansion sets the accepted Kvrocks data-class bits (g$lshzt) and makes KEA pass set / del through filtering.

Emission:

• SET publishes set only on success; NX / XX condition failures publish nothing.
• DEL publishes one del per actually deleted key, none for missing keys.
• UNLINK, APPEND, SETRANGE, INCR, and expiry paths publish no notifications.
• Disabled configuration emits nothing; replica subscribers receive set / del from master writes.

LogData and replication:

• Write policy: with primary class bits off, supported commands add no notification command context and replicas have nothing to publish; with them on, the extended WriteBatchLogData context is present.
• Codec: existing WriteBatchLogData still decodes Redis type and arguments; new notification command codes for set / del are recognized, while unknown command context is skipped by the notification handler without aborting WriteBatch::Iterate.
• WAL compatibility: handlers that only understand existing Redis type LogData continue to iterate and apply a batch containing the extended arguments.
• Key extraction: notification keys are decoded from metadata-column-family PutCF / DeleteCF records, not from raw key names stored in LogData arguments.
• A malformed notification context on a replica is logged/skipped without blocking replication progress or ack.

Semantics:

• Duplicate prevention: one successful primary write yields exactly one publication, not command-layer plus WAL-replay.
• Crash recovery: WAL recovery republishes no notifications; apply-time publication is best-effort and not replayed after restart.
• Runtime race: flags changed by CONFIG SET between LogData-context write and publication take effect at publish time.
• Ordering: keyspace precedes keyevent for one affected key; no test relies on cross-key ordering.
• Namespace mapping: default maps to 0; non-default names are prefixed and percent-encoded, including numeric names like "0", so tenant channels cannot collide.

7. References

• #2915 — feature request driving this design.
• Redis Keyspace Notifications — the compatibility target for channels, payloads, and flags.

[jihuayu]

I think the overall design looks good, but for the first phase, we should only support publishing changes from the primary node and not implement synchronization to replica nodes.

The synchronization logic for replica nodes may need further consideration. I think the current approach is not very efficient from a performance perspective.

[Aetherance]

Hi @jihuayu , when you have time, I’d appreciate your thoughts on the direction of the proposal. Once the general approach is confirmed, I can revise it into a complete proposal and move this feature forward. Thanks!

[jihuayu]

Hi @Aetherance.
I’ve been thinking about this for the past few days. I think we can reuse the existing WriteBatchLogData mechanism and extend it slightly. That should allow us to preserve accuracy while keeping the additional WAL overhead relatively small.

However, please don’t implement both the subscription mechanism and the replication/synchronization logic in a single PR. That would make the PR very large and difficult to review. We should start with a smaller scope and break this larger effort into multiple PRs.

In general, a PR with more than about 400 non-test lines of changes is already considered fairly large from a reviewer’s perspective. Larger PRs take substantially more time and effort to review.

[jihuayu]

Because Kvrocks uses a different expiration mechanism from Redis, I don’t think expired events can be implemented correctly on our side. Therefore, I don’t think we need to support this event.

[Aetherance]

Hi @jihuayu , thanks for the suggestion. I agree with the direction of reusing and slightly extending the existing LogData mechanism, and I’ve updated the proposal accordingly.

With the updated design, replicas can use the command context carried in LogData, together with the write batch records, to infer the original command semantics more accurately.

Could you please review the current proposal again when you have time? If it looks reasonable, I’ll start working on the implementation. I’ll split the changes into reasonably scoped PRs to make review easier.

Thanks again for your guidance!

[Aetherance]

Because Kvrocks uses a different expiration mechanism from Redis, I don’t think expired events can be implemented correctly on our side. Therefore, I don’t think we need to support this event.

I agree with this. expired should stay out of scope for this proposal. Kvrocks handles expiration differently from Redis, so it’s hard to provide correct Redis-compatible expired events, especially on replicas. I’ll keep this MVP focused on set and del only as requested in issue #2915. Whether to support other commands/events can be discussed separately later.

[Aetherance]

@jihuayu I think I'm ready to start implementing. My plan is to split it into two PRs to keep each one easy to review: the first one implementing keyspace notifications on the primary node (the notify-keyspace-events config plus set/del publishing on the __keyspace@__ / __keyevent@__ channels), and a second one adding replica support on top, deriving the same events from the WAL.

Does that sound reasonable? If so I'll get started on the first PR. Thanks!

[jihuayu]

Thanks. I think you can get started. I'll go over the sync stuff tomorrow