Hash Field Expiration #13303
Merged
Hash Field Expiration #13303
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add ebuckets & mstr data structures - Integrate active & lazy expiration - Add most of the commands - Add support for dict (listpack is missing) TODOs: RDB, notification, listpack, HSET, HGETF, defrag, aof
Unify infra of `HSETF`, `HEXPIRE`, `HSET` and provide API for RDB load as well. Whereas setting plain fields is rather straightforward, setting expiration time to fields might be time-consuming and complex since each update of expiration time, not only updates `ebuckets` of corresponding hash, but also might update `ebuckets` of global HFE DS. It is required to opt sequence of field updates with expirartion for a given hash, such that only once done, the global HFE DS will get updated. To do so, follow the scheme: 1. Call `hashTypeSetExInit()` to initialize the HashTypeSetEx struct. 2. Call `hashTypeSetEx()` one time or more, for each field/expiration update. 3. Call `hashTypeSetExDone()` for notification and update of global HFE. If expiration is not required, then avoid this API and use instead hashTypeSet().
- On ebExpire() verify the logic of update expired value to a new time rather than remove it. - Refine ebuckets benchmark
**Changes:**
- Adds listpack support to hash field expiration
- Implements hgetf/hsetf commands
**Listpack support for hash field expiration**
We keep field name and value pairs in listpack for the hash type. With
this PR, if one of hash field expiration command is called on the key
for the first time, it converts listpack layout to triplets to hold
field name, value and ttl per field. If a field does not have a TTL, we
store zero as the ttl value. Zero is encoded as two bytes in the
listpack. So, once we convert listpack to hold triplets, for the fields
that don't have a TTL, it will be consuming those extra 2 bytes per
item. Fields are ordered by ttl in the listpack to find the field with
minimum expiry time efficiently.
**New command implementations as part of this PR:**
- HGETF command
For each specified field get its value and optionally set the field's
expiration time in sec/msec /unix-sec/unix-msec:
```
HGETF key
[NX | XX | GT | LT]
[EX seconds | PX milliseconds | EXAT unix-time-seconds | PXAT
unix-time-milliseconds | PERSIST]
<FIELDS count field [field ...]>
```
- HSETF command
For each specified field value pair: set field to value and optionally
set the field's expiration time in sec/msec /unix-sec/unix-msec:
```
HSETF key
[DC]
[DCF | DOF]
[NX | XX | GT | LT]
[GETNEW | GETOLD]
[EX seconds | PX milliseconds | EXAT unix-time-seconds | PXAT
unix-time-milliseconds | KEEPTTL]
<FVS count field value [field value …]>
```
Todo:
- Performance improvement.
- rdb load/save
- aof
- defrag
1. Add `hpersist` notification for `hpersist` command. 2. Add `pexpire` notification for `hexpire`, `hexpireat` and `hpexpire`.
If encoding is listpack, hgetf and hsetf commands reply field value type as integer. This PR fixes it by returning string. Problematic cases: ``` 127.0.0.1:6379> hset hash one 1 (integer) 1 127.0.0.1:6379> hgetf hash fields 1 one 1) (integer) 1 127.0.0.1:6379> hsetf hash GETOLD fvs 1 one 2 1) (integer) 1 127.0.0.1:6379> hsetf hash DOF GETNEW fvs 1 one 2 1) (integer) 2 ``` Additional fixes: - hgetf/hsetf command description text Fixes #13261, #13262
## Background 1. All hash objects that contain HFE are referenced by db->hexpires. 2. All fields in a dict hash object with HFE are referenced by an ebucket. So when we defrag the hash object or the field in a dict with HFE, we also need to update the references in them. ## Interface 1. Add a new interface `ebDefragItem`, which can accept a defrag callback to defrag items in ebuckets, and simultaneously update their references in the ebucket. ## Mainly changes 1. The key type of dict of hash object is no longer sds, so add new `activeDefragHfieldDict()` to defrag the dict instead of `activeDefragSdsDict()`. 2. When we defrag the dict of hash object by using `dictScanDefrag()`, we always set the defrag callback `defragKey` of `dictDefragFunctions` to NULL, because we can't reallocate a field with out updating it's reference in ebuckets. Instead, we will defrag the field of the dict and update its reference in the callback `dictScanDefrag` of dictScanFunction(). 3. When we defrag the hash robj with HFE, we will use `ebDefragItem` to defrag the robj and update the reference in db->hexpires. ## TODO: Defrag ebucket structure incremently, which will be handler in a future PR. --------- Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com> Co-authored-by: Moti Cohen <moti.cohen@redis.com>
The same goes to: HPEXPIRE, HEXPIREAT, HPEXPIREAT, HEXPIRETIME, HPEXPIRETIME, HPTTL, HTTL, HPERSIST
Add RDB de/serialization for HFE This PR adds two new RDB types: `RDB_TYPE_HASH_METADATA` and `RDB_TYPE_HASH_LISTPACK_TTL` to save HFE data. When the hash RAM encoding is dict, it will be saved in the former, and when it is listpack it will be saved in the latter. Both formats just add the TTL value for each field after the data that was previously saved, i.e HASH_METADATA will save the number of entries and, for each entry, key, value and TTL, whereas listpack is saved as a blob. On read, the usual dict <--> listpack conversion takes place if required. In addition, when reading a hash that was saved as a dict fields are actively expired if expiry is due. Currently this slao holds for listpack encoding, but it is supposed to be removed. TODO: Remove active expiry on load when loading from listpack format (unless we'll decide to keep it)
Add the following validations: 1. Get TTL using the lpGetIntegerValue() method instead of lpGetValue(), Ref #13209 (comment) 2. The TTL of listpackex is a number in the valid range (0~EB_EXPIRE_TIME_MAX) and ordered. 3. The TTL fields of listpackex are ordered. 4. The TTL of hashtable is within the valid range (0~EB_EXPIRE_TIME_MAX). Other: Fix the missing of handling OBJ_ENCODING_LISTPACK_EX in dismissHashObject(). --------- Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
This PR contains a few optimizations for hfe listpack. - Hfe fields are ordered by TTL in the listpack. There are two cases that we want to search listpack according to TTLs: - As part of active-expiry, we need to find the fields that are expired. e.g. find fields that have smaller TTLs than given timestamp. - When we want to add a new field, we need to find the correct position to maintain the order by TTL. e.g. find the field that has a higher TTL than the one we want to insert. Iterating with lpNext() to compare TTLs has a performance cost as lpNext() calls lpValidateIntegrity() for each entry. Instead, this PR adds `lpFindCb()` to the listpack which accepts a comparator callback. It preserves same validation logic of lpFind() which is faster than search with lpNext(). - We have field name, value, ttl for a single hfe field. Inserting these items one by one to listpack is costly. Especially, as we place fields according to TTL, most additions will end up in the middle of the listpack. Each insert causes realloc + memmove. This PR introduces `lpBatchInsert()` to add multiple items in one go. - For hsetf, if we are going to update value and TTL at the same time, currently, we update the value first and later update the TTL (two distinct listpack operation). This PR improves it by doing it with a single update operation. --------- Co-authored-by: debing.sun <debing.sun@redis.com>
on active-expire of buckets, at function `ebExpire()` -> `ebSegExpire()`, if callback `onExpireItem()` indicated to stop, Yet iterator (iter) was wrongly already got updated to point to next item. In turn the segment will be updated without current item.
Added hashes_with_expiry_fields. Optimially it would better to have statistic of that counts all fields with expiry. But it requires careful logic and computation to follow and deep dive listpacks and hashes. This statistics is trivial to achieve and reflected by global HFE DS that has builtin enumeration of all the hashes that are registered in it.
Changes: - Delete hsetf and hgetf commands - Hfe commands will return empty array instead of nil. --------- Co-authored-by: Moti Cohen <moticless@gmail.com>
In #13291, we've changed that hfe commands to return empty array if the key does not exist. Forgot to update json schemas.
In the last step of hscan, while replying to client, we assume all items in the result list are keys which are mstr instances. Though, there might be values which are sds instances. Added a check to avoid calling mstrlen() for value objects. To reproduce: ``` 127.0.0.1:6379> hset myhash1 a 11111111111111111111111111111111111111111111111111111111111111111 (integer) 0 127.0.0.1:6379> hscan myhash1 0 1) "0" 2) 1) "a" 2) "11111111111111111111111111111111111111111111111111111111111111111\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" ```
* For replica sake, rewrite commands `H*EXPIRE*` , `HSETF`, `HGETF` to have absolute unix time in msec. * On active-expiration of field, propagate HDEL to replica (`propagateHashFieldDeletion()`) * On lazy-expiration, propagate HDEL to replica (`hashTypeGetValue()` now calls `hashTypeDelete()`. It also takes care to call `propagateHashFieldDeletion()`). * Fix `H*EXPIRE*` command such that if it gets flag `LT` and it doesn’t have any expiration on the field then it will considered as valid condition. Note, replicas doesn’t make any active expiration, and should avoid lazy expiration. On `hashTypeGetValue()` it doesn't check expiration (As long as the master didn’t request to delete the field, it is valid) TODO: * Attach `dbid` to HASH metadata. See [here](#13209 (comment)) --------- Co-authored-by: debing.sun <debing.sun@redis.com>
Fix position of numfields in H(P)EXPIRE json files
sundb
added
release-notes
|
fully CI passed: https://github.com/redis/redis/actions/runs/9285519601 |
moticless
approved these changes
May 30, 2024
tezc
approved these changes
May 30, 2024
|
This'll be an amazing feature to have, thanks so much for the work on this! 🙏 |
12 tasks
axeloradmin
pushed a commit
to axelor/axelor-open-platform
that referenced
this pull request
Jul 21, 2025
Native eviction currently requires Redis 7.4+ Redis: redis/redis#13303 Valkey: valkey-io/valkey#640 Addresses #1296
funny-dog
pushed a commit
to funny-dog/redis
that referenced
this pull request
Sep 17, 2025
## Background This PR introduces support for field-level expiration in Redis hashes. Previously, Redis supported expiration only at the key level, but this enhancement allows setting expiration times for individual fields within a hash. ## New commands * HEXPIRE * HEXPIREAT * HEXPIRETIME * HPERSIST * HPEXPIRE * HPEXPIREAT * HPEXPIRETIME * HPTTL * HTTL ## Short example from @moticless ```sh 127.0.0.1:6379> hset myhash f1 v1 f2 v2 f3 v3 (integer) 3 127.0.0.1:6379> hpexpire myhash 10000 NX fields 2 f2 f3 1) (integer) 1 2) (integer) 1 127.0.0.1:6379> hpttl myhash fields 3 f1 f2 f3 1) (integer) -1 2) (integer) 9997 3) (integer) 9997 127.0.0.1:6379> hgetall myhash 1) "f3" 2) "v3" 3) "f2" 4) "v2" 5) "f1" 6) "v1" ... after 10 seconds ... 127.0.0.1:6379> hgetall myhash 1) "f1" 2) "v1" 127.0.0.1:6379> ``` ## Expiration strategy 1. Integrate active Redis periodically performs active expiration and deletion of hash keys that contain expired fields, with a maximum attempt limit. 3. Lazy expiration When a client touches fields within a hash, Redis checks if the fields are expired. If a field is expired, it will be deleted. However, we do not delete expired fields during a traversal, we implicitly skip over them. ## RDB changes Add two new rdb type s`RDB_TYPE_HASH_METADATA` and `RDB_TYPE_HASH_LISTPACK_EX`. ## Notification 1. Add `hpersist` notification for `HPERSIST` command. 5. Add `hexpire` notification for `HEXPIRE`, `HEXPIREAT`, `HPEXPIRE` and `HPEXPIREAT` commands. ## Internal 1. Add new data structure `ebuckets`, which is used to store TTL and keys, enabling quick retrieval of keys based on TTL. 2. Add new data structure `mstr` like sds, which is used to store a string with TTL. This work was done by @moticless, @tezc, @ronen-kalish, @sundb, I just release it.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
This PR introduces support for field-level expiration in Redis hashes. Previously, Redis supported expiration only at the key level, but this enhancement allows setting expiration times for individual fields within a hash.
New commands
Short example
from @moticless
Expiration strategy
Redis periodically performs active expiration and deletion of hash keys that contain expired fields, with a maximum attempt limit.
When a client touches fields within a hash, Redis checks if the fields are expired. If a field is expired, it will be deleted. However, we do not delete expired fields during a traversal, we implicitly skip over them.
RDB changes
Add two new rdb type s
RDB_TYPE_HASH_METADATAandRDB_TYPE_HASH_LISTPACK_EX.Notification
hpersistnotification forHPERSISTcommand.hexpirenotification forHEXPIRE,HEXPIREAT,HPEXPIREandHPEXPIREATcommands.Internal
ebuckets, which is used to store TTL and keys, enabling quick retrieval of keys based on TTL.mstrlike sds, which is used to store a string with TTL.This work was done by @moticless, @tezc, @ronen-kalish, @sundb, I just release it.