Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace all usage of ziplist with listpack for t_hash #8887

Merged
merged 78 commits into from Aug 10, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
d08be0a
Add some interfaces to listpack for listpack migraion
sundb Apr 28, 2021
f459af5
Revert some code and fix valgrind
sundb Apr 28, 2021
98856fb
Remove unsed dot
sundb Apr 28, 2021
1b4a166
Make the hash object support both listpack and ziplsit
sundb Apr 29, 2021
0406a9d
Revert some code
sundb Apr 29, 2021
bfe9dd0
Remove some code
sundb Apr 29, 2021
7e4a1a9
Add listContainerListpack
sundb Apr 29, 2021
16e7298
Fix test fail
sundb Apr 29, 2021
44ad433
Fix module test fail
sundb Apr 29, 2021
f5a7e79
Change code style and incr rdb ver
sundb Apr 30, 2021
b49f9a6
Remove unused code and fix code style
sundb May 6, 2021
a3c27c7
Change return type of listLen and listBlogLen
sundb May 7, 2021
29e508f
Change list_container to packedClass
sundb May 7, 2021
1e86e6d
Add default packed encoding and add test
sundb May 7, 2021
bcfe84d
Fix lpValidateIntegrity bug
sundb May 7, 2021
5b9d556
Simplify lpValidateIntegrity
sundb May 7, 2021
9d30ddb
Change type of val length to size_t
sundb May 7, 2021
b03f34f
Fix packed_encoding error in test
sundb May 7, 2021
5b77c47
Convert ziplist to listpack when deep sanitization
sundb May 8, 2021
06b29a6
Add support to convert ziplist to listpack when rdb loading
sundb May 8, 2021
b0536d9
Optimize some code
sundb May 8, 2021
59e32e1
Convert ziplist to listpack when call hashTypeInitIterator
sundb May 8, 2021
b6f5028
Revert "Change type of val length to size_t"
sundb May 10, 2021
70041f6
Revert remain size_t type of val
sundb May 10, 2021
bd97653
Fix cr
sundb May 10, 2021
8ac548d
Change type of length of listpack and ziplist to unsigned long
sundb May 10, 2021
3cfc5e1
Add assert_packed_encoding support
sundb May 10, 2021
fc354b8
Revert some code
sundb May 10, 2021
be46b39
Optimize some code
sundb May 10, 2021
a06e42c
Optimize config default-packed-encoding
sundb May 10, 2021
45a6222
Missing a space
sundb May 10, 2021
0c081b3
Add --run-on-startup support
sundb May 11, 2021
012898f
Fix test fail
sundb May 11, 2021
abee5f4
Pre-alloc listpack when covert ziplist
sundb May 11, 2021
1fd6dd8
Fix missing ziplist method modification
sundb May 11, 2021
4f5f3f8
Speed up lpFind
sundb May 11, 2021
85fe4d9
Add some listpack unittest
sundb May 12, 2021
3c30078
Avoid use lpNext to speed up lpFind
sundb May 12, 2021
b84734e
Fix call lpBytes error
sundb May 12, 2021
85cadab
Use createEnumConfig to default-packed-encoding config
sundb May 14, 2021
32071aa
Add lpFind num benchmark for listpack and ziplsit
sundb May 14, 2021
14dc4a6
Remove unused code
sundb May 14, 2021
f0c4d34
Fix ci error
sundb May 14, 2021
049f847
Optimize entry size calculation to increase lpFind speed
sundb May 14, 2021
b9fff11
Add entry size macro for listpack
sundb May 18, 2021
a628c7e
Shrink listpack after convert from ziplist
sundb May 20, 2021
936de3e
Merge remote-tracking branch 'upstream/unstable' into listpack-migrat…
sundb Jun 10, 2021
9d6091b
Add external:skip tag to convert-ziplist-hash-on-load test
sundb Jun 10, 2021
c41edc2
Merge remote-tracking branch 'upstream/unstable' into listpack-migrat…
sundb Jun 22, 2021
bbc9b77
Optimize lpFind
sundb Jun 23, 2021
a8f9488
Revert some code
sundb Jun 24, 2021
e60f3a6
Fix compile warning for listpack under 32bit
sundb Jun 24, 2021
8794416
Fix confilic resolution error and add coverage test
sundb Jun 25, 2021
32ff0e3
Merge remote-tracking branch 'upstream/unstable' into listpack-migrat…
sundb Jul 6, 2021
129c019
Convert ziplist to listpack when RDB loading
sundb Jul 7, 2021
e897a2a
Add listpackEntry
sundb Jul 8, 2021
b9a807f
Fix corrupt-dump-fuzzer stream test fail
sundb Jul 9, 2021
cc98aa6
Remove unused test
sundb Jul 9, 2021
2e3ea44
Rename lpPushHead/lpPushTail to lpPrepend/lpAppend
sundb Jul 12, 2021
738ddf2
Convert ziplsit to hashtable direcly when rdb loading
sundb Jul 12, 2021
64c497a
Sanitize dump payload: hash listpack with duplicate records
sundb Jul 12, 2021
598aea8
Remove lpEntrySizeUnsafe method
sundb Jul 13, 2021
dd2af4b
Merge branch 'redis:unstable' into listpack-migration-hash
sundb Jul 19, 2021
f05f7b9
Change ziplist name to listpack for some comment, function name and c…
sundb Jul 19, 2021
b3c7763
Rename hash_max_ziplist_entries/hash_max_ziplist_value to hash_max_li…
sundb Jul 19, 2021
181a33a
Update src/t_hash.c
sundb Jul 22, 2021
ca49b55
Fix CR
sundb Jul 27, 2021
0d97b85
Update src/listpack.c
sundb Jul 27, 2021
d391ed6
Fix CR and add support to insert listpack with a integer directly
sundb Jul 28, 2021
4c034bb
Sanitize dump payload: hash listpack first element too long entry len
sundb Jul 28, 2021
2e1073d
Fix code style and add comment
sundb Jul 29, 2021
ac4985c
Add test for lpAppendInteger
sundb Jul 29, 2021
573a46f
Fix CR
sundb Aug 3, 2021
55e96b5
Update src/t_hash.c
sundb Aug 6, 2021
7766f66
Fix compile error
sundb Aug 6, 2021
46095fb
Merge branch 'unstable' into listpack-migration-hash
sundb Aug 6, 2021
23115d0
Fix CR & Re-add corrupt-dump test & Fix deep validate empty ziplsit fail
sundb Aug 6, 2021
76ecbaa
Merge remote-tracking branch 'upstream/unstable' into listpack-migrat…
sundb Aug 10, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions redis.conf
Expand Up @@ -1704,8 +1704,8 @@ notify-keyspace-events ""
# Hashes are encoded using a memory efficient data structure when they have a
# small number of entries, and the biggest entry does not exceed a given
# threshold. These thresholds can be configured using the following directives.
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
hash-max-listpack-entries 512
hash-max-listpack-value 64

# Lists are also encoded in a special way to save a lot of space.
# The number of entries allowed per internal list node can be specified
Expand Down
4 changes: 2 additions & 2 deletions src/aof.c
Expand Up @@ -1105,12 +1105,12 @@ int rewriteSortedSetObject(rio *r, robj *key, robj *o) {
*
* The function returns 0 on error, non-zero on success. */
static int rioWriteHashIteratorCursor(rio *r, hashTypeIterator *hi, int what) {
if (hi->encoding == OBJ_ENCODING_ZIPLIST) {
if (hi->encoding == OBJ_ENCODING_LISTPACK) {
unsigned char *vstr = NULL;
unsigned int vlen = UINT_MAX;
long long vll = LLONG_MAX;

hashTypeCurrentFromZiplist(hi, what, &vstr, &vlen, &vll);
hashTypeCurrentFromListpack(hi, what, &vstr, &vlen, &vll);
if (vstr)
return rioWriteBulkString(r, (char*)vstr, vlen);
else
Expand Down
4 changes: 2 additions & 2 deletions src/config.c
Expand Up @@ -2636,11 +2636,11 @@ standardConfig configs[] = {
createULongLongConfig("maxmemory", NULL, MODIFIABLE_CONFIG, 0, ULLONG_MAX, server.maxmemory, 0, MEMORY_CONFIG, NULL, updateMaxmemory),

/* Size_t configs */
createSizeTConfig("hash-max-ziplist-entries", NULL, MODIFIABLE_CONFIG, 0, LONG_MAX, server.hash_max_ziplist_entries, 512, INTEGER_CONFIG, NULL, NULL),
createSizeTConfig("hash-max-listpack-entries", "hash-max-ziplist-entries", MODIFIABLE_CONFIG, 0, LONG_MAX, server.hash_max_listpack_entries, 512, INTEGER_CONFIG, NULL, NULL),
createSizeTConfig("set-max-intset-entries", NULL, MODIFIABLE_CONFIG, 0, LONG_MAX, server.set_max_intset_entries, 512, INTEGER_CONFIG, NULL, NULL),
createSizeTConfig("zset-max-ziplist-entries", NULL, MODIFIABLE_CONFIG, 0, LONG_MAX, server.zset_max_ziplist_entries, 128, INTEGER_CONFIG, NULL, NULL),
createSizeTConfig("active-defrag-ignore-bytes", NULL, MODIFIABLE_CONFIG, 1, LLONG_MAX, server.active_defrag_ignore_bytes, 100<<20, MEMORY_CONFIG, NULL, NULL), /* Default: don't defrag if frag overhead is below 100mb */
createSizeTConfig("hash-max-ziplist-value", NULL, MODIFIABLE_CONFIG, 0, LONG_MAX, server.hash_max_ziplist_value, 64, MEMORY_CONFIG, NULL, NULL),
createSizeTConfig("hash-max-listpack-value", "hash-max-ziplist-value", MODIFIABLE_CONFIG, 0, LONG_MAX, server.hash_max_listpack_value, 64, MEMORY_CONFIG, NULL, NULL),
createSizeTConfig("stream-node-max-bytes", NULL, MODIFIABLE_CONFIG, 0, LONG_MAX, server.stream_node_max_bytes, 4096, MEMORY_CONFIG, NULL, NULL),
createSizeTConfig("zset-max-ziplist-value", NULL, MODIFIABLE_CONFIG, 0, LONG_MAX, server.zset_max_ziplist_value, 64, MEMORY_CONFIG, NULL, NULL),
createSizeTConfig("hll-sparse-max-bytes", NULL, MODIFIABLE_CONFIG, 0, LONG_MAX, server.hll_sparse_max_bytes, 3000, MEMORY_CONFIG, NULL, NULL),
Expand Down
14 changes: 13 additions & 1 deletion src/db.c
Expand Up @@ -935,7 +935,7 @@ void scanGenericCommand(client *c, robj *o, unsigned long cursor) {
while(intsetGet(o->ptr,pos++,&ll))
listAddNodeTail(keys,createStringObjectFromLongLong(ll));
cursor = 0;
} else if (o->type == OBJ_HASH || o->type == OBJ_ZSET) {
} else if (o->type == OBJ_ZSET) {
unsigned char *p = ziplistIndex(o->ptr,0);
unsigned char *vstr;
unsigned int vlen;
Expand All @@ -949,6 +949,18 @@ void scanGenericCommand(client *c, robj *o, unsigned long cursor) {
p = ziplistNext(o->ptr,p);
}
cursor = 0;
} else if (o->type == OBJ_HASH) {
unsigned char *p = lpFirst(o->ptr);
unsigned char *vstr;
int64_t vlen;
unsigned char intbuf[LP_INTBUF_SIZE];

while(p) {
vstr = lpGet(p,&vlen,intbuf);
listAddNodeTail(keys, createStringObject((char*)vstr,vlen));
p = lpNext(o->ptr,p);
}
cursor = 0;
} else {
serverPanic("Not handled encoding in SCAN.");
}
Expand Down
2 changes: 1 addition & 1 deletion src/defrag.c
Expand Up @@ -864,7 +864,7 @@ long defragKey(redisDb *db, dictEntry *de) {
serverPanic("Unknown sorted set encoding");
}
} else if (ob->type == OBJ_HASH) {
if (ob->encoding == OBJ_ENCODING_ZIPLIST) {
if (ob->encoding == OBJ_ENCODING_LISTPACK) {
if ((newzl = activeDefragAlloc(ob->ptr)))
defragged++, ob->ptr = newzl;
} else if (ob->encoding == OBJ_ENCODING_HT) {
Expand Down