Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace all usage of ziplist with listpack for t_hash #8887

Merged
merged 78 commits into from Aug 10, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
d08be0a
Add some interfaces to listpack for listpack migraion
sundb Apr 28, 2021
f459af5
Revert some code and fix valgrind
sundb Apr 28, 2021
98856fb
Remove unsed dot
sundb Apr 28, 2021
1b4a166
Make the hash object support both listpack and ziplsit
sundb Apr 29, 2021
0406a9d
Revert some code
sundb Apr 29, 2021
bfe9dd0
Remove some code
sundb Apr 29, 2021
7e4a1a9
Add listContainerListpack
sundb Apr 29, 2021
16e7298
Fix test fail
sundb Apr 29, 2021
44ad433
Fix module test fail
sundb Apr 29, 2021
f5a7e79
Change code style and incr rdb ver
sundb Apr 30, 2021
b49f9a6
Remove unused code and fix code style
sundb May 6, 2021
a3c27c7
Change return type of listLen and listBlogLen
sundb May 7, 2021
29e508f
Change list_container to packedClass
sundb May 7, 2021
1e86e6d
Add default packed encoding and add test
sundb May 7, 2021
bcfe84d
Fix lpValidateIntegrity bug
sundb May 7, 2021
5b9d556
Simplify lpValidateIntegrity
sundb May 7, 2021
9d30ddb
Change type of val length to size_t
sundb May 7, 2021
b03f34f
Fix packed_encoding error in test
sundb May 7, 2021
5b77c47
Convert ziplist to listpack when deep sanitization
sundb May 8, 2021
06b29a6
Add support to convert ziplist to listpack when rdb loading
sundb May 8, 2021
b0536d9
Optimize some code
sundb May 8, 2021
59e32e1
Convert ziplist to listpack when call hashTypeInitIterator
sundb May 8, 2021
b6f5028
Revert "Change type of val length to size_t"
sundb May 10, 2021
70041f6
Revert remain size_t type of val
sundb May 10, 2021
bd97653
Fix cr
sundb May 10, 2021
8ac548d
Change type of length of listpack and ziplist to unsigned long
sundb May 10, 2021
3cfc5e1
Add assert_packed_encoding support
sundb May 10, 2021
fc354b8
Revert some code
sundb May 10, 2021
be46b39
Optimize some code
sundb May 10, 2021
a06e42c
Optimize config default-packed-encoding
sundb May 10, 2021
45a6222
Missing a space
sundb May 10, 2021
0c081b3
Add --run-on-startup support
sundb May 11, 2021
012898f
Fix test fail
sundb May 11, 2021
abee5f4
Pre-alloc listpack when covert ziplist
sundb May 11, 2021
1fd6dd8
Fix missing ziplist method modification
sundb May 11, 2021
4f5f3f8
Speed up lpFind
sundb May 11, 2021
85fe4d9
Add some listpack unittest
sundb May 12, 2021
3c30078
Avoid use lpNext to speed up lpFind
sundb May 12, 2021
b84734e
Fix call lpBytes error
sundb May 12, 2021
85cadab
Use createEnumConfig to default-packed-encoding config
sundb May 14, 2021
32071aa
Add lpFind num benchmark for listpack and ziplsit
sundb May 14, 2021
14dc4a6
Remove unused code
sundb May 14, 2021
f0c4d34
Fix ci error
sundb May 14, 2021
049f847
Optimize entry size calculation to increase lpFind speed
sundb May 14, 2021
b9fff11
Add entry size macro for listpack
sundb May 18, 2021
a628c7e
Shrink listpack after convert from ziplist
sundb May 20, 2021
936de3e
Merge remote-tracking branch 'upstream/unstable' into listpack-migrat…
sundb Jun 10, 2021
9d6091b
Add external:skip tag to convert-ziplist-hash-on-load test
sundb Jun 10, 2021
c41edc2
Merge remote-tracking branch 'upstream/unstable' into listpack-migrat…
sundb Jun 22, 2021
bbc9b77
Optimize lpFind
sundb Jun 23, 2021
a8f9488
Revert some code
sundb Jun 24, 2021
e60f3a6
Fix compile warning for listpack under 32bit
sundb Jun 24, 2021
8794416
Fix confilic resolution error and add coverage test
sundb Jun 25, 2021
32ff0e3
Merge remote-tracking branch 'upstream/unstable' into listpack-migrat…
sundb Jul 6, 2021
129c019
Convert ziplist to listpack when RDB loading
sundb Jul 7, 2021
e897a2a
Add listpackEntry
sundb Jul 8, 2021
b9a807f
Fix corrupt-dump-fuzzer stream test fail
sundb Jul 9, 2021
cc98aa6
Remove unused test
sundb Jul 9, 2021
2e3ea44
Rename lpPushHead/lpPushTail to lpPrepend/lpAppend
sundb Jul 12, 2021
738ddf2
Convert ziplsit to hashtable direcly when rdb loading
sundb Jul 12, 2021
64c497a
Sanitize dump payload: hash listpack with duplicate records
sundb Jul 12, 2021
598aea8
Remove lpEntrySizeUnsafe method
sundb Jul 13, 2021
dd2af4b
Merge branch 'redis:unstable' into listpack-migration-hash
sundb Jul 19, 2021
f05f7b9
Change ziplist name to listpack for some comment, function name and c…
sundb Jul 19, 2021
b3c7763
Rename hash_max_ziplist_entries/hash_max_ziplist_value to hash_max_li…
sundb Jul 19, 2021
181a33a
Update src/t_hash.c
sundb Jul 22, 2021
ca49b55
Fix CR
sundb Jul 27, 2021
0d97b85
Update src/listpack.c
sundb Jul 27, 2021
d391ed6
Fix CR and add support to insert listpack with a integer directly
sundb Jul 28, 2021
4c034bb
Sanitize dump payload: hash listpack first element too long entry len
sundb Jul 28, 2021
2e1073d
Fix code style and add comment
sundb Jul 29, 2021
ac4985c
Add test for lpAppendInteger
sundb Jul 29, 2021
573a46f
Fix CR
sundb Aug 3, 2021
55e96b5
Update src/t_hash.c
sundb Aug 6, 2021
7766f66
Fix compile error
sundb Aug 6, 2021
46095fb
Merge branch 'unstable' into listpack-migration-hash
sundb Aug 6, 2021
23115d0
Fix CR & Re-add corrupt-dump test & Fix deep validate empty ziplsit fail
sundb Aug 6, 2021
76ecbaa
Merge remote-tracking branch 'upstream/unstable' into listpack-migrat…
sundb Aug 10, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 6 additions & 4 deletions src/listpack.c
Expand Up @@ -949,15 +949,17 @@ unsigned char *lpAppendInteger(unsigned char *lp, long long lval) {
return lpInsertInteger(lp, lval, eofptr, LP_BEFORE, NULL);
}

/* Remove the element pointed by 'p'. */
/* This is just a wrapper for lpInsert() to directly use a string to replace
* the current element. The function returns the new listpack as return
* value, and also updates the current cursor by updating '*p'. */
unsigned char *lpReplace(unsigned char *lp, unsigned char **p, unsigned char *s, uint32_t slen) {
return lpInsert(lp, s, NULL, slen, *p, LP_REPLACE, p);
}

/* This is just a wrapper for lpInsertInteger() to directly use a 64 bit integer
* instead of a string to replace the current element. The function returns
* the new listpack as return value, and also updates the current cursor
* by updating '*pos'. */
* by updating '*p'. */
unsigned char *lpReplaceInteger(unsigned char *lp, unsigned char **p, long long lval) {
return lpInsertInteger(lp, lval, *p, LP_REPLACE, p);
}
Expand Down Expand Up @@ -1114,8 +1116,8 @@ int lpValidateIntegrity(unsigned char *lp, size_t size, int deep,
while(p && p[0] != LP_EOF) {
sundb marked this conversation as resolved.
Show resolved Hide resolved
oranagra marked this conversation as resolved.
Show resolved Hide resolved
unsigned char *prev = p;

/* Move to the next entry in advance to avoid crash
* due to corrupt listpack. */
/* Validate this entry and move to the next entry in advance
* to avoid callback crash due to corrupt listpack. */
if (!lpValidateNext(lp, &p, bytes))
return 0;

Expand Down
2 changes: 1 addition & 1 deletion src/ziplist.c
Expand Up @@ -1527,7 +1527,7 @@ int ziplistValidateIntegrity(unsigned char *zl, size_t size, int deep,
return 0;

/* Make sure the <zltail> entry really do point to the start of the last entry. */
if (prev != ZIPLIST_ENTRY_TAIL(zl))
if (prev != NULL && prev != ZIPLIST_ENTRY_TAIL(zl))
oranagra marked this conversation as resolved.
Show resolved Hide resolved
return 0;

/* Check that the count in the header is correct */
Expand Down
10 changes: 10 additions & 0 deletions tests/integration/corrupt-dump.tcl
Expand Up @@ -670,5 +670,15 @@ test {corrupt payload: fuzzer findings - hash with len of 0} {
}
}

test {corrupt payload: fuzzer findings - hash listpack first element too long entry len} {
start_server [list overrides [list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r debug set-skip-checksum-validation 1
r config set sanitize-dump-payload yes
catch { r restore _hash 0 "\x10\x15\x15\x00\x00\x00\x06\x00\xF0\x01\x00\x01\x01\x01\x82\x5F\x31\x03\x02\x01\x02\x01\xFF\x0A\x00\x94\x21\x0A\xFA\x06\x52\x9F\x44" replace } err
assert_match "*Bad data format*" $err
verify_log_message 0 "*integrity check failed*" 0
}
}

} ;# tags