rdb-parser fails for ziplists when an entry is more than 16383 bytes - Wrong Endian Format #16

Merged
merged 2 commits into from Oct 4, 2012

Conversation

Projects
None yet
2 participants

cgordon commented Oct 4, 2012

The dump.rdb for one of our Redis databases was crashing when I ran "rdb" on it, but was loading fine into Redis. After a few hours of hunting around, I figured out that there was an endian issue. The code in ziplist.c, from Redis, writes out the length of ziplist entries as a 4-byte, big-endian value, but only when the entry is more than 16,383 bytes. That doesn't happen by default, since hash-max-ziplist-value is 4096. The database that was having trouble just happened to have a higher value for that option. Everything else written out by Redis is little-endian, as far as I can, so this was pretty surprising!

The change here fixes the bug, and I've added a test to check for it (but you have to run Redis with the appropriate option, or the test doesn't check anything).

cgordon added some commits Oct 4, 2012

@cgordon cgordon Fixing a endian-ness bug in read_ziplist_entry.
The Redis dump file is using little endian numbers everywhere except for the
length of ziplist entries of more than 16,383 bytes. That is a hard edge case
to trigger, because it requires increasing hash-max-ziplist-value, since the
default insures that this case will never happen. I've added a test to catch
the issue, but it requires setting hash-max-ziplist-value to over 20,000.
4a4b7e0
@cgordon cgordon Adding the code to create the new test entry. 3e6b705
Owner

sripathikrishnan commented Oct 4, 2012

This was a nasty little bug. I only can imagine the pain you'd have gone through to fix this. Appreciate your efforts!

@sripathikrishnan sripathikrishnan added a commit that referenced this pull request Oct 4, 2012

@sripathikrishnan sripathikrishnan Merge pull request #16 from cgordon/master
rdb-parser fails for ziplists when an entry is more than 16383 bytes - Wrong Endian Format
b44e210

@sripathikrishnan sripathikrishnan merged commit b44e210 into sripathikrishnan:master Oct 4, 2012

1 check passed

default The Travis build passed
Details
Owner

sripathikrishnan commented Oct 4, 2012

For posterity, methods zipEncodeLength and the macro ZIP_DECODE_LENGTH in ziplist.c are the ones that encode and decode the length in big endian format.

cgordon commented Oct 4, 2012

I had a blast finding this bug. Everyone uses JSON/XML/HTTP these days, so I almost never get to work with binary formats anymore. Thanks again for writing the library, it has saved us a ton of time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment