Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
SLOWLOG related crash in Redis 2.4.4 #322
@jeznet reported the following:
I got an error while Redis was under heavy load:
@jeznet well this are very useful informations indeed! Please can you tell me if it ever happened when you where not so near to the 4GB 32bit process addressing space? I think this is very likely not a bug but an issue with jemalloc when you allocate all the available address space in a 32bit process. Thanks.
So the 4GB limit is likely our issue here, however I'm not sure why jemalloc would behave like this, Redis should crash for out of memory in such a case. I'll try to simulate the issue locally and report back in this thread. Thanks!
p.s. please if you see any further issue post the stack trace here.
This comment has been minimized.
This comment has been minimized.Show comment Hide comment
I tried it with 2.6 and got this:
zmalloc: Out of memory trying to allocate 123 bytes
So Redis did not crashed with sigsegv but just more "gracefully" for out of memory.
Please could you tell me if this instance is a 64 bit box with a 32 bit Redis instance or if the box is 32 bit itself? Because I tried in the former setup of a 64 bit box with a 32 bit instance and this may lead to differences.
Hey @jeznet, allright, as expected I got a corruption as well once I retried with a different workload in your exact setup:
Long story short: jemalloc() can corrupt badly once you are out of memory (mine was at: used_memory_human:3.89G) and we should probably place a default limit around 3.8 GB in all the 32 bit instances that will prevent Redis to accept more data when this limit is reached.
So we can be sure what you experienced is not a bug with SLOWLOG or broken code, but a matter of out of memory. However I'm taking this open since I want to resolve the 32bit limit issue in some way.
@jeznet thanks, I suggest that you set some limit to avoid having this issue again for now, but I'll provide a general fix for 32 bit instances, However you can already get 99% of what the final fix will be with: