redis acts like a db, not a cache #5

Closed
6a68 opened this Issue Jul 9, 2013 · 1 comment

Projects

None yet

1 participant

@6a68
Member
6a68 commented Jul 9, 2013

The default redis install process seen in post-create.sh sets redis to never evict stuff from memory. This led to an outage today because redis had gradually sucked up over a gig of memory, starving the VM.

The fix is pretty simple, we just set maxmemory to a limit in the redis conf file, and specify an eviction algorithm using maxmemory-policy.

@jedp is preparing a PR that'll set TTLs on keys. Once that's done, redis will reap expired keys by scanning 1000/second and killing expired ones. To guarantee we don't run out of memory, though, we can additionally set maxmemory-policy volatile-lru to evict least-recently-used expired keys when maxmemory is reached.

Annoyingly, installing redis via its install_server shell script doesn't give us any time between creating the config file and starting the server--and I've had trouble, for some reason, with /etc/init.d/redis_6379 stop not actually doing anything on the VM.

I assume we're building redis because the amzn repo doesn't have a redis package. I kinda think the best solution is to add an extra repo like EPEL, get redis from it, and then append the maxmemory and maxmemory-policy lines to it, assuming the init.d script will work properly with the upstream package:

echo "maxmemory 524288000 >> /etc/redis/6379.conf"
echo "maxmemory-policy volatile-lru >> etc/redis/6379.conf"
/etc/init.d/redis_6379 stop
/etc/init.d/redis_6379 start
@6a68
Member
6a68 commented Jul 12, 2013

@jedp After looking into enforcing memory limits programmatically, I think it'd be way simpler and adequate to just use TTLs to enforce a memory limit. Closing as fixed by your TTL changes--reopen if you disagree.

@6a68 6a68 closed this Jul 12, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment