The default redis install process seen in post-create.sh sets redis to never evict stuff from memory. This led to an outage today because redis had gradually sucked up over a gig of memory, starving the VM.
The fix is pretty simple, we just set maxmemory to a limit in the redis conf file, and specify an eviction algorithm using maxmemory-policy.
@jedp is preparing a PR that'll set TTLs on keys. Once that's done, redis will reap expired keys by scanning 1000/second and killing expired ones. To guarantee we don't run out of memory, though, we can additionally set maxmemory-policy volatile-lru to evict least-recently-used expired keys when maxmemory is reached.
Annoyingly, installing redis via its install_server shell script doesn't give us any time between creating the config file and starting the server--and I've had trouble, for some reason, with /etc/init.d/redis_6379 stop not actually doing anything on the VM.
I assume we're building redis because the amzn repo doesn't have a redis package. I kinda think the best solution is to add an extra repo like EPEL, get redis from it, and then append the maxmemory and maxmemory-policy lines to it, assuming the init.d script will work properly with the upstream package:
echo "maxmemory 524288000 >> /etc/redis/6379.conf"
echo "maxmemory-policy volatile-lru >> etc/redis/6379.conf"
@jedp After looking into enforcing memory limits programmatically, I think it'd be way simpler and adequate to just use TTLs to enforce a memory limit. Closing as fixed by your TTL changes--reopen if you disagree.