krmt: Kit of Redis Module Tools
Welcome to a kit of tools you can easily add to Redis.
This repository gives you multiple ready-to-use Redis Add-on Modules.
You can also use the modules here for examples and templates to create your own your own loadable Redis modules.
Note: You must use a Dynamic Redis server if you want to use modules. Regular Redis doesn't know about modules.
- Redis is single threaded for all command processing.
- While Redis is inside your command function, nothing else runs.
- Your command should operate and return as quickly as possible. See existing
Redis commands in the main
redisCommandTableat the top of redis.c for examples of how to write different Redis commands.
- Make sure to release objects and free memory you allocate. You will introduce memory leaks if you forget to free things.
- You don't have to write in C. You can use C, C++, Go (?), Guile, Chicken, or anything else generating a shared library with C symbols for your operating system. You can even link additional libraries (want numerical routines? embedded sqlite3? embedded snappy/zippy?)
Writing Your Module
To write your module: start out with the simple module provided in this repository and modify commands, functions, fields, and structs as needed.
You will need to study the existing Redis source to learn how existing Redis commands work.
Also see the notes at Dynamic Redis: Use Command Modules.
json.so- an experiment in storing JSON documents in Redis
- See Redis JSON Document Store for usage details.
geo.so- provides efficient geographic coordinate range searches
- See Redis Geo Commands for usage details.
HMSETIF(renamed from HMSETCK)
HMSETNX(from a pull request and updated to integrate into more recent Redis versions).
BITALLPOScommand returning the positions of all set bits in a string (from a pull request).
EVALNAMEcommands allowing you to bind user friendly names to loaded script SHA hashes, then you can call scripts by name (using
EVALNAME) instead of by a 40 character long hash reference.
- Right now, this module has the best comment structure and best design patterns for creating your own modules.
- Use the
dynamic-redis-unstablebranch to build
scriptname.cdepends on a header not exported on released versions yet.
poong.so- minimal module showing how the basic Dynamic Redis interface API works.
For building, you need a copy of the Dynamic Redis source tree.
If you want to build against the most recent 2.8 commits, use:
mkdir -p ~/repos cd ~/repos # We need YAJL for JSON commands and the Makefile refuses to run without it. git clone https://github.com/lloyd/yajl cd yajl ./configure; make cd .. git clone https://github.com/mattsta/redis cd redis git checkout dynamic-redis-2.8 make cd .. git clone https://github.com/mattsta/krmt cd krmt make clean; make -j
If you want to build against the current development branch,
After building your module, you can load it into Dynamic Redis.
As Redis adds or removes features, sometimes modules may need to be updated to take into account different functions or interfaces available to them.
Stable releases of Dynamic Redis (as of 0.5.2) define a few
DYN_FEATURE_[feature] constants you can use for preprocessor selective
including or excluding of code.
The currently defined features are negative features allowing the absense of them to mean "use behavior from unstable branch."
You can check all the current feature defines in
version.h of your
Dynamic Redis checkout.
For the unstable branch, no features are defined.
Automatic Module Reloading
During development, you can automatically reload your modules using some simple loops. Just run these from the directory where you compile your modules and everything should work.
while inotifywait -e modify module.so; do redis-cli config set module-add `pwd`/module.so done
brew install kqwait while kqwait module.so; do redis-cli config set module-add `pwd`/module.so done
If you are on OS X, you should monitor your redis-server for
new memory leaks by running
leaks in a persistent terminal window:
watch "leaks redis-server"
valgrind, except it can monitor leaks in any
live process. We run
leaks in a
watch loop to refresh the
check fairly often.
Under some use cases,
leaks can cause an infinite loop in OS X's
malloc routine. If that happens, rest assured it's a problem
leaks and not your code. You can also try running under
jemalloc on OS X (not the default) to clear up any strange OS X
If you are on Linux, you should run your module tests under
valgrind after a good day of development. Any changes should
also be re-evaluated for memory leaks and invalid memory access.
Perfecting a Redis command takes time and understanding. Using memory allocation debuggers will help you eventually internalize the complexities of how and when Redis uses reference counted objects and malloc'd strings for certain operations.
Sample output on startup on OS X (Linux will complain if you give it a module name without a path):
matt@ununoctium:~/repos/redis/src% ./redis-server --module-strict no --module-add ping.so  28 Mar 14:43:14.500 * Module [<builtin>] loaded 145 commands.  28 Mar 14:43:14.501 * Loading new [ping.so] module.  28 Mar 14:43:14.501 # [ping.so] version mismatch. Module was compiled against 2.9.11, but we are 2.8.8-dynamic-0.5.  28 Mar 14:43:14.501 # Strict version enforcement is disabled. Loading [ping.so] but undefined behavior may occur.  28 Mar 14:43:14.501 * Added command poong [ping.so]  28 Mar 14:43:14.501 * Added command pooooong [ping.so]  28 Mar 14:43:14.501 * Added command pooong [ping.so]  28 Mar 14:43:14.501 * Added command pinger [ping.so]  28 Mar 14:43:14.501 * Module [ping.so] loaded 4 commands.  28 Mar 14:43:14.501 * Running load function of module [ping.so].
Module reloaded with:
CONFIG SET module-add /Users/matt/repos/krmt/poong.so
Sample output on reloading a module (using a different filename, but the same module name):
 28 Mar 14:43:29.796 * Running cleanup function for [ping.so] module.  28 Mar 14:43:29.796 * Closed previous [ping.so] module.  28 Mar 14:43:29.796 * Loading new [/Users/matt/repos/krmt/poong.so] module.  28 Mar 14:43:29.796 * Replaced existing command poong [/Users/matt/repos/krmt/poong.so]  28 Mar 14:43:29.796 * Replaced existing command pooooong [/Users/matt/repos/krmt/poong.so]  28 Mar 14:43:29.796 * Replaced existing command pooong [/Users/matt/repos/krmt/poong.so]  28 Mar 14:43:29.796 * Replaced existing command pinger [/Users/matt/repos/krmt/poong.so]  28 Mar 14:43:29.796 * Module [/Users/matt/repos/krmt/poong.so] loaded 4 commands.  28 Mar 14:43:29.796 * Running load function of module [/Users/matt/repos/krmt/poong.so].
INFO modules output:
module_<builtin>:filename=<builtin>,compiled_against=2.8.8-dynamic-0.5, module_version=2.8.8-dynamic-0.5,first_loaded=0,last_loaded=0,commands=get,set, setnx,setex,psetex,append,strlen,del,exists,setbit,getbit,setrange,getrange, substr,incr,decr,mget,rpush,lpush,rpushx,lpushx,linsert,rpop,lpop,brpop, brpoplpush,blpop,llen,lindex,lset,lrange,ltrim,lrem,rpoplpush,sadd,srem,smove, sismember,scard,spop,srandmember,sinter,sinterstore,sunion,sunionstore,sdiff, sdiffstore,smembers,sscan,zadd,zincrby,zrem,zremrangebyscore,zremrangebyrank, zunionstore,zinterstore,zrange,zrangebyscore,zrevrangebyscore,zcount,zrevrange, zcard,zscore,zrank,zrevrank,zscan,hset,hsetnx,hget,hmset,hmget,hincrby, hincrbyfloat,hdel,hlen,hkeys,hvals,hgetall,hexists,hscan,incrby,decrby, incrbyfloat,getset,mset,msetnx,randomkey,select,move,rename,renamenx,expire, expireat,pexpire,pexpireat,keys,scan,dbsize,auth,ping,echo,save,bgsave, bgrewriteaof,shutdown,lastsave,type,multi,exec,discard,sync,psync,replconf, flushdb,flushall,sort,info,monitor,ttl,pttl,persist,slaveof,debug,config, subscribe,unsubscribe,psubscribe,punsubscribe,publish,pubsub,watch,unwatch, restore,migrate,dump,object,client,eval,evalsha,slowlog,script,time,bitop, bitcount,bitpos module_sh.matt.test.pong:filename=/Users/matt/repos/krmt/poong.so, compiled_against=2.8.8-dynamic-0.5,module_version=0.0001, first_loaded=0,last_loaded=1396032209,commands=poong,pooooong,pooong,pinger
Did you make a nice Redis Add-On Module? Does it have a beginning, a middle, and an end? Got a protagonist with some obstacles to overcome?
Just open a pull request against this repository if you want
to share your modules and get them added to
Modules are more likely to be accepted if they have meaningful comments, don't leak memory, don't crash Redis, and show us something new and delightful while solving problems we've had forever (or solving problems we didn't even know we had).