Redis Overview

Additionally, here is a very good video guide for installing/learning the basic functionality of Redis. More will be covered in this document.

Setting Up

Installation

First download Redis either from the redis.io website or from this special link that always directs you to the latest version of Redis: http://download.redis.io/redis-stable.tar.gz
To compile Redis, enter the following lines in your terminal in sequence:

wget http://download.redis.io/redis-stable.tar.gz

tar xvzf redis-stable.tar.gz

cd redis-stable

make

*** Note about the src directory ***

The src directory contains multiple executables. Here is a brief overview of each one:

redis-server is the Redis Server itself
redis-sentinel is the Redis Sentinel executable (monitoring and failover)
redis-cli is the command line interface utility to talk with Redis
redis-benchmark is used to check Redis performances
redis-check-aof and redis-check-dump are useful in the rare event of corrupted data files

(Optional but highly recommended) Test whether the build works properly with: make test. If it works properly, you should see "All tests passed without errors!".
(Optional but highly recommended) To copy the command line and interface into the proper places for use, type the command: sudo make install. This is synonymous to typing:

sudo cp src/redis-server /usr/local/bin/

sudo cp src/redis-cli /usr/local/bin/

Starting

From here on out, it'll be assumed that /usr/local/bin is in your PATH environment. Type redis-server into the command line to start the Redis server (this is simply executing the redis-server binary) **Note this starts Redis without any configuration file. This is sufficient when first starting out or when you're intending to play around with it. In order to start Redis with a configuration file, simply type in the full file path after redis-server as an argument. I.e. if your Redis file is called "redis.conf" and it's in the etc/ directory, you would type redis-server /etc/redis.conf

Check if Redis is Working

To first check if Redis is working, type redis-cli ping (WHILE the Redis server is running). You should get a response of "PONG". This sends a command and its arguments to the instance of the Redis server currently running. You can also simply type in redis-cli to start an interactive mode.

Source

Data Structures

Redis is a data structures server so unlike in plain-key value stores in which string keys are associated with string values, string keys can be associated with other types of values (more complex data structures). Some of the data structures Redis provides support for are:

Binary-safe strings
Lists- essentially linked lists that are sorted according to order of assertion
Sets- unique unsorted elements
Sorted Sets- sets in which every string element is assorted to a floating point value. Gives the possibility of retrieving a certain subset of elements (i.e. top 10)
Hashes- just like an normal hash in which string hashes are associated with string values
Bit arrays (bitmaps)- allows for the possibility of handling string values like an array of bits and manipulating bits
HyperLogLogs- probabilistic data structure which is used in order to estimate the cardinality of a set

*** Commands will be denoted in all caps

Keys

You can use any binary sequence for a key
Maximum allowed key size is 512 MB
Be consistent with keys
Find a balance in length for keys

Expiration

Redis has a functionality of setting an expiration time for a key to expire. Time can be set in seconds or milliseconds, expiration time resolution is 1 millisecond, and the expiration time can be removed (not after it has expired however). Read more about it on the source linked at the bottom.

Strings

SET assigns a value to a string key. Reassigns if key already has a value. Read more about options for SET here.
```
 set [key name] [value]
```
GET retrieves the value of a string key. Read more about options for GET here.
```
 get [key name]
```
INCR parses a string value into an integer, increments it by 1, and reassigns it to the original key. Read more about options for INCR here. Additionally, this is atomic, meaning if two people on the same server perform this operation on the same existing key, it will increment by 2. Similar commands to INCR include INCRBY, DECR and DECRBY.
```
 incr [key name]
```
GETSET sets a key to a new value and returns the old value. Read more about options for GETSET here.
```
 getset [key name] [value]
```
MSET and MGET respectively set multiple keys to values at a time and get the values of multiple keys at a time. Read more about MSET here and about MGET here.
```
 mset [key1] [key2] [key3] [value1] [value2] [value3]
 mget [key1] [key2] [key3]
```
EXISTS returns a 1 if a key exists and 0 if not. Read more about EXISTS here.
```
 exists [key name]
```
DEL returns a 1 if a key has been successfully deleted and 0 if not. Read more about DEL here.
```
 del [key name]
```
TYPE returns the type of a key. Read more about TYPE here.
```
 type [key name]
```

Lists

For the sake of brevity, functions will not be explicitly covered as they were with strings from here on out. They will be mentioned in passing and any reference to critical ones can be found on the source linked at the bottom.

Redis lists are implemented via linked lists. This is to say that regardless of list size, the time it takes to add an element to the list is performed in constant time. The tradeoff is the ability to access an element of a list by index (which lists implemented via arrays have). Redis lists are implemented via linked lists precisely because for a database system, it's very important to have the ability to append to large lists quickly.

Lists have the functionalities of appending to the head and tail, retrieving a range of values, and popping from the head and tail. Redis also provides support for capped lists which are lists that only store the N latest added items. Blocking operations on lists are also supported, which circumvent useless requests to Redis (popping when there are no elements in the list) and the delay associated with calling a useless operation on Redis. These operations only return when the operation can be properly called on the list or after a used specified amount of time. General behavior of lists (and of all Redis data types) are denoted as:

When we add an element to an aggregate data type, if the target key does not exist, an empty aggregate data type is created before adding the element.
When we remove elements from an aggregate data type, if the value remains empty, the key is automatically destroyed.
Calling a read-only command such as LLEN (which returns the length of the list), or a write command removing elements, with an empty key, always produces the same result as if the key is holding an empty aggregate type of the type the command expects to find.

Some important functions include: LPUSH, RPUSH, LRANGE, LPOP, RPOP, LTRIM (for capped lists), BRPOP (for blocking), BLPOP (for blocking), LLEN.

Hashes

Hashes are what you'd expect as a data structure with field-value pairs. You can put as many fields as you want into a hash (so long as memory permits) so it has more utility than just representing an object.

Some important functions include: HMSET, HGET, HMGET. Here's a full list.

Sets

Sets are an unordered collection of unique elements. Redis provides support for getting the union, intersection, difference sum and more of sets. Sets are useful for implementing tags for an object. SPOP is also particularly useful since it removes a random element from a set and returns it (has many applications i.e. simulating a card game). SUNIONSTORE can be used to duplicate a set since it takes multiple sets as an argument and stores the union into a set (if you take the union of one set that's simply duplicating it). SCARD provides the cardinality, or the number of elements, in a set.

Some important functions include SADD, SMEMBERS, SISMEMBER, SINTER, SPOP, SUNIONSTORE, SCARD, SRANDMEMBER.

Sorted Sets

Redis Sorted Sets are a sort of marriage between hashes and sets. While they have unique string elements, each string element is associated with a floating point value. This is called the score of an element. If A and B are two elements of the same set and A has a higher score, then A > B. If A and B have the same score, then A > B because A is lexicographically greater. Because the set is sorted automatically, adding an element to the sorted set has O(log(n)) complexity at the benefit of having a constant time for returning a sorted list of the elements. Redis provides support for getting ranges by score (both forwards and backwards), removing them by score and returning the rank of an element. Recently, there's also been support provided for ranging by lexicographical score. A common use case for sorted sets is leaderboard rankings for games, due to the ability to sort lexicographically and by score. Common use cases for bitmaps include data analytics and space efficient storage that still allows for high performance boolean operations.

Some important functions include ZADD, ZRANGE, ZREVRANGE, ZRANGEBYSCORE, ZREMRANGEBYSCORE, ZREVRANK, ZRANGEBYLEX, ZREVRANGEBYLEX, ZLEXCOUNT.

Bitmaps

Bitmaps are not really a data structure, but rather a set of bit operations on strings that Redis provides support for. Operations are divided by constant time operations (operations on a single bit) vs. operations on groups of bits. The value bitmaps provide lies in the ability to provide space savings for data.

Some important functions include SETBIT, GETBIT, BITOP, BITCOUNT, BITPOS.

HyperLogLogs

A HyperLogLog is a probabilistic data structure provides support for counting unique things. Traditionally, this requires space proportional to the amount of unique items, but, because of the algorithm Redis uses for these, a constant amount of space, with 12K bytes at worst case, is needed. The tradeoff for space saving is usually precision, but in the case of the Redis algorithm, the standard error under 1%. HLL (HyperLogLogs) are technically encoded as Redis strings. This means you can call string functions like GET and SET on HLL's to serialize and deserialize onto the server. Items aren't technically added to an HLL since an HLL is a state. For example, every time you see a new item, you update the HLL with PFADD. A common use case is counting unique queries by users on a searching platform. Here is the full documentation.

Some important functions include PFADD, PFCOUNT.

Source

Making a Redis Module

Modules are basically C shared libraries that Redis can load at runtime or at startup, but with additional functionality and benefits.

Modules contain command handlers
The standard entry point for a module is at the function RedisModule_OnLoad which tells Redis what commands are in the module and maps them to the handler
RedisModule_Call command allows a Redis function to be called within another function
Include "redismodule.h" and if the entry point function has also been implemented, the module should be good to go in terms of linking

To compile the module on Linux (where the module name is module):

 $ gcc -fPIC -std=gnu99 -c -o module.o module.c
 $ ld -o module.so module.o -shared -Bsymbolic -lc

To compile the module on OSX (where the module name is module):

 $ gcc -dynamic -fno-common -std=gnu99 -c -o module.o module.c
 $ ld -o module.so module.o -bundle -undefined dynamic_lookup -lc

To load the module (where the module name is module):
```
 $ redis-server --loadmodule /path/to/module.so
```

Source

Native types in Redis

Redis modules can access Redis' built-in data structures in one of two ways -at a high level using the preexisting Redis commands -at a low level by directly manipulating the data structure

Using these capabilities it is possible to create modules that on a very less complex level export new data types. The ability of Redis modules to implement new data structures that feel like native Redis ones is called "native types support"

Components of a native type:
- The implementation of some kind of new data structure and of commands operating on the new data structure.
- A set of callbacks that handle: RDB saving, RDB loading, AOF rewriting, releasing of a value associated with a key, calculation of a value digest (hash) to be used with the DEBUG DIGEST command.
- A 9 characters name that is unique to each module native data type. (I found the reason for such a specific amount of characters to be interesting, see here for more.)
- An encoding version, used to persist into RDB files a module-specific data version, so that a module will be able to load older representations from RDB files.

Registering a New Native Data type

Additionally, all new all new data types must be registered into the Redis core. For any new data types that we wish to create, the module holding it, must declare a global variable that will hold a reference to the data type. The API to register the data type will return a data type reference that will be stored in the global variable.

An example outline of registering a new native type

static RedisModuleType *MyType;
#define MYTYPE_ENCODING_VERSION 0

int RedisModule_OnLoad(RedisModuleCtx *ctx) {
    MyType = RedisModule_CreateDataType("MyType-AZ", MYTYPE_ENCODING_VERSION,
        MyTypeRDBLoad, MyTypeRDBSave, MyTypeAOFRewrite, MyTypeDigest,
        MyTypeFree);
    if (MyType == NULL) return REDISMODULE_ERR;
}

An example module

moduleType *RedisModule_CreateDataType(RedisModuleCtx *ctx,
           const char *name, int encver,
           moduleTypeLoadFunc rdb_load,
           moduleTypeSaveFunc rdb_save,
           moduleTypeRewriteFunc aof_rewrite,
           moduleTypeDigestFunc digest,
           moduleTypeFreeFunc free);

Note: The ctx argument is the context that we receive in the OnLoad function. The type name is a 9 character name in the character set that includes from A-Z, a-z, 0-9, plus the underscore _ and minus - characters.

There is a lot more to be specificities around creating your own module that implements a native data type, There are specific things such as

Setting and Getting Keys

After registering our new data type with the RedisModule_OnLoad() function, you need to be able to set Redis keys having the same value as our native type.

This normally happens in the context of commands that write data to a key. The native types API allow to set and get keys to module native data types, and to test if a given key is already associated to a value of a specific data type.

The API uses the normal modules RedisModule_OpenKey() low level key access interface in order to deal with this. This is an eaxmple of setting a native type private data structure to a Redis key:

RedisModuleKey *key = RedisModule_OpenKey(ctx,keyname,REDISMODULE_WRITE);
struct some_private_struct *data = createMyDataStructure();
RedisModule_ModuleTypeSetValue(key,MyType,data);

Special Way To Free The New Native Type

Redis needs to free a key holding a native type value, it needs help from the module in order to release the memory.

Below is a example free, however this only works for data structures composed of a single allocation, So the trie one will have to be a bit more complex

typedef void (*RedisModuleTypeFreeFunc)(void *value);
void MyTypeFreeCallback(void *value) {
    RedisModule_Free(value);
}

RDB load and save methods

The RDB saving and loading callbacks need to create (and load back) a representation of the data type on disk. Redis offers an high level API that can automatically store inside the RDB file the following types:

Unsigned 64 bit integers.
Signed 64 bit integers.
Doubles.
Strings.

It is up to the module to find a viable representation using the above base types. However note that while the integer and double values are stored and loaded in an architecture and endianess agnostic way, if you use the raw string saving API to, for example, save a structure on disk, you have to care those details yourself.

AOF rewriting

See the source links and the Data Persistence section for more on this

Allocating memory

Modules data types should try to use RedisModule_Alloc() functions family in order to allocate, reallocate and release heap memory used to implement the native data structures (see the other Redis Modules documentation for detailed information).

Shell code

This link contains a very good shell of how a newly implemented native type module should look like.

Data Persistence

Redis provides two distinct kinds of data persistence, RDB and AOF. They can be used concurrently, individually or not at all. RDB provides snapshots of the working dataset at specified intervals while AOF logs every operation received by the server, essentially reconstructing original dataset. Read more about this topic here.

Navigation Menu

Uh oh!

Redis Overview

Setting Up

Installation

Starting

Check if Redis is Working

Data Structures

Keys

Expiration

Strings

Lists

Hashes

Sets

Sorted Sets

Bitmaps

HyperLogLogs

Making a Redis Module

Native types in Redis

Registering a New Native Data type

Setting and Getting Keys

Special Way To Free The New Native Type

RDB load and save methods

AOF rewriting

Allocating memory

Shell code

Data Persistence

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally