Loose python framework for kickass redis patterns
Python
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
example
src
.gitignore
readme.md
setup.py

readme.md

Kickass-Redis - a loose framework of Redis based data solutions

This project aims to create a repository of useful python libraries built on top of redis (and using each other), to automate data modeling with Redis.

For discussion, help and contributing code - join the google group at https://groups.google.com/forum/#!forum/kickass-redis

Redis is relatively low level, and while it is simple to start using, getting a good knowledge of how to model problems with it in an efficient way can be tricky. So I've created this project to wrap common use cases, into a loose framework of redis based solutions for real world problems.

The project has started out as a bunch of code examples for a presentation I recently gave on IL Tech Talks, that can be found here: http://www.slideshare.net/dvirsky/kicking-ass-with-redis

Installation

Option 1:

  1. Clone this repo or download the sources

  2. cd kickass-redis

  3. python setup.py build

  4. sudo python setup.py install

Option 2:

sudo pip install kickass_redis

Components


To kick things off, the framework includes the following components:

object_store

a fast yet simple ORM (well, OM actually) that automates creation, indexing and searching for complex objects using redis.

Indexes include: simple string index, numeric index that supports sorting and ranges, simplistic full text index, and a unique key.

Example:

from kickass_redis.patterns.object_store.objects import IndexedObject, KeySpec
from kickass_redis.patterns.object_store.indexing import UnorderedKey, OrderedNumericalKey
from kickass_redis.patterns.object_store.condition import Condition

class User(IndexedObject):


    #which fields should be saved to redis
    _spec = ('id', 'name', 'email', 'pwhash', 'registrationDate', 'score')

    #The keys for this object
    _keySpec = KeySpec(
        UnorderedKey(prefix='users',fields=('name',)),
        OrderedNumericalKey(prefix='users', field='score')
    )

    def __init__(self, **kwargs):
        IndexedObject.__init__(self, **kwargs)
        self.registrationDate = int(kwargs.get('registrationDate', time.time()))

#Creating a user
user = User(email = 'user@domain.com', name = 'John Doe', pwhash = 'eabc626ec26bc6ae6cb2', score = 100)
user.save()

#loading by name key
users =  User.get(Condition({'name': 'John Doe'}))

#loading by id:
users = User.loadObjects((1,))

#See example/users_example for a more detailed exmample and some benchmarks

bitmap_counter

efficient unique value counter (to be used mostly as a unique users counter) with time slots, making use of redis bitmaps.

It makes use of new redis-2.6 commands BITCOUNT and BITOP, so it will not function on redis-2.4.

Example:

from kickass_redis.patterns.bitmap_counter import BitmapCounter

#Daily unique users counter
counter = BitmapCounter('unique_users', timeResolutions=(BitmapCounter.RES_DAY,))

#sampling current user
counter.add(3)

#Getting the unique user count for today
counter.getCount((time.time(),), counter.RES_DAY)

#Getting cohort analysis on your users for the past week
week = tuple((int(time.time() - i*86400) for i in  xrange(7, 0, -1)))
print counter.cohortAnalysis(week, counter.RES_DAY)

#Getting funnel analysis on your users for the past week
print counter.funnelAnalysis(week, counter.RES_DAY)

New:

It now also supports mapping of non sequential or non numeric ids to incemental ids, that makes it memory optimized.

LuaCall

A convenience wrapper that allows you to edit, precache and call Lua scripts available in redis-2.6, as if they were native python functions.

Example:

let mult.lua contain the code:

local val = ARGV[1]*ARGV[2]
redis.call('set', KEYS[1], val)
return redis.call('get', KEYS[1])

Running it from python:

import redis
from kickass_redis.patterns.lua import LuaCall, LuaScriptError
conn = redis.Redis()

#Define the call, and make it runn on our connection
mult = LuaCall(open('mult.lua'), conn)

#Call it once:
print "Result: %s" % mult(keys = ('foor',), args = (3,10))

#call it again
print "Result: %s" % mult(keys = ('foor2',), args = (5,20))

idgenerator

Used in the object store, this can also be used standalone, as a centralized unique, incremental id generator using redis. To optimize performance, it reserves in local memory many ids when accessing redis, which can be tuned.

redis_unit

A unit-test like set of assertions about redis data to be used to validate the data inside a redis database.

requirements

Example:

from kickass_redis.patterns.redis_unit import RedisDataTest

class MyTest(RedisDataTest):

    def testSomeStuff(self):


        self.assertPrefixCount('users:*', minAmount=100000)
        self.assertKeysExists('users:1')
        self.assertKeysType(self.test.T_HASH, 'users:1')
        self.assertHashValue('users:1', 'name', self.equals('John'))


test = MyTest('localhost', 6379)
test.run()


To follow soon:

  • geo search

  • full text search

  • hierarchical counters

  • MySQL data sync

  • Generic expiring object cache.

Feel free to contribute more recipes...

Project TODO:

  1. Add unit tests for all objects

  2. Add geo-key