Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Universal Binary Format 2.2 and Redis

tree: 70c73ead09

Fetching latest commit…

Cannot retrieve the latest commit at this time

README.md

Universal Binary Format and Redis

Copyright (c) 2012 by Joseph Wayne Norton

Authors: Joseph Wayne Norton (norton@alum.mit.edu).

This is UBF-REDIS, a framework for integrating UBF and the Redis Unified Format (RUF) protocol. This repository depends on the ubf open source repository.

This repository is experimental in nature - use at your own risk and please contribute if you find UBF-REDIS useful.

Quick Start Recipe

To download, build, and test the ubf_redis application in one shot, please follow this recipe:

$ mkdir working-directory-name
$ cd working-directory-name
$ git clone https://github.com/ubf/ubf-redis.git ubf_redis
$ cd ubf_redis
$ ./rebar get-deps
$ ./rebar clean
$ ./rebar compile
$ ./rebar eunit

For an alternative recipe with other "features" albeit more complex, please read further.

Documentation

Where should I start?

This README is a good first step.

The UBF User's Guide is the best next step. Check out http://ubf.github.com/ubf/ubf-user-guide.en.html for further detailed information.

This repository has building blocks for constructing your own Redis server clone in Erlang.

  • src/ruf.erl Redis network protocol (a.k.a. RUF) encoder/decoder

  • src/ruf_term.erl Redis raw Erlang encoder/decoder

  • src/ubf_redis_types_plugin.erl Erlang types for Redis RPC

  • src/ubf_redis_plugin.erl Erlang Redis RPC request/response pairs

The QC (a.k.a. QuickCheck, PropEr, etc.) tests underneath the "tests/qc" directory should be helpful for understanding the specification and behavior of these building blocks.

  • test/qc/ruf_tests.erl prop_ruf_requests() tests encoding and decoding of RUF requests. Test inputs are automagically generated using the above Redis RPC types.This QC test has been completed.

  • test/qc/ruf_tests.erl prop_ruf_responses() tests encoding and decoding of RUF responses. Test inputs are automagically generated using the above Redis RPC types.This QC test has been completed.

  • test/qc/redis_proxy_tests.erl tests an Erlang Redis client talking with an Erlang Redis "Proxy" server. The Redis "Proxy" server forwards all requests transparently to a "real" Redis server (using another internal Erlang Redis client). All requests and responses sent to the Redis "Proxy" server are automagically generated and checked using UBF's contract manager with the above Redis RPC types and request/response pairs.This QC test is stil in-progress.

A few "hello world" Eunit tests can also be found in the test/eunit directory.

See below for further details on how to run these tests.

What is UBF?

UBF is the "Universal Binary Format", designed and implemented by Joe Armstrong. UBF is a language for transporting and describing complex data structures across a network. It has three components:

  • UBF(a) is a "language neutral" data transport format, roughly equivalent to well-formed XML.

  • UBF(b) is a programming language for describing types in UBF(a) and protocols between clients and servers. This layer is typically called the "protocol contract". UBF(b) is roughly equivalent to Verified XML, XML-schemas, SOAP and WDSL.

  • UBF(c) is a meta-level protocol used between a UBF client and a UBF server.

See http://norton.github.com/ubf for further details.

What is Redis?

Redis is an open source, advanced key-value store. See http://redis.io for full details. In particular, see http://redis.io/topics/protocol for a description of the Redis protocol.

To download

  1. Configure your e-mail and name for Git

    $ git config \--global user.email "you@example.com"
    $ git config \--global user.name "Your Name"

  2. Install Repo

    $ mkdir -p ~/bin
    $ wget -O - https://dl-ssl.google.com/dl/googlesource/git-repo/repo > ~/bin/repo
    $ chmod a+x ~/bin/repo

  3. Create working directory

    $ mkdir working-directory-name
    $ cd working-directory-name
    $ repo init -u https://github.com/ubf/manifests.git -m ubf-redis-default.xml
    Note Your "Git" identity is needed during the init step. Please enter the name and email of your GitHub account if you have one. Team members having read-write access are recommended to use "repo init -u git@github.com:ubf/manifests.git -m ubf-redis-default-rw.xml".
    Tip If you want to checkout the latest development version, please append " -b dev" to the repo init command.

  4. Download Git repositories

    $ cd working-directory-name
    $ repo sync

For futher information and help for related tools, please refer to the following links:

To build - basic recipe

  1. Get and install an erlang system http://www.erlang.org

  2. Build

    $ cd working-directory-name
    $ make compile
  3. Run the unit tests

    $ cd working-directory-name
    $ make eunit

To build - optional features

  1. Dialyzer Testing basic recipe

    1. Build Dialyzer's PLT (required once)

      $ cd working-directory-name
      $ make build-plt
      Tip Check Makefile and dialyzer's documentation for further information.
    2. Dialyze with specs

      $ cd working-directory-name
      $ make dialyze
      Caution If you manually run dialyzer with the "-r" option, execute "make clean compile" first to avoid finding duplicate beam files underneath rebar's .test directory. Check Makefile for further information.
    3. Dialyze without specs

      $ cd working-directory-name
      $ make dialyze-nospec

To test - QuickCheck

  1. Make sure QuickCheck is in your Erlang code path. One simple way to accomplish this is by adding the code path to your ~/.erlang resource file.

    true = code:add_pathz(os:getenv("HOME")++"/.erlang.d/deps/quviq/eqc-X.Y.Z/ebin").
  2. Compile for QuickCheck

    $ cd working-directory-name
    $ make clean
    $ make eqc-compile
  3. Run 5,000 Redis Encoder/Decoder QuickCheck tests

    $ cd working-directory-name/deps/ubf_redis/.test
    $ erl -smp +A 5 -pz ../../ubf/ebin -pz ../../qc/ebin
    
    1> ruf_tests:qc_run(5000).
    prop_ruf_requests: ...
    ...
    Ok, passed 5000 tests
    prop_ruf_responses: ...
    ...
    Ok, passed 5000 tests
    []
    .......
  4. Run 500 Redis Proxy QuickCheck tests

    $ cd working-directory-name/deps/ubf_redis/.test
    $ erl -smp +A 5 -pz ../../ubf/ebin -pz ../../qc/ebin
    
    1> redis_proxy_tests:qc_run(500).
    #Port: {data,{eol,>}}
    #Port: {data,{eol,>}}
    #Port: {data,{eol,>}}
    ....
    OK, passed 500 tests
    
    100.0% {1,attempts}
    
    1.01% {decr_req,'->','integer()'}
    1.01% {bitop_and_req,'->',{error,'ERR unknown command \'BITOP\''}}
    0.98% {bitop_xor_req,'->',{error,'ERR unknown command \'BITOP\''}}
    0.96% {srandmember_req,'->',undefined}
    0.95% {hmset_req,'->',ok}
    0.95% {bitop_not_req,'->',{error,'ERR unknown command \'BITOP\''}}
    0.93% {zcount_req,'->','integer()'}
    0.93% {zadd_req,'->','integer()'}
    0.93% {msetnx_req,'->','integer()'}
    0.92% {zscore_req,'->',undefined}
    0.92% {zremrangebyrank_req,'->','integer()'}
    0.92% {pexpire_req,'->',{error,'ERR unknown command \'PEXPIRE\''}}
    0.92% {incr_req,'->','integer()'}
    0.91% {scard_req,'->','integer()'}
    0.91% {object_encoding_req,'->',undefined}
    0.90% {bitop_or_req,'->',{error,'ERR unknown command \'BITOP\''}}
    0.90% {append_req,'->','integer()'}
    0.89% {time_req,'->',{error,'ERR unknown command \'TIME\''}}
    0.89% {sinterstore_req,'->','integer()'}
    0.89% {pttl_req,'->',{error,'ERR unknown command \'PTTL\''}}
    0.88% {zrem_req,'->','integer()'}
    0.88% {smembers_req,'->','list()'}
    0.88% {move_req,'->',{error,'ERR index out of range'}}
    0.88% {monitor_req,'->',ok}
    0.88% {mget_req,'->','list()'}
    0.88% {lindex_req,'->',undefined}
    0.88% {info_req,'->','list()'}
    0.88% {flushall_req,'->',ok}
    0.88% {config_resetstat_req,'->',ok}
    0.87% {zrevrank_req,'->',undefined}
    0.87% {setex_req,'->',ok}
    0.87% {rpushx_req,'->','integer()'}
    0.87% {hincrbyfloat_req,'->',{error,'ERR unknown command \'HINCRBYFLOAT\''}}
    0.87% {expireat_req,'->','integer()'}
    0.87% {exists_req,'->','integer()'}
    0.86% {sunionstore_req,'->','integer()'}
    0.86% {strlen_req,'->','integer()'}
    0.86% {renamenx_req,'->',{error,'ERR no such key'}}
    0.86% {rename_req,'->',{error,'ERR no such key'}}
    0.86% {keys_req,'->','list()'}
    0.86% {bitcount_req,'->',{error,'ERR unknown command \'BITCOUNT\''}}
    0.86% {sadd_req,'->','integer()'}
    0.86% {mset_req,'->',ok}
    0.86% {hlen_req,'->','integer()'}
    0.86% {bgrewriteaof_req,'->',ok}
    0.85% {sdiff_req,'->','list()'}
    0.85% {getrange_req,'->','binary()'}
    0.84% {slaveof_req,'->',ok}
    0.84% {incrbyfloat_req,'->',{error,'ERR unknown command \'INCRBYFLOAT\''}}
    0.84% {hsetnx_req,'->','integer()'}
    0.84% {decrby_req,'->','integer()'}
    0.83% {zcard_req,'->','integer()'}
    0.83% {select_req,'->',{error,'ERR invalid DB index'}}
    0.83% {getset_req,'->',undefined}
    0.82% {zremrangebyscore_req,'->','integer()'}
    0.82% {srem_req,'->','integer()'}
    0.82% {sismember_req,'->','integer()'}
    0.82% {restore_req,'->',{error,'ERR unknown command \'RESTORE\''}}
    0.82% {pexpireat_req,'->',{error,'ERR unknown command \'PEXPIREAT\''}}
    0.82% {incrby_req,'->','integer()'}
    0.82% {sdiffstore_req,'->','integer()'}
    0.82% {ltrim_req,'->',ok}
    0.82% {lset_req,'->',{error,'ERR no such key'}}
    0.82% {lpush_req,'->','integer()'}
    0.82% {del_req,'->','integer()'}
    0.81% {type_req,'->',none}
    0.81% {multi_req,'->',ok}
    0.81% {lrem_req,'->','integer()'}
    0.81% {hexists_req,'->','integer()'}
    0.81% {debug_object_req,'->',{error,'ERR no such key'}}
    0.80% {smove_req,'->','integer()'}
    0.80% {object_idletime_req,'->',undefined}
    0.80% {linsert_req,'->','integer()'}
    0.80% {get_req,'->',undefined}
    0.80% {flushdb_req,'->',ok}
    0.79% {zrangebyscore_req,'->','list()'}
    0.79% {watch_req,'->',ok}
    0.79% {migrate_req,'->',{error,'ERR unknown command \'MIGRATE\''}}
    0.78% {zrevrangebyscore_req,'->','list()'}
    0.78% {zrank_req,'->',undefined}
    0.78% {set_req,'->',ok}
    0.78% {lpushx_req,'->','integer()'}
    0.78% {config_get_req,'->','list()'}
    0.78% {lpop_req,'->',undefined}
    0.78% {hvals_req,'->','list()'}
    0.78% {hgetall_req,'->','list()'}
    0.77% {rpush_req,'->','integer()'}
    0.77% {psetex_req,'->',{error,'ERR unknown command \'PSETEX\''}}
    0.77% {dump_req,'->',{error,'ERR unknown command \'DUMP\''}}
    0.76% {spop_req,'->',undefined}
    0.76% {hincrby_req,'->','integer()'}
    0.76% {hget_req,'->',undefined}
    0.76% {discard_req,'->',{error,'ERR DISCARD without MULTI'}}
    0.75% {llen_req,'->','integer()'}
    0.75% {expire_req,'->','integer()'}
    0.74% {zrevrange_req,'->',{error,'ERR syntax error'}}
    0.74% {lrange_req,'->','list()'}
    0.74% {hset_req,'->','integer()'}
    0.74% {getbit_req,'->','integer()'}
    0.74% {hdel_req,'->','integer()'}
    0.73% {sunion_req,'->','list()'}
    0.73% {hmget_req,'->','list()'}
    0.73% {hkeys_req,'->','list()'}
    0.73% {config_set_req,'->','error()'}
    0.72% {setnx_req,'->','integer()'}
    0.72% {rpop_req,'->',undefined}
    0.72% {persist_req,'->','integer()'}
    0.72% {lastsave_req,'->','integer()'}
    0.72% {dbsize_req,'->','integer()'}
    0.71% {setbit_req,'->',{error,'ERR bit is not an integer or out of range'}}
    0.71% {auth_req,'->',{error,'ERR Client sent AUTH, but no password is set'}}
    0.70% {sinter_req,'->','list()'}
    0.70% {ping_req,'->',ok}
    0.70% {object_refcount_req,'->',undefined}
    0.69% {zrange_req,'->',{error,'ERR syntax error'}}
    0.68% {sort_req,'->','list()'}
    0.65% {ttl_req,'->','integer()'}
    0.63% {rpoplpush_req,'->',undefined}
    0.63% {echo_req,'->','binary()'}
    0.61% {setrange_req,'->',{error,'ERR string exceeds maximum allowed size (512MB)'}}
    0.55% {randomkey_req,'->','binary()'}
    0.44% {zincrby_req,'->','integer()'}
    0.23% {zincrby_req,'->','float()'}
    0.15% {randomkey_req,'->',undefined}
    0.14% {setrange_req,'->','integer()'}
    0.10% {zrevrange_req,'->','list()'}
    0.10% {sort_req,'->','integer()'}
    0.09% {zrange_req,'->','list()'}
    0.06% {setbit_req,'->','integer()'}
    0.02% {sdiff_req,'->',{error,'ERR Operation against a key holding the wrong kind of value'}}
    0.02% {bgrewriteaof_req,'->',{error,'ERR Background append only file rewriting already in progress'}}
    0.01% {sunion_req,'->',{error,'ERR Operation against a key holding the wrong kind of value'}}
    0.01% {sdiffstore_req,'->',{error,'ERR Operation against a key holding the wrong kind of value'}}
    .......
    Tip This test requires redis-server to be installed under the /usr/local/ directory.
    Caution This test removes all data under the /usr/local/var/db/redis/ directory.

To test - PropEr

Under Construction - To Be Added

Roadmap

  • Documentation

    • Explain overall implementation and test strategy.

    • Explain relevant UBF features.

    • Explain how one can start to build their own Redis server in Erlang.

  • Testing - Black Box

    • Implement real "statem" test model for the Redis Proxy.

Modules

ruf
ruf_driver
ruf_term
ubf_redis_plugin
ubf_redis_types_plugin
Something went wrong with that request. Please try again.