A Redis HTTP interface with JSON output
C Python Makefile Other
Latest commit 9ee6fe2 Feb 6, 2016 Nicolas Favre-Felix Version bump after release
Failed to load latest commit information.
b64 Rename libb64 to b64 for compatibility reasons, move to DEPS. Nov 11, 2013
formats Fix unused warning, use llabs instead of abs Nov 1, 2015
hiredis Update to hiredis 0.13.1 Jun 4, 2015
http-parser Fixed indentation and whitespace Jul 7, 2015
jansson Fix unused warning, use llabs instead of abs Nov 1, 2015
md5 Added ETag and If-None-Match. Jan 4, 2011
sha1 Started implementing the new WebSocket protocol Jul 20, 2011
tests Allow test programs use custom port using environment variable or com… Nov 11, 2013
.gitignore Started pub/sub test. Apr 13, 2011
.travis.yml Fix travis yaml file Sep 22, 2012
COPYING Added missing license file. Dec 28, 2010
Dockerfile Add Dockerfile Nov 17, 2013
Makefile Update to hiredis 0.13.1 Jun 4, 2015
README.markdown Write pidfile when daemonized (GitHub issue #10) Sep 7, 2013
acl.c Disable SELECT command - issue #80 Sep 7, 2013
acl.h Fix ACLs Jan 22, 2011
client.c Fixed indentation and whitespace Jul 7, 2015
client.h Handle memory allocation issues better. Feb 29, 2012
cmd.c Remove access to free'd memory in cmd_free May 7, 2013
cmd.h Create new connection for new separate DBs. Jun 23, 2012
conf.c Rename libb64 to b64 for compatibility reasons, move to DEPS. Nov 11, 2013
conf.h Write pidfile when daemonized (GitHub issue #10) Sep 7, 2013
http.c #89: Add Authorization to CORS headers Nov 18, 2013
http.h Make pub/sub chunks async. Jul 24, 2011
pool.c Schedule a reconnect if connecting fails Jun 4, 2015
pool.h Free Redis context for custom connections Apr 22, 2013
server.c Merge pull request #55 from littlefrog/master Nov 26, 2013
server.h Fixed log buf, removed spin lock. Apr 21, 2011
slog.c Remove comments mentioning borrowing code from redis repository. Nov 11, 2013
slog.h Reopen logs on SIGHUP (GitHub issue #10) Sep 7, 2013
version.h Version bump after release Feb 6, 2016
webdis.c Partial rewrite, adding WebSockets, threads, pool. Apr 8, 2011
webdis.json Handle SIGTERM & SIGINT (issue #10) Sep 7, 2013
webdis.prod.json Added fix-config.diff and install-target.diff by @jollyroger. Jul 17, 2011
websocket.c Rename libb64 to b64 for compatibility reasons, move to DEPS. Nov 11, 2013
websocket.h Removed endian.h Dec 29, 2011
worker.c Fixed indentation and whitespace Jul 7, 2015
worker.h Partial rewrite, adding WebSockets, threads, pool. Apr 8, 2011

README.markdown

About

A very simple web server providing an HTTP interface to Redis. It uses hiredis, jansson, libevent, and http-parser.

Webdis depends on libevent-dev. You can install it on Ubuntu by typing sudo apt-get install libevent-dev or on OS X by typing brew install libevent.

make clean all

./webdis &

curl http://127.0.0.1:7379/SET/hello/world
→ {"SET":[true,"OK"]}

curl http://127.0.0.1:7379/GET/hello
→ {"GET":"world"}

curl -d "GET/hello" http://127.0.0.1:7379/
→ {"GET":"world"}

Features

  • GET and POST are supported, as well as PUT for file uploads.
  • JSON output by default, optional JSONP parameter (?jsonp=myFunction or ?callback=myFunction).
  • Raw Redis 2.0 protocol output with .raw suffix
  • MessagePack output with .msg suffix
  • HTTP 1.1 pipelining (70,000 http requests per second on a desktop Linux machine.)
  • Multi-threaded server, configurable number of worker threads.
  • WebSocket support (Currently using the “hixie-76” specification).
  • Connects to Redis using a TCP or UNIX socket.
  • Restricted commands by IP range (CIDR subnet + mask) or HTTP Basic Auth, returning 403 errors.
  • Possible Redis authentication in the config file.
  • Pub/Sub using Transfer-Encoding: chunked, works with JSONP as well. Webdis can be used as a Comet server.
  • Drop privileges on startup.
  • Custom Content-Type using a pre-defined file extension, or with ?type=some/thing.
  • URL-encoded parameters for binary data or slashes and question marks. For instance, %2f is decoded as / but not used as a command separator.
  • Logs, with a configurable verbosity.
  • Cross-origin requests, usable with XMLHttpRequest2 (Cross-Origin Resource Sharing - CORS).
  • File upload with PUT.
  • With the JSON output, the return value of INFO is parsed and transformed into an object.
  • Optional daemonize: set "daemonize": true and "pidfile": "/var/run/webdis.pid" in webdis.json.
  • Default root object: Add "default_root": "/GET/index.html" in webdis.json to substitute the request to / with a Redis request.
  • HTTP request limit with http_max_request_size (in bytes, set to 128MB by default).
  • Database selection in the URL, using e.g. /7/GET/key to run the command on DB 7.

Ideas, TODO...

  • Add better support for PUT, DELETE, HEAD, OPTIONS? How? For which commands?
    • This could be done using a “strict mode” with a table of commands and the verbs that can/must be used with each command. Strict mode would be optional, configurable. How would webdis know of new commands remains to be determined.
  • MULTI/EXEC/DISCARD/WATCH are disabled at the moment; find a way to use them.
  • Support POST of raw Redis protocol data, and execute the whole thing. This could be useful for MULTI/EXEC transactions.
  • Enrich config file:
    • Provide timeout (maybe for some commands only?). What should the response be? 504 Gateway Timeout? 503 Service Unavailable?
  • Multi-server support, using consistent hashing.
  • SSL?
    • Not sure if this is such a good idea.
  • SPDY?
    • SPDY is mostly useful for parallel fetches. Not sure if it would make sense for Webdis.
  • Send your ideas using the github tracker, on twitter @yowgi or by mail to n.favrefelix@gmail.com.

HTTP error codes

  • Unknown HTTP verb: 405 Method Not Allowed.
  • Redis is unreachable: 503 Service Unavailable.
  • Matching ETag sent using If-None-Match: 304 Not Modified.
  • Could also be used:
    • Timeout on the redis side: 503 Service Unavailable.
    • Missing key: 404 Not Found.
    • Unauthorized command (disabled in config file): 403 Forbidden.

Command format

The URI /COMMAND/arg0/arg1/.../argN.ext executes the command on Redis and returns the response to the client. GET, POST, and PUT are supported:

  • GET /COMMAND/arg0/.../argN.ext
  • POST / with COMMAND/arg0/.../argN in the HTTP body.
  • PUT /COMMAND/arg0.../argN-1 with argN in the HTTP body (see section on file uploads.)

.ext is an optional extension; it is not read as part of the last argument but only represents the output format. Several formats are available (see below).

Special characters: / and . have special meanings, / separates arguments and . changes the Content-Type. They can be replaced by %2f and %2e, respectively.

ACL

Access control is configured in webdis.json. Each configuration tries to match a client profile according to two criterias:

Each ACL contains two lists of commands, enabled and disabled. All commands being enabled by default, it is up to the administrator to disable or re-enable them on a per-profile basis. Examples:

{
    "disabled": ["DEBUG", "FLUSHDB", "FLUSHALL"],
},

{
    "http_basic_auth": "user:password",
    "disabled": ["DEBUG", "FLUSHDB", "FLUSHALL"],
    "enabled":  ["SET"]
},

{
    "ip":       "192.168.10.0/24",
    "enabled":  ["SET"]
},

{
    "http_basic_auth": "user:password",
    "ip":       "192.168.10.0/24",
    "enabled":  ["SET", "DEL"]
}

ACLs are interpreted in order, later authorizations superseding earlier ones if a client matches several. The special value "*" matches all commands.

JSON output

JSON is the default output format. Each command returns a JSON object with the command as a key and the result as a value.

Examples:

// string
$ curl http://127.0.0.1:7379/GET/y
{"GET":"41"}

// number
$ curl http://127.0.0.1:7379/INCR/y
{"INCR":42}

// list
$ curl http://127.0.0.1:7379/LRANGE/x/0/1
{"LRANGE":["abc","def"]}

// status
$ curl http://127.0.0.1:7379/TYPE/y
{"TYPE":[true,"string"]}

// error, which is basically a status
$ curl http://127.0.0.1:7379/MAKE-ME-COFFEE
{"MAKE-ME-COFFEE":[false,"ERR unknown command 'MAKE-ME-COFFEE'"]}

// JSONP callback:
$ curl  "http://127.0.0.1:7379/TYPE/y?jsonp=myCustomFunction"
myCustomFunction({"TYPE":[true,"string"]})

RAW output

This is the raw output of Redis; enable it with the .raw suffix.

// string
$ curl http://127.0.0.1:7379/GET/z.raw
$5
hello

// number
curl http://127.0.0.1:7379/INCR/a.raw
:2

// list
$ curl http://127.0.0.1:7379/LRANGE/x/0/-1.raw
*2
$3
abc
$3
def

// status
$ curl http://127.0.0.1:7379/TYPE/y.raw
+zset

// error, which is basically a status
$ curl http://127.0.0.1:7379/MAKE-ME-COFFEE.raw
-ERR unknown command 'MAKE-ME-COFFEE'

Custom content-type

Several content-types are available:

  • .json for application/json (this is the default Content-Type).
  • .msg for application/x-msgpack. See http://msgpack.org/ for the specs.
  • .txt for text/plain
  • .html for text/html
  • xhtml for application/xhtml+xml
  • xml for text/xml
  • .png for image/png
  • jpg or jpeg for image/jpeg
  • Any other with the ?type=anything/youwant query string.
  • Add a custom separator for list responses with ?sep=, query string.
curl -v "http://127.0.0.1:7379/GET/hello.html"
[...]
< HTTP/1.1 200 OK
< Content-Type: text/html
< Date: Mon, 03 Jan 2011 20:43:36 GMT
< Content-Length: 137
<
<!DOCTYPE html>
<html>
[...]
</html>

curl -v "http://127.0.0.1:7379/GET/hello.txt"
[...]
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Date: Mon, 03 Jan 2011 20:43:36 GMT
< Content-Length: 137
[...]

curl -v "http://127.0.0.1:7379/GET/big-file?type=application/pdf"
[...]
< HTTP/1.1 200 OK
< Content-Type: application/pdf
< Date: Mon, 03 Jan 2011 20:45:12 GMT
[...]

File upload

Webdis supports file upload using HTTP PUT. The command URI is slightly different, as the last argument is taken from the HTTP body. For example: instead of /SET/key/value, the URI becomes /SET/key and the value is the entirety of the body. This works for other commands such as LPUSH, etc.

Uploading a binary file to webdis:

$ file redis-logo.png
redis-logo.png: PNG image, 513 x 197, 8-bit/color RGBA, non-interlaced

$ wc -c redis-logo.png
16744 redis-logo.png

$ curl -v --upload-file redis-logo.png http://127.0.0.1:7379/SET/logo
[...]
> PUT /SET/logo HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: 127.0.0.1:7379
> Accept: */*
> Content-Length: 16744
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Content-Type: application/json
< ETag: "0db1124cf79ffeb80aff6d199d5822f8"
< Date: Sun, 09 Jan 2011 16:48:19 GMT
< Content-Length: 19
<
{"SET":[true,"OK"]}

$ curl -vs http://127.0.0.1:7379/GET/logo.png -o out.png
> GET /GET/logo.png HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: 127.0.0.1:7379
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: image/png
< ETag: "1991df597267d70bf9066a7d11969da0"
< Date: Sun, 09 Jan 2011 16:50:51 GMT
< Content-Length: 16744

$ md5sum redis-logo.png out.png
1991df597267d70bf9066a7d11969da0  redis-logo.png
1991df597267d70bf9066a7d11969da0  out.png

The file was uploaded and re-downloaded properly: it has the same hash and the content-type was set properly thanks to the .png extension.

WebSockets

Webdis supports WebSocket clients implementing dixie-76.
Web Sockets are supported with the following formats, selected by the connection URL:

  • JSON (on / or /.json)
  • Raw Redis wire protocol (on /.raw)

Example:

function testJSON() {
    var jsonSocket = new WebSocket("ws://127.0.0.1:7379/.json");
    jsonSocket.onopen = function() {

        console.log("JSON socket connected!");
        jsonSocket.send(JSON.stringify(["SET", "hello", "world"]));
        jsonSocket.send(JSON.stringify(["GET", "hello"]));
    };
    jsonSocket.onmessage = function(messageEvent) {
        console.log("JSON received:", messageEvent.data);
    };
}
testJSON();

This produces the following output:

JSON socket connected!
JSON received: {"SET":[true,"OK"]}
JSON received: {"GET":"world"}

Pub/Sub with chunked transfer encoding

Webdis exposes Redis PUB/SUB channels to HTTP clients, forwarding messages in the channel as they are published by Redis. This is done using chunked transfer encoding.

Example using XMLHttpRequest:

var previous_response_length = 0
xhr = new XMLHttpRequest()
xhr.open("GET", "http://127.0.0.1:7379/SUBSCRIBE/hello", true);
xhr.onreadystatechange = checkData;
xhr.send(null);

function checkData() {
    if(xhr.readyState == 3)  {
        response = xhr.responseText;
        chunk = response.slice(previous_response_length);
        previous_response_length = response.length;
        console.log(chunk);
    }
};

Publish messages to redis to see output similar to the following:

{"SUBSCRIBE":["subscribe","hello",1]}
{"SUBSCRIBE":["message","hello","some message"]}
{"SUBSCRIBE":["message","hello","some other message"]}