Skip to content
This repository

A Redis HTTP interface with JSON output

branch: master

Merge pull request #55 from littlefrog/master

Enable KEEPALIVE socket option
latest commit 00ab5518c3
Nicolas Favre-Felix authored
Octocat-spinner-32 b64 Rename libb64 to b64 for compatibility reasons, move to DEPS. November 11, 2013
Octocat-spinner-32 formats Remove BSON support September 21, 2012
Octocat-spinner-32 hiredis Update hiredis to fd38e55d71 February 17, 2013
Octocat-spinner-32 http-parser Start changing HTTP parser. January 22, 2011
Octocat-spinner-32 jansson Added missing file. December 28, 2010
Octocat-spinner-32 md5 Added ETag and If-None-Match. January 04, 2011
Octocat-spinner-32 sha1 Started implementing the new WebSocket protocol July 20, 2011
Octocat-spinner-32 tests Allow test programs use custom port using environment variable or com… November 11, 2013
Octocat-spinner-32 .gitignore Started pub/sub test. April 13, 2011
Octocat-spinner-32 .travis.yml Fix travis yaml file September 22, 2012
Octocat-spinner-32 COPYING Added missing license file. December 28, 2010
Octocat-spinner-32 Dockerfile Add Dockerfile November 17, 2013
Octocat-spinner-32 Makefile Make dependencies' *_OBJS variables overridable via environment varia… November 11, 2013
Octocat-spinner-32 README.markdown Write pidfile when daemonized (GitHub issue #10) September 07, 2013
Octocat-spinner-32 acl.c Disable SELECT command - issue #80 September 07, 2013
Octocat-spinner-32 acl.h Fix ACLs January 22, 2011
Octocat-spinner-32 client.c Handle memory allocation issues better. February 29, 2012
Octocat-spinner-32 client.h Handle memory allocation issues better. February 29, 2012
Octocat-spinner-32 cmd.c Remove access to free'd memory in cmd_free May 07, 2013
Octocat-spinner-32 cmd.h Create new connection for new separate DBs. June 23, 2012
Octocat-spinner-32 conf.c Rename libb64 to b64 for compatibility reasons, move to DEPS. November 11, 2013
Octocat-spinner-32 conf.h Write pidfile when daemonized (GitHub issue #10) September 07, 2013
Octocat-spinner-32 http.c #89: Add Authorization to CORS headers November 18, 2013
Octocat-spinner-32 http.h Make pub/sub chunks async. July 24, 2011
Octocat-spinner-32 pool.c Free Redis context for custom connections April 22, 2013
Octocat-spinner-32 pool.h Free Redis context for custom connections April 22, 2013
Octocat-spinner-32 server.c Merge pull request #55 from littlefrog/master November 26, 2013
Octocat-spinner-32 server.h Fixed log buf, removed spin lock. April 21, 2011
Octocat-spinner-32 slog.c Remove comments mentioning borrowing code from redis repository. November 11, 2013
Octocat-spinner-32 slog.h Reopen logs on SIGHUP (GitHub issue #10) September 07, 2013
Octocat-spinner-32 version.h Version bump after release November 17, 2013
Octocat-spinner-32 webdis.c Partial rewrite, adding WebSockets, threads, pool. April 08, 2011
Octocat-spinner-32 webdis.json Handle SIGTERM & SIGINT (issue #10) September 07, 2013
Octocat-spinner-32 webdis.prod.json Added fix-config.diff and install-target.diff by @jollyroger. July 17, 2011
Octocat-spinner-32 websocket.c Rename libb64 to b64 for compatibility reasons, move to DEPS. November 11, 2013
Octocat-spinner-32 websocket.h Removed endian.h December 29, 2011
Octocat-spinner-32 worker.c #91: Add missing return after sending HTTP Options November 26, 2013
Octocat-spinner-32 worker.h Partial rewrite, adding WebSockets, threads, pool. April 08, 2011
README.markdown

About

A very simple web server providing an HTTP interface to Redis. It uses hiredis, jansson, libevent, and http-parser.

Webdis depends on libevent-dev. You can install it on Ubuntu by typing sudo apt-get install libevent-dev or on OS X by typing brew install libevent.

make clean all

./webdis &

curl http://127.0.0.1:7379/SET/hello/world
→ {"SET":[true,"OK"]}

curl http://127.0.0.1:7379/GET/hello
→ {"GET":"world"}

curl -d "GET/hello" http://127.0.0.1:7379/
→ {"GET":"world"}

Features

  • GET and POST are supported, as well as PUT for file uploads.
  • JSON output by default, optional JSONP parameter (?jsonp=myFunction or ?callback=myFunction).
  • Raw Redis 2.0 protocol output with .raw suffix
  • MessagePack output with .msg suffix
  • HTTP 1.1 pipelining (70,000 http requests per second on a desktop Linux machine.)
  • Multi-threaded server, configurable number of worker threads.
  • WebSocket support (Currently using the “hixie-76” specification).
  • Connects to Redis using a TCP or UNIX socket.
  • Restricted commands by IP range (CIDR subnet + mask) or HTTP Basic Auth, returning 403 errors.
  • Possible Redis authentication in the config file.
  • Pub/Sub using Transfer-Encoding: chunked, works with JSONP as well. Webdis can be used as a Comet server.
  • Drop privileges on startup.
  • Custom Content-Type using a pre-defined file extension, or with ?type=some/thing.
  • URL-encoded parameters for binary data or slashes and question marks. For instance, %2f is decoded as / but not used as a command separator.
  • Logs, with a configurable verbosity.
  • Cross-origin requests, usable with XMLHttpRequest2 (Cross-Origin Resource Sharing - CORS).
  • File upload with PUT.
  • With the JSON output, the return value of INFO is parsed and transformed into an object.
  • Optional daemonize: set "daemonize": true and "pidfile": "/var/run/webdis.pid" in webdis.json.
  • Default root object: Add "default_root": "/GET/index.html" in webdis.json to substitute the request to / with a Redis request.
  • HTTP request limit with http_max_request_size (in bytes, set to 128MB by default).
  • Database selection in the URL, using e.g. /7/GET/key to run the command on DB 7.

Ideas, TODO...

  • Add better support for PUT, DELETE, HEAD, OPTIONS? How? For which commands?
    • This could be done using a “strict mode” with a table of commands and the verbs that can/must be used with each command. Strict mode would be optional, configurable. How would webdis know of new commands remains to be determined.
  • MULTI/EXEC/DISCARD/WATCH are disabled at the moment; find a way to use them.
  • Support POST of raw Redis protocol data, and execute the whole thing. This could be useful for MULTI/EXEC transactions.
  • Enrich config file:
    • Provide timeout (maybe for some commands only?). What should the response be? 504 Gateway Timeout? 503 Service Unavailable?
  • Multi-server support, using consistent hashing.
  • SSL?
    • Not sure if this is such a good idea.
  • SPDY?
    • SPDY is mostly useful for parallel fetches. Not sure if it would make sense for Webdis.
  • Send your ideas using the github tracker, on twitter @yowgi or by mail to n.favrefelix@gmail.com.

HTTP error codes

  • Unknown HTTP verb: 405 Method Not Allowed.
  • Redis is unreachable: 503 Service Unavailable.
  • Matching ETag sent using If-None-Match: 304 Not Modified.
  • Could also be used:
    • Timeout on the redis side: 503 Service Unavailable.
    • Missing key: 404 Not Found.
    • Unauthorized command (disabled in config file): 403 Forbidden.

Command format

The URI /COMMAND/arg0/arg1/.../argN.ext executes the command on Redis and returns the response to the client. GET, POST, and PUT are supported:

  • GET /COMMAND/arg0/.../argN.ext
  • POST / with COMMAND/arg0/.../argN in the HTTP body.
  • PUT /COMMAND/arg0.../argN-1 with argN in the HTTP body (see section on file uploads.)

.ext is an optional extension; it is not read as part of the last argument but only represents the output format. Several formats are available (see below).

Special characters: / and . have special meanings, / separates arguments and . changes the Content-Type. They can be replaced by %2f and %2e, respectively.

ACL

Access control is configured in webdis.json. Each configuration tries to match a client profile according to two criterias:

Each ACL contains two lists of commands, enabled and disabled. All commands being enabled by default, it is up to the administrator to disable or re-enable them on a per-profile basis. Examples:

{
    "disabled": ["DEBUG", "FLUSHDB", "FLUSHALL"],
},

{
    "http_basic_auth": "user:password",
    "disabled": ["DEBUG", "FLUSHDB", "FLUSHALL"],
    "enabled":  ["SET"]
},

{
    "ip":       "192.168.10.0/24",
    "enabled":  ["SET"]
},

{
    "http_basic_auth": "user:password",
    "ip":       "192.168.10.0/24",
    "enabled":  ["SET", "DEL"]
}

ACLs are interpreted in order, later authorizations superseding earlier ones if a client matches several. The special value "*" matches all commands.

JSON output

JSON is the default output format. Each command returns a JSON object with the command as a key and the result as a value.

Examples:

// string
$ curl http://127.0.0.1:7379/GET/y
{"GET":"41"}

// number
$ curl http://127.0.0.1:7379/INCR/y
{"INCR":42}

// list
$ curl http://127.0.0.1:7379/LRANGE/x/0/1
{"LRANGE":["abc","def"]}

// status
$ curl http://127.0.0.1:7379/TYPE/y
{"TYPE":[true,"string"]}

// error, which is basically a status
$ curl http://127.0.0.1:7379/MAKE-ME-COFFEE
{"MAKE-ME-COFFEE":[false,"ERR unknown command 'MAKE-ME-COFFEE'"]}

// JSONP callback:
$ curl  "http://127.0.0.1:7379/TYPE/y?jsonp=myCustomFunction"
myCustomFunction({"TYPE":[true,"string"]})

RAW output

This is the raw output of Redis; enable it with the .raw suffix.


// string
$ curl http://127.0.0.1:7379/GET/z.raw
$5
hello

// number
curl http://127.0.0.1:7379/INCR/a.raw
:2

// list
$ curl http://127.0.0.1:7379/LRANGE/x/0/-1.raw
*2
$3
abc
$3
def

// status
$ curl http://127.0.0.1:7379/TYPE/y.raw
+zset

// error, which is basically a status
$ curl http://127.0.0.1:7379/MAKE-ME-COFFEE.raw
-ERR unknown command 'MAKE-ME-COFFEE'

Custom content-type

Several content-types are available:

  • .json for application/json (this is the default Content-Type).
  • .msg for application/x-msgpack. See http://msgpack.org/ for the specs.
  • .txt for text/plain
  • .html for text/html
  • xhtml for application/xhtml+xml
  • xml for text/xml
  • .png for image/png
  • jpg or jpeg for image/jpeg
  • Any other with the ?type=anything/youwant query string.
  • Add a custom separator for list responses with ?sep=, query string.
curl -v "http://127.0.0.1:7379/GET/hello.html"
[...]
< HTTP/1.1 200 OK
< Content-Type: text/html
< Date: Mon, 03 Jan 2011 20:43:36 GMT
< Content-Length: 137
<
<!DOCTYPE html>
<html>
[...]
</html>

curl -v "http://127.0.0.1:7379/GET/hello.txt"
[...]
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Date: Mon, 03 Jan 2011 20:43:36 GMT
< Content-Length: 137
[...]

curl -v "http://127.0.0.1:7379/GET/big-file?type=application/pdf"
[...]
< HTTP/1.1 200 OK
< Content-Type: application/pdf
< Date: Mon, 03 Jan 2011 20:45:12 GMT
[...]

File upload

Webdis supports file upload using HTTP PUT. The command URI is slightly different, as the last argument is taken from the HTTP body. For example: instead of /SET/key/value, the URI becomes /SET/key and the value is the entirety of the body. This works for other commands such as LPUSH, etc.

Uploading a binary file to webdis:

$ file redis-logo.png
redis-logo.png: PNG image, 513 x 197, 8-bit/color RGBA, non-interlaced

$ wc -c redis-logo.png
16744 redis-logo.png

$ curl -v --upload-file redis-logo.png http://127.0.0.1:7379/SET/logo
[...]
> PUT /SET/logo HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: 127.0.0.1:7379
> Accept: */*
> Content-Length: 16744
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Content-Type: application/json
< ETag: "0db1124cf79ffeb80aff6d199d5822f8"
< Date: Sun, 09 Jan 2011 16:48:19 GMT
< Content-Length: 19
<
{"SET":[true,"OK"]}

$ curl -vs http://127.0.0.1:7379/GET/logo.png -o out.png
> GET /GET/logo.png HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: 127.0.0.1:7379
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: image/png
< ETag: "1991df597267d70bf9066a7d11969da0"
< Date: Sun, 09 Jan 2011 16:50:51 GMT
< Content-Length: 16744

$ md5sum redis-logo.png out.png
1991df597267d70bf9066a7d11969da0  redis-logo.png
1991df597267d70bf9066a7d11969da0  out.png

The file was uploaded and re-downloaded properly: it has the same hash and the content-type was set properly thanks to the .png extension.

WebSockets

Webdis supports WebSocket clients implementing dixie-76.
Web Sockets are supported with the following formats, selected by the connection URL:

  • JSON (on / or /.json)
  • Raw Redis wire protocol (on /.raw)

Example:

function testJSON() {
    var jsonSocket = new WebSocket("ws://127.0.0.1:7379/.json");
    jsonSocket.onopen = function() {

        console.log("JSON socket connected!");
        jsonSocket.send(JSON.stringify(["SET", "hello", "world"]));
        jsonSocket.send(JSON.stringify(["GET", "hello"]));
    };
    jsonSocket.onmessage = function(messageEvent) {
        console.log("JSON received:", messageEvent.data);
    };
}
testJSON();

This produces the following output:

JSON socket connected!
JSON received: {"SET":[true,"OK"]}
JSON received: {"GET":"world"}

Pub/Sub with chunked transfer encoding

Webdis exposes Redis PUB/SUB channels to HTTP clients, forwarding messages in the channel as they are published by Redis. This is done using chunked transfer encoding.

Example using XMLHttpRequest:

var previous_response_length = 0
xhr = new XMLHttpRequest()
xhr.open("GET", "http://127.0.0.1:7379/SUBSCRIBE/hello", true);
xhr.onreadystatechange = checkData;
xhr.send(null);

function checkData() {
    if(xhr.readyState == 3)  {
        response = xhr.responseText;
        chunk = response.slice(previous_response_length);
        previous_response_length = response.length;
        console.log(chunk);
    }
};

Publish messages to redis to see output similar to the following:

{"SUBSCRIBE":["subscribe","hello",1]}
{"SUBSCRIBE":["message","hello","some message"]}
{"SUBSCRIBE":["message","hello","some other message"]} 
Something went wrong with that request. Please try again.