REST API Proposal

Voldemort REST API

There are multiple benefits of the REST API:

1. Ease of development, espeically for non-production projects (e.g., a viewer tool for prototype data sets).
2. Ease of building clients in new languages. This is especially true for scripting languages where performance is not a significant concern in the first place.
3. Operational tooling: operations can use this in place of the client shell, especially for automated validation. They can simply use curl to query the stores.

The API is based on Riak, but with some modifications due to differences between Voldemort and Riak and “if we were to do this again, we would…” discussions with Riak developers.

Passing vector clocks

To pass a vector clock (from client to server and server to client), we will use an X-Voldemort-VClock header, containing the representation of the vector clock as a JSON string.

The other alternative is to use an ETag.

GETs

Naturally, a Voldemort GET maps onto an HTTP GET. The URL is of the format /v1/{storename}/{key}. In the case of a compound key, the key can be represent as JSON (see the discussion the Serialization section). GET responses are returned with an X-Voldemort-VClock header set.

Resolving conflicts

With partial causal ordering, it is possible that there are multiple versions of a value will exist that can not be automatically reconciled. Riak uses mime multi-part for this, however this is mainly done for a historical reason.

A more restful approach for this is to use the HTTP-300 response: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.1

In this case, we will use a separate URL to represent each version (with a checksum of the vector clock as part of the url e.g., /v1/storename/key/deadbeef).

PUTs

• To do a clobbering put use a PUT without the X-Voldemort-VClock header.

• To do a conditional put, use a PUT with X-Voldemort-VClock header set. In the case of a new value being created, use an X-Voldemort-VClock header e.g., {}

Multi-get

Use multi-part/related with Content-ID header for each part. X-Voldemort-VClock header becomes a (JSON-encoded) map, with a content-id for each value part pointing to a corresponding vector clock.

Serialization

We could do this opaquely, but this makes it difficult for clients written in scripting languages (or ops people using curl) to interact with stores using Protocol Buffers or Voldemort’s binary JSON.

Instead, a more reasonable approach might be to use something akin to Jackson’s POJO mapper to map already deserialized (via the Java based serializer) objects to JSON. To avoid additional reflection, we could also write direct translators for the most common (Avro, binary JSON) serializers.

Clients can also pass their own Accept headers to specify which content-type they’d like to use i.e., whether they want to receive opaque binary data or JSON.

Contributions

Thanks to Coda Hale and Justin Sheehy for their suggestions.