What?

PearlDB is a durable HTTP Key-Value pair database. It uses LMDB for storing data, and H2O for HTTP.

PearlDB is completely written in C.

Persistent connections and pipelining are built-in.

PearlDB uses bmon to batch LMDB writes.

Goals

• Speed
• Low latency
• Durability - An HTTP response means the write is on disk
• Simplicity outside (RESTful inteface)
• Simplicity inside (succinct codebase)
• HTTP caching - Because the CRUD is RESTful you could hypothetically use an HTTP reverse proxy cache to scale out reads. You could use multiple caches to create an eventually consistent database

Ubuntu Quick Start

sudo add-apt-repository -y ppa:willemt/pearldb
sudo apt-get update
sudo apt-get install pearldb

Usage

Examples below make use of the excellent httpie

Starting

sudo build/pearl --daemonize --port 80 --db_size 1
echo daemonizing...

daemonizing...

Get

We obtain a value by GET'ng the key.

In this case the key is "x". But we get a 404 if it doesn't exist.

http -h --ignore-stdin 127.0.0.1/x/

HTTP/1.1 404 NOT FOUND
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
content-length: 0

You MUST specify a path.

http -h --ignore-stdin 127.0.0.1/ | head -n 1

HTTP/1.1 400 BAD PATH

Put

We use PUT for creating or updating a key value pair. PUTs are durable - we only respond when the change has been made to disk.

echo "MY VALUE" | http -h PUT 127.0.0.1/x/

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
transfer-encoding: chunked

PUTs have an immediate change on the resource. There is full isolation, and therefore no dirty reads.

Now we can finally retrieve our data via a GET:

http --ignore-stdin 127.0.0.1/x/

MY VALUE

The slash at the end is optional.

http --ignore-stdin 127.0.0.1/x

MY VALUE

The user must specify the capacity of the database upfront. PearlDB does not support automatic resizing. A PUT will fail if it would put the database over capacity.

head -c 1000000 /dev/urandom | base64 > tmp_file
du -h tmp_file | awk '{ print $1 }'
cat tmp_file | http -h PUT 127.0.0.1/1/
rm tmp_file

1.3M
HTTP/1.1 400 NOT ENOUGH SPACE
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
content-length: 0

You can't PUT under nested resources.

echo 'DATA' | http -h PUT 127.0.0.1/x/nested_resource/

HTTP/1.1 400 BAD PATH
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
content-length: 0

Put without a key (POST)

If you want PearlDB to generate a key for you, just use POST.

echo "MY POSTED VALUE" | http -h POST 127.0.0.1/ > posted.txt
cat posted.txt

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
location: ...
transfer-encoding: chunked

The Location header in the response has the URI of the newly created resource. The URI is the URL safe base64 encoded UUID4.

http --ignore-stdin -b GET 127.0.0.1$(grep location: posted.txt | sed -e 's/location: //' | tr -d '\r\n')

MY POSTED VALUE

Providing a URL (ie. key) with POST doesn't make sense, and will result in a 400.

echo "MY POSTED VALUE" | http -h POST 127.0.0.1/xxxx/

HTTP/1.1 400 BAD
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
content-length: 0

Get keys

You can get the keys that match a prefix by using the /key/XXX/ nested resource.

echo '' | http PUT 127.0.0.1/1/ > /dev/null
echo '' | http PUT 127.0.0.1/199/ > /dev/null
echo '' | http PUT 127.0.0.1/102/ > /dev/null
echo '' | http PUT 127.0.0.1/2/ > /dev/null
http GET 127.0.0.1/key/1/

1
102
199

Without a prefix you get all keys.

http GET 127.0.0.1/key// | sed -e '/^.*=$/d'

1
102
199
2
x

Existence Check

To check for existence use the HEAD method. This is great, because PearlDB doesn't waste bandwidth sending the document body.

http -h --ignore-stdin HEAD 127.0.0.1/x/

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive

Delete

DELETEs are durable - we only respond when the change has been made to disk.

http -h --ignore-stdin DELETE 127.0.0.1/x/

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
transfer-encoding: chunked

Of course, after a DELETE the key doesn't exist anymore:

http -h --ignore-stdin 127.0.0.1/x/

HTTP/1.1 404 NOT FOUND
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
content-length: 0

Compare and Swap (CAS)

A form of opportunistic concurrency control is available through ETags.

When the client provides the Prefers: ETag header on a GET request we generate an ETag. A client can then use the If-Match header with the ETag to perform a conditional update, (ie. a CAS operation). If the ETag has changed then the PUT operation will fail. CAS operations are great because there is no locking; if a CAS operation fails for one client that means it has succeeded for another, ie. there has been progress.

Imagine two clients trying to update the same key. Client 1 requests an ETag. The ETag is provided via the etag header.

echo 'SWEET DATA' | http -h --ignore-stdin PUT 127.0.0.1/x/ > /dev/null
http -h --ignore-stdin GET 127.0.0.1/x/ Prefers:ETag > etag.txt
cat etag.txt

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
etag: ...
transfer-encoding: chunked

If client 1 requests an ETag again, the same ETag is sent:

http -h --ignore-stdin GET 127.0.0.1/x/ Prefers:ETag > etag2.txt
cat etag2.txt
diff <(grep etag etag.txt) <(grep etag etag2.txt)

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
etag: ...
transfer-encoding: chunked

Client 2 does a PUT on x. This will invalidate the ETag.

echo 'SURPRISE' | http -h PUT 127.0.0.1/x/

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
transfer-encoding: chunked

Client 1 uses a conditional PUT to update "x" using the If-Match tag. Because the ETag was invalidated, we don't commit, and respond with 412 Precondition Failed.

echo 'MY NEW VALUE BASED OFF OLD VALUE' | http -h PUT 127.0.0.1/x/ If-Match:$(grep etag: etag.txt | sed -e 's/etag: //' | tr -d '\r\n')

HTTP/1.1 412 BAD ETAG
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
content-length: 0

Once this happens we can retry the PUT after we do a new GET.

http -h GET 127.0.0.1/x/ Prefers:ETag > etag3.txt
cat etag3.txt

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
etag: ...
transfer-encoding: chunked

The PUT will succeed because the ETag is still valid.

echo 'NEW VALUE' | http -h PUT 127.0.0.1/x/ If-Match:$(grep etag: etag3.txt | sed -e 's/etag: //' | tr -d '\r\n')

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
transfer-encoding: chunked

However, if we use the ETag again it will fail.

echo 'NEW VALUE2' | http -h PUT 127.0.0.1/x/ If-Match:$(grep etag: etag3.txt | sed -e 's/etag: //' | tr -d '\r\n')

HTTP/1.1 412 BAD ETAG
Date: ..., ... .... ........ GMT
Server: h2o/1.3.1
Connection: keep-alive
content-length: 0

Notes about ETags:

• On reboots, PearlDB loses all ETag information
• On launch PearlDB generates a random ETag prefix
• ETags are expected to have a short life (ie. < 1 day)

Shutting down

cat /var/run/pearl.pid | sudo xargs kill -9
echo shutdown

shutdown

Building

$ make libuv

$ make libh2o

$ make libck

$ make