and cliff said "let there be documentation" and it was good

README.mdown

@@ -0,0 +1,105 @@
+Luwak
+=========
+
+The Riak key/value store is extremely fast and reliable, but has limitations on the size of object that preclude its use for some applications.  For example, storing high definition still images or video is difficult or impossible.  Luwak is a service layer on Riak that provides a simple, file-oriented abstraction for enormous objects.
+
+Examples
+========
+
+Synchronous API
+--------
+
+Creating files and reading and writing to them.
+
+    {ok, File} = luwak_file:create(Riak, <<"filename">>, dict:new()),
+    {ok, _, File1} = luwak_io:put_range(Riak, File, 0, <<"myfilecontents">>),
+    IOList = luwak_io:get_range(Riak, File1, 0, 15),
+
+Files have arbitrary metadata called 'attributes'.  They are stored as a dictionary along with your file and can store just about any kind of metadata you want to associate with your file.
+
+    {ok, File2} = luwak_file:set_attributes(Riak, File1, dict:store(mykey, myvalue, dict:new())),
+
+You can also truncate a file.
+
+    {ok, File3} = luwak_io:truncate(Riak, File1, 0),
+
+Or delete it altogether.
+
+    ok = luwak_file:delete(Riak, <<"filename">>),
+
+You can test for the existence of a file.
+
+    {ok, false} = luwak_file:exists(Riak, <<"filename">>).
+
+Asynchronous API
+--------
+
+Luwak can also operate asynchronously, allowing you to stream your reads and writes to and from files, allowing you to achieve much higher bandwidth than the synchronous API is capable of.
+
+    {ok, File} = luwak_file:create(Riak, <<"mystreamingfile">>, dict:new()),
+    PutStream = luwak_put_stream:start(Riak, File, 0, 1000),
+    luwak_put_stream:send(PutStream, <<"mymilkshake">>),
+    luwak_put_stream:send(PutStream, <<"bringsalltheboy">>),
+    luwak_put_stream:send(PutStream, <<"totheyard">>),
+    luwak_put_stream:close(PutStream),
+    {ok, File1} = luwak_put_stream:status(File),
+
+When we call close, it forces the put stream to flush the contents of the file.  Check out the API docs for luwak_put_stream to get a feel for how the streaming write API operates.
+
+    GetStream = luwak_get_stream:start(Riak, File1, 0, 36),
+    {<<"mymilkshakebringsalltheboystotheyard">>, 0} = luwak_get_stream:recv(GetStream, 1000),
+    eos = luwak_get_stream(GetStream, 1000).
+
+Opening a get stream actually kicks off a riak map reduce job.  The job will recurse down the tree, following links until it gets to the actual data blocks.  It will then send all of these data blocks to the intermediary get stream process.  While it is possible to close a get stream before it is finished, there is no way currently to cancel the map reduce job in process.  So when you close a get stream before it is finished, the intermediary process exits and Erlang ought to drop the messages bound for it from the map reduce job.
+
+Operational Considerations
+=========
+
+Luwak is purely library code.  It does not have a supervisor tree nor does it need to be started like a tradition erlang application.  Luwak only requires that its code be on the load path of the client as well as on the load path of all of your Riak nodes.
+
+Let It Crash Design
+--------
+
+Luwak is designed according to Erlang's "let it crash" design philosophy.  Most of the publicly accessible functions in Luwak will intentionally crash in the case of a failure.  Luwak, however, will never corrupt your files.  If a write operation is aborted at any step before it returns to the user then no changes will have actually occurred to the file.  Likewise, concurrent reads can happen to a file during write operations.  If the reads start before the write is completed then they will simply read the previous version of the file.
+
+Internal Architecture
+=====================
+
+Luwak stores large objects in Riak as a metadata document, named for the object being stored, and a sequence of immutable block documents (henceforth referred to simple as 'blocks'), each of which is named with a hash of its contents.  All blocks for an object are the same size, except the last which may be shorter to accommodate objects of lengths not evenly divisible by the block size.  All hash operations use Skein 512.
+
+When a new object is passed to Luwak for storage in Riak, a new metadata document is created for it.  This metadata document uses riak's internal concept of links in order to link to a top-level document describing a merkle tree for the data contents of the object.
+
+		{<<"luwak_tld">>, <<"file001">>}: {
+      "tree_order": 100,
+			"block_size": 1000000,
+			"created": <timestamp>,
+			"modified": <timestamp>,
+			"ancestors": [<hashes>],
+		  "root": <<"38d0f91a99c57d189416439ce377ccdcd92639d0">>
+		}
+		
+			1. Example top level document
+			
+		{<<"luwak_node">>, <<"38d0f91a99c57d189416439ce377ccdcd92639d0">>} : {
+			"created": <timestamp>,
+			"children": [children]
+		}
+		
+		2. Example of a node document
+		
+		{<<"luwak_node">>, <<"46dfea8f78ddbebfc12f5ff822c7ae5bbbc4f638">>} : {
+		  "created": <timestamp>,
+			"data": <<bytes>>
+		}
+
+		3. Example data document
+
+Node and block documents are by definition immutable in luwak.  This is because their key is computed according to their contents.  Therefore, in order to mutate the contents of a file a new tree must be created.  As with most immutable data structures, this can be accomplished by only computing new nodes and trees for the parts which were actually changed, and keeping references to the subtrees which stay the same.
+
+This may seem slow, however it has several benefits:
+
+Breaks up the read - update - write cycle that is necessary with most KV stores.  Assuming that a node document's child hashes are already in memory, or a data document's data is already in memory, we need merely execute a create - write cycle, cutting a database round-trip out of the picture.  The only part of luwak that mutates is the top level document, and this is only to update the root tree hash.
+
+Efficient versioning and conflict detection of luwak objects.  Two documents which have many common blocks can share storage for those blocks.  Showing a diff via the interface then merely becomes an exercise in tree walking.  Conflict resolution is even easier.  When a conflict is detected between 2 TLD's in riak, the conflict resolution code must merely add the hash of the older trees to the ancestors list.
+
+Tree density is kept optimal.  Luwak trees are immutable and overwrite only, which means that a tree can only be appended to or overwritten.  This allows us to tune the B+Tree algorithm to keep luwak tree nodes completely full except for the very tail of the tree.  Optimally full tree nodes means that retrieval is guaranteed to only have to travel down log(n) / log(tree_order) levels before reaching the requested block.

apps/luwak/doc/edoc-info

@@ -0,0 +1,4 @@
+{application,luwak}.
+{packages,[]}.
+{modules,[luwak_app,luwak_block,luwak_file,luwak_get_stream,luwak_io,
+          luwak_put_stream,luwak_sup,luwak_tree,luwak_tree_utils]}.

apps/luwak/doc/index.html

@@ -0,0 +1,17 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+<head>
+<title>The luwak application</title>
+</head>
+<frameset cols="20%,80%">
+<frame src="modules-frame.html" name="modulesFrame" title="">
+
+<frame src="overview-summary.html" name="overviewFrame" title="">
+<noframes>
+<h2>This page uses frames</h2>
+<p>Your browser does not accept frames.
+<br>You should go to the <a href="overview-summary.html">non-frame version</a> instead.
+</p>
+</noframes>
+</frameset>
+</html>

apps/luwak/doc/luwak_app.html

@@ -0,0 +1,37 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+<head>
+<title>Module luwak_app</title>
+<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
+</head>
+<body bgcolor="white">
+<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
+<hr>
+
+<h1>Module luwak_app</h1>
+<ul class="index"><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>
+
+<p><b>Behaviours:</b> <a href="application.html"><tt>application</tt></a>.</p>
+
+<h2><a name="index">Function Index</a></h2>
+<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#start-2">start/2</a></td><td></td></tr>
+<tr><td valign="top"><a href="#stop-1">stop/1</a></td><td></td></tr>
+</table>
+
+<h2><a name="functions">Function Details</a></h2>
+
+<h3 class="function"><a name="start-2">start/2</a></h3>
+<div class="spec">
+<p><tt>start(StartType, StartArgs) -&gt; any()</tt></p>
+</div>
+
+<h3 class="function"><a name="stop-1">stop/1</a></h3>
+<div class="spec">
+<p><tt>stop(State) -&gt; any()</tt></p>
+</div>
+<hr>
+
+<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
+<p><i>Generated by EDoc, May 31 2010, 11:13:39.</i></p>
+</body>
+</html>

apps/luwak/doc/luwak_block.html

@@ -0,0 +1,42 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+<head>
+<title>Module luwak_block</title>
+<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
+</head>
+<body bgcolor="white">
+<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
+<hr>
+
+<h1>Module luwak_block</h1>
+<ul class="index"><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>
+
+
+<h2><a name="index">Function Index</a></h2>
+<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#create-2">create/2</a></td><td></td></tr>
+<tr><td valign="top"><a href="#data-1">data/1</a></td><td></td></tr>
+<tr><td valign="top"><a href="#name-1">name/1</a></td><td></td></tr>
+</table>
+
+<h2><a name="functions">Function Details</a></h2>
+
+<h3 class="function"><a name="create-2">create/2</a></h3>
+<div class="spec">
+<p><tt>create(Riak, Data) -&gt; any()</tt></p>
+</div>
+
+<h3 class="function"><a name="data-1">data/1</a></h3>
+<div class="spec">
+<p><tt>data(Val) -&gt; any()</tt></p>
+</div>
+
+<h3 class="function"><a name="name-1">name/1</a></h3>
+<div class="spec">
+<p><tt>name(Object) -&gt; any()</tt></p>
+</div>
+<hr>
+
+<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
+<p><i>Generated by EDoc, May 31 2010, 11:13:39.</i></p>
+</body>
+</html>

apps/luwak/doc/luwak_file.html

@@ -0,0 +1,93 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+<head>
+<title>Module luwak_file</title>
+<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
+</head>
+<body bgcolor="white">
+<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
+<hr>
+
+<h1>Module luwak_file</h1>
+<ul class="index"><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>
+
+
+<h2><a name="index">Function Index</a></h2>
+<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#create-3">create/3</a></td><td>Create a luwak file handle with the given name and attributes.</td></tr>
+<tr><td valign="top"><a href="#create-4">create/4</a></td><td>Create a luwak file handle with the given name and attributes.</td></tr>
+<tr><td valign="top"><a href="#delete-2">delete/2</a></td><td>deletes the named file from luwak.</td></tr>
+<tr><td valign="top"><a href="#exists-2">exists/2</a></td><td>Checks for the existence of the named file.</td></tr>
+<tr><td valign="top"><a href="#get-2">get/2</a></td><td>returns a filehandle for the named file.</td></tr>
+<tr><td valign="top"><a href="#get_attributes-1">get_attributes/1</a></td><td>Gets the attribute dictionary from the file handle.</td></tr>
+<tr><td valign="top"><a href="#get_property-2">get_property/2</a></td><td>retrieves the named property from the filehandle.</td></tr>
+<tr><td valign="top"><a href="#length-2">length/2</a></td><td>returns the length in bytes of the file.</td></tr>
+<tr><td valign="top"><a href="#name-1">name/1</a></td><td>returns the name of the given file handle.</td></tr>
+<tr><td valign="top"><a href="#set_attributes-3">set_attributes/3</a></td><td>Sets the new attributes, saves them, and returns a new file handle.</td></tr>
+</table>
+
+<h2><a name="functions">Function Details</a></h2>
+
+<h3 class="function"><a name="create-3">create/3</a></h3>
+<div class="spec">
+<p><tt>create(Riak::<a href="#type-riak">riak()</a>, Name::binary(), Attributes::<a href="#type-dict">dict()</a>) -&gt; {ok, File::<a href="#type-luwak_file">luwak_file()</a>}</tt></p>
+</div><p>Equivalent to <a href="#create-4"><tt>create(Riak, Name, [], Attributes)</tt></a>.</p>
+<p>Create a luwak file handle with the given name and attributes.  Will
+       overwrite an existing file of the same name.</p>
+
+<h3 class="function"><a name="create-4">create/4</a></h3>
+<div class="spec">
+<p><tt>create(Riak::<a href="#type-riak">riak()</a>, Name::binary(), Properties::<a href="#type-proplist">proplist()</a>, Attributes::<a href="#type-dict">dict()</a>) -&gt; {ok, File::<a href="#type-luwak_file">luwak_file()</a>}</tt></p>
+</div><p>Create a luwak file handle with the given name and attributes.
+       Recognized properties:
+       {block_size, int()} - The maximum size of an individual data chunk in bytes.  Default
+                             is 1000000.
+       {tree_order, int()} - The maximum number of children for an individual tree node. Default
+                             is 250.</p>
+
+<h3 class="function"><a name="delete-2">delete/2</a></h3>
+<div class="spec">
+<p><tt>delete(Riak::<a href="#type-riak">riak()</a>, Name::binary()) -&gt; ok | {error, Reason}</tt></p>
+</div><p>deletes the named file from luwak.  This is a fast operation since
+       the blocks and tree for that file remain untouched.  A GC operation
+       (not yet implemented) will be required to clean them up properly.</p>
+
+<h3 class="function"><a name="exists-2">exists/2</a></h3>
+<div class="spec">
+<p><tt>exists(Riak::<a href="#type-riak">riak()</a>, Name::binary()) -&gt; {ok, true} | {ok, false} | {error, Reason}</tt></p>
+</div><p>Checks for the existence of the named file.</p>
+
+<h3 class="function"><a name="get-2">get/2</a></h3>
+<div class="spec">
+<p><tt>get(Riak::<a href="#type-riak">riak()</a>, Name::binary()) -&gt; {ok, File} | {error, Reason}</tt></p>
+</div><p>returns a filehandle for the named file.</p>
+
+<h3 class="function"><a name="get_attributes-1">get_attributes/1</a></h3>
+<div class="spec">
+<p><tt>get_attributes(Obj::<a href="#type-luwak_file">luwak_file()</a>) -&gt; <a href="#type-dict">dict()</a></tt></p>
+</div><p>Gets the attribute dictionary from the file handle.</p>
+
+<h3 class="function"><a name="get_property-2">get_property/2</a></h3>
+<div class="spec">
+<p><tt>get_property(Obj::<a href="#type-luwak_file">luwak_file()</a>, PropName::atom()) -&gt; Property</tt></p>
+</div><p>retrieves the named property from the filehandle.</p>
+
+<h3 class="function"><a name="length-2">length/2</a></h3>
+<div class="spec">
+<p><tt>length(Riak::<a href="#type-riak">riak()</a>, File::<a href="#type-luwak_file">luwak_file()</a>) -&gt; Length</tt></p>
+</div><p>returns the length in bytes of the file.</p>
+
+<h3 class="function"><a name="name-1">name/1</a></h3>
+<div class="spec">
+<p><tt>name(Obj::<a href="#type-luwak_file">luwak_file()</a>) -&gt; binary()</tt></p>
+</div><p>returns the name of the given file handle.</p>
+
+<h3 class="function"><a name="set_attributes-3">set_attributes/3</a></h3>
+<div class="spec">
+<p><tt>set_attributes(Riak::<a href="#type-riak">riak()</a>, Obj::<a href="#type-luwak_file">luwak_file()</a>, Attributes::<a href="#type-dict">dict()</a>) -&gt; {ok, NewFile}</tt></p>
+</div><p>Sets the new attributes, saves them, and returns a new file handle.</p>
+<hr>
+
+<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
+<p><i>Generated by EDoc, May 31 2010, 11:13:39.</i></p>
+</body>
+</html>