transforming.html

<title>Rendering Content Based-On Multiple Documents with List Functions</title>

<meta charset="utf-8">

<link rel="stylesheet" href="../style.css">

<link rel="prev" href="show.html">

<link rel="next" href="standalone.html">

<script src="../script.js"></script>

<h2 id="transforming">Rendering Content Based-On Multiple Documents with List Functions</h2>

<p>Just as <em>show  functions</em> convert an <strong>individual</strong> document into an arbitrary output format, CouchDB <em>list functions</em> are used to render documents as a <strong>group</strong>. 

<p>A list function is invoked with a URL specifying both the list function name and also the underlying view which will provide and organize the data.

<p>For example, given this simple view, which indexes documents by "user_id":
<pre>
http://mysite/mydb/_design/myapp/_view/mydocs-by-user
</pre>
<pre>
function(doc) {
  if (doc.user_id) {
    emit(doc.user_id, null);
  }
};
</pre>


<p>
This list function renders the list of users in a simple HTML page:
<pre>
http://mysite/mydb/_design/myapp/_list/mylist/mydocs-by-user
</pre>
<pre>
function(head, req) {
    provides("html", function() {
        html = "&lthtml&gt&ltbody&gt&ltol&gt\n";
        while (row = getRow()) {
            html += "&ltli&gt" + row.key + ":" + row.value + "&lt/li&gt\n";
        }   
        html += "&lt/ol&gt&lt/body&gt&lt/html&gt";
        return html;
    }); 
}
</pre>

<p>Note that a list function can be used with a variety of views, or might be tailored to produce an elaborate page from a view designed specifically to organize data for it.

<p>The powerful iterator API allows for flexibility to filter and aggregate rows on the fly, as well as output raw transformations for an easy way to make Atom feeds, HTML lists, CSV files, config files, or even just modified JSON.

<p>List functions are stored under the <code>lists</code> field of a design document. Here’s an example design document that contains two list functions:

<pre>
{
  "_id" : "_design/foo",
  "_rev" : "1-67aGt7bg",
  "lists" : {
    "bar" : "function(head, req) { var row; while (row = getRow()) { ... } }",
    "zoom" : "function() { return 'zoom!' }",
  }
}
</pre>

<h3 id="arguments">Arguments to the List Function</h3>

<p>The function is called with two arguments, which can sometimes be ignored, as the row data itself is loaded during function execution. The first argument, <code>head</code>, contains information about the view. Here’s what you might see looking at a JSON representation of <code>head</code>:

<pre>
{total_rows:10, offset:0}
</pre>

<p>The request itself is a much richer data structure. This is the same request object that is available to show, update, and filter functions. We’ll go through it in detail here as a reference. Here’s the example <code>req</code> object:

<pre>
{
  "info": {
    "db_name": "test_suite_db","doc_count": 11,"doc_del_count": 0,
    "update_seq": 11,"purge_seq": 0,"compact_running": false,"disk_size": 4930,
    "instance_start_time": "1250046852578425","disk_format_version": 4},
</pre>

<p>The database information, as available in an information request against a database’s URL, is included in the request parameters. This allows you to stamp rendered rows with an update sequence and know the database you are working with.

<pre>
  "method": "GET",
  "path": ["test_suite_db","_design","lists","_list","basicJSON","basicView"],
</pre>

<p>The HTTP method and the path in the client from the client request are useful, especially for rendering links to other resources within the application.

<pre>
  "query": {"foo":"bar"},
</pre>

<p>If there are parameters in the query string (in this case corresponding to <code>?foo=bar</code>), they will be parsed and available as a JSON object at <code>req.query</code>.

<pre>
  "headers":
    {"Accept": "text/html,application/xhtml+xml ,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7","Accept-Encoding":
    "gzip,deflate","Accept-Language": "en-us,en;q=0.5","Connection": "keep-alive",
    "Cookie": "_x=95252s.sd25; AuthSession=","Host": "127.0.0.1:5984",
    "Keep-Alive": "300",
    "Referer": "http://127.0.0.1:5984/_utils/couch_tests.html?script/couch_tests.js",
    "User-Agent": "Mozilla/5.0 Gecko/20090729 Firefox/3.5.2"},
  "cookie": {"_x": "95252s.sd25","AuthSession": ""},
</pre>

<p>Headers give list and show functions the ability to provide the Content-Type response that the client prefers, as well as other nifty things like cookies. Note that cookies are also parsed into a JSON representation. Thanks, MochiWeb!

<pre>
  "body": "undefined",
  "form": {},
</pre>

<p>In the case where the method is <code>POST</code>, the request body (and a form-decoded JSON representation of it, if applicable) are available as well.

<pre>
  "userCtx": {"db": "test_suite_db","name": null,"roles": ["_admin"]}
}
</pre>

<p>Finally, the <code>userCtx</code> is the same as that sent to the validation function. It provides access to the database the user is authenticated against, the user’s name, and the roles they’ve been granted. In the previous example, you see an anonymous user working with a CouchDB node that is in “admin party” mode. Unless an admin is specified, everyone is an admin.

<p>That’s enough about the arguments to list functions. Now it’s time to look at the mechanics of the function itself.

<h3 id="example">An Example List Function</h3>

<p>Let’s put this knowledge to use. In the chapter introduction, we mentioned using lists to generate config files. One fun thing about this is that if you keep your configuration information in CouchDB and generate it with lists, you don’t have to worry about being able to regenerate it again, because you know the config will be generated by a pure function from your database and not other sources of information. This level of isolation will ensure that your config files can be generated correctly as long as CouchDB is running. Because you can’t fetch data from other system services, files, or network sources, you can’t accidentally write a config file generator that fails due to external factors.

<div class="aside note">

<p>J. Chris got excited about the idea of using list functions to generate config files for the sort of services people usually configure using CouchDB, specifically via Chef, an Apache-licensed infrastructure automation tool. The key feature of infrastructure automation is that deployment scripts are idempotent—that is, running your scripts multiple times will have the same intended effect as running them once, something that becomes critical when a script fails halfway through. This encourages crash-only design, where your scripts can bomb out multiple times but your data remains consistent, because it takes the guesswork out of provisioning and updating servers in the case of previous failures.

<p>Like map, reduce, and show functions, lists are pure functions, from a view query and an HTTP request to an output format. They can’t make queries against remote services or otherwise access outside data, so you know they are repeatable. Using a list function to generate an HTTP server configuration file ensures that the configuration is generated repeatably, based on only the state of the database.

</div>

<p>Imagine you are running a shared hosting platform, with one name-based virtual host per user. You’ll need a config file that starts out with some node configuration (which modules to use, etc.) and is followed by one config section per user, setting things like the user’s HTTP directory, subdomain, forwarded ports, etc.

<pre>
function(head, req) {
  // helper function definitions would be here...
  var row, userConf, configHeader, configFoot;
  configHeader = renderTopOfApacheConf(head, req.query.hostname);
  send(configHeader);
</pre>

<p>In the first block of the function, we’re rendering the top of the config file using the function <code>renderTopOfApacheConf(head, req.query.hostname)</code>. This may include information that’s posted into the function, like the internal name of the server that is being configured or the root directory in which user HTML files are organized. We won’t show the function body, but you can imagine that it would return a long multi-line string that handles all the global configuration for your server and sets the stage for the per-user configuration that will be based on view data.

<p>The call to <code>send(configHeader)</code> is the heart of your ability to render text using list functions. Put simply, it just sends an HTTP chunk to the client, with the content of the strings pasted to it. There is some batching behind the scenes, as CouchDB speaks with the JavaScript runner with a synchronous protocol, but from the perspective of a programmer, <code>send()</code> is how HTTP chunks are born.

<p>Now that we’ve rendered and sent the file’s head, it’s time to start rendering the list itself. Each list item will be the result of converting a view row to a virtual host’s configuration element. The first thing we do is call <code>getRow()</code> to get a row of the view.

<pre>
  while (row = getRow()) {
    var userConf = renderUserConf(row);
    send(userConf)
  }
</pre>

<p>The <code>while</code> loop used here will continue to run until <code>getRow()</code> returns null, which is how CouchDB signals to the list function that all valid rows (based on the view query parameters) have been exhausted. Before we get ahead of ourselves, let’s check out what happens when we do get a row.

<p>In this case, we simply render a string based on the row and send it to the client. Once all rows have been rendered, the loop is complete. Now is a good time to note that the function has the option to return early. Perhaps it is programmed to stop iterating when it sees a particular user’s document or is based on a tally it’s been keeping of some resource allocated in the configuration. In those cases, the loop can end early with a <code>break</code> statement or other method. There’s no requirement for the list function to render every row that is sent to it.

<pre>
  configFoot = renderConfTail();
  return configFoot;
}
</pre>

<p>Finally, we close out the configuration file and return the final string value to be sent as the last HTTP chunk. The last action of a list function is always to return a string, which will be sent as the final HTTP chunk to the client.

<p>To use our config file generation function in practice, we might run a command-line script that looks like:

<pre>
curl http://localhost:5984/config_db/_design/files/_list/apache/users?hostname=foobar &gt; apache.conf
</pre>

<p>This will render our Apache config based on data in the user’s view and save it to a file. What a simple way to build a reliable configuration generator!

<h3 id="theory">List Theory</h3>

<p>Now that we’ve seen a complete list function, it’s worth mentioning some of the helpful properties they have.

<p>The most obvious thing is the iterator-style API. Because each row is loaded independently by calling <code>getRow()</code>, it’s easy not to leak memory. The list function API is capable of rendering lists of arbitrary length without error, when used correctly.

<p>On the other hand, this API gives you the flexibility to bundle a few rows in a single chunk of output, so if you had a view of, say, user accounts, followed by subdomains owned by that account, you could use a slightly more complex loop to build up some state in the list function for rendering more complex chunks. Let’s look at an alternate loop section:

<pre>
var subdomainOwnerRow, subdomainRows = [];
while (row = getRow()) {
</pre>

<p>We’ve entered a loop that will continue until we have reached the <code>endkey</code> of the view. The view is structured so that a user profile row is emitted, followed by all of that user’s subdomains. We’ll use the profile data and the subdomain information to template the configuration for each individual user. This means we can’t render any subdomain configuration until we know we’ve received all the rows for the current user.

<pre>
  if (!subdomainOwnerRow) {
    subdomainOwnerRow = row;
</pre>

<p>This case is true only for the first user. We’re merely setting up the initial conditions.

<pre>
  } else if (row.value.user != subdomainOwnerRow.value.user) {
</pre>

<p>This is the end case. It will be called only after all the subdomain rows for the current user have been exhausted. It is triggered by a row with a mismatched user, indicating that we have all the subdomain rows.

<pre>
    send(renderUserConf(subdomainOwnerRow, subdomainRows));
</pre>

<p>We know we are ready to render everything for the current user, so we pass the profile row and the subdomain rows to a render function (which nicely hides all the gnarly nginx config details from our fair reader). The result is sent to the HTTP client, which writes it to the config file.

<pre>
    subdomainRows = [];
    subdomainOwnerRow = row;
</pre>

<p>We’ve finished with that user, so let’s clear the rows and start working on the next user.

<pre>
  } else {
    subdomainRows.push(row);
</pre>

<p>Ahh, back to work, collecting rows.

<pre>
  }
}
send(renderUserConf(subdomainOwnerRow, subdomainRows));
</pre>

<p>This last bit is tricky—after the loop is finished (we’ve reached the end of the view query), we’ve still got to render the last user’s config. Wouldn’t want to forget that!

<p>The gist of this loop section is that we collect rows that belong to a particular user until we see a row that belongs to another user, at which point we render output for the first user, clear our state, and start working with the new user. Techniques like this show how much flexibility is allowed by the list iterator API.

<p>More uses along these lines include filtering rows that should be hidden from a particular result set, finding the top <em>N</em> grouped reduce values (e.g., to sort a tag cloud by popularity), and even writing custom reduce functions (as long as you don’t mind that reductions are not stored incrementally).

<h3 id="querying">Querying Lists</h3>

<p>We haven’t looked in detail at the ways list functions are queried. Just like show functions, they are resources available on the design document. The basic path to a list function is as follows:

<pre>
/db/_design/foo/_list/list-name/view-name
</pre>

<p>Because the list name and the view name are both specified, this means it is possible to render a list against more than one view. For instance, you could have a list function that renders blog comments in the Atom XML format, and then run it against both a global view of recent comments as well as a view of recent comments by blog post. This would allow you to use the same list function to provide an Atom feed for comments across an entire site, as well as individual comment feeds for each post.

<p>After the path to the list comes the view query parameter. Just like a regular view, calling a list function without any query parameters results in a list that reflects every row in the view. Most of the time you’ll want to call it with query parameters to limit the returned data.

<p>You’re already familiar with the view query options from <a href="views.html">Chapter 6, Finding Your Data with Views</a>. The same query options apply to the <code>_list</code> query. Let’s look at URLs side by side; see <a href="#example/1">Example 1, “A JSON view query”</a>.

<div class="figure" id="example/1">

<pre>
GET /db/_design/sofa/_view/recent-posts?descending=true&amp;limit=10
</pre>

<p class="caption">Example 1. A JSON view query

</div>

<p>This view query is just asking for the 10 most recent blog posts. Of course, this query could include parameters like <code>startkey</code> or <code>skip</code>—we’re leaving them out for simplicity. To run the same query through a list function, we access it via the list resource, as shown in <a href="#example/2">Example 2, “The HTML list query”</a>.

<div class="figure" id="example/2">

<pre>
GET /db/_design/sofa/_list/index/recent-posts?descending=true&amp;limit=10
</pre>

<p class="caption">Example 2. The HTML list query

</div>

<p>The <code>index</code> list here is a function from JSON to HTML. Just like the preceding view query, additional query parameters can be applied to paginate through the list. As we’ll see in <a href="index.html#part/iii">Part III, “Example Application”</a>, once you have a working list, adding pagination is trivial. See <a href="#example/3">Example 3, “The Atom list query”</a>.

<div class="figure" id="example/3">

<pre>
GET /db/_design/sofa/_list/index/recent-posts?descending=true&amp;limit=10&amp;format=atom
</pre>

<p class="caption">Example 3. The Atom list query

</div>

<p>The list function can also look at the query parameters and do things like switch that output to render based on parameters. You can even do things like pass the username into the list using a query parameter (but it’s not recommended, as you’ll ruin cache efficiency).

<h3 id="caching">Lists, Etags, and Caching</h3>

<p>Just like show functions and view queries, lists are sent with proper HTTP Etags, which makes them cacheable by intermediate proxies. This means that if your server is starting to bog down in list-rendering code, it should be possible to relieve load by using a caching reverse proxy like Squid. We won’t go into the details of Etags and caching here, as they were covered in <a href="show.html">Chapter 8, Show Functions</a>.