Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

import of work in progress website

  • Loading branch information...
commit e2c4f69d180adb8e122109ef33027ab02f680ba6 1 parent ad0b65e
Noah Slater authored
Showing with 8,231 additions and 1 deletion.
  1. +61 −0 README.md
  2. +5 −0 buy.html
  3. +488 −0 draft/api.html
  4. +41 −0 draft/balancing.html
  5. +61 −0 draft/btree.html
  6. BIN  draft/btree/01.png
  7. +113 −0 draft/clustering.html
  8. +19 −0 draft/colophon.html
  9. +266 −0 draft/conflicts.html
  10. BIN  draft/conflicts/01.png
  11. BIN  draft/conflicts/02.png
  12. BIN  draft/conflicts/03.png
  13. BIN  draft/conflicts/04.png
  14. BIN  draft/conflicts/05.png
  15. BIN  draft/conflicts/06.png
  16. BIN  draft/conflicts/07.png
  17. BIN  draft/conflicts/08.png
  18. +213 −0 draft/consistency.html
  19. BIN  draft/consistency/01.png
  20. BIN  draft/consistency/02.png
  21. BIN  draft/consistency/03.png
  22. BIN  draft/consistency/04.png
  23. BIN  draft/consistency/05.png
  24. BIN  draft/consistency/06.png
  25. BIN  draft/consistency/07.png
  26. +421 −0 draft/cookbook.html
  27. +143 −0 draft/design.html
  28. BIN  draft/design/01.png
  29. +368 −0 draft/documents.html
  30. BIN  draft/documents/01.png
  31. BIN  draft/documents/02.png
  32. BIN  draft/documents/03.png
  33. BIN  draft/documents/04.png
  34. +31 −0 draft/foreword.html
  35. +133 −0 draft/formats.html
  36. BIN  draft/formats/01.png
  37. +63 −0 draft/index.html
  38. +113 −0 draft/json.html
  39. +268 −0 draft/lists.html
  40. BIN  draft/lists/01.png
  41. +79 −0 draft/mac.html
  42. +338 −0 draft/managing.html
  43. BIN  draft/managing/01.png
  44. BIN  draft/managing/02.png
  45. +234 −0 draft/notifications.html
  46. +218 −0 draft/performance.html
  47. +53 −0 draft/preface.html
  48. +460 −0 draft/recipes.html
  49. +118 −0 draft/replication.html
  50. +69 −0 draft/scaling.html
  51. +243 −0 draft/security.html
  52. +298 −0 draft/show.html
  53. BIN  draft/show/01.png
  54. BIN  draft/show/02.png
  55. +263 −0 draft/source.html
  56. +219 −0 draft/standalone.html
  57. BIN  draft/standalone/01.png
  58. BIN  draft/standalone/02.png
  59. BIN  draft/standalone/03.png
  60. BIN  draft/standalone/04.png
  61. BIN  draft/standalone/05.png
  62. BIN  draft/standalone/06.png
  63. BIN  draft/standalone/07.png
  64. BIN  draft/standalone/08.png
  65. BIN  draft/standalone/09.png
  66. BIN  draft/standalone/10.png
  67. BIN  draft/standalone/11.png
  68. BIN  draft/standalone/12.png
  69. +410 −0 draft/tour.html
  70. BIN  draft/tour/01.png
  71. BIN  draft/tour/02.png
  72. BIN  draft/tour/03.png
  73. BIN  draft/tour/04.png
  74. BIN  draft/tour/05.png
  75. BIN  draft/tour/06.png
  76. BIN  draft/tour/07.png
  77. BIN  draft/tour/08.png
  78. BIN  draft/tour/09.png
  79. BIN  draft/tour/10.png
  80. +260 −0 draft/transforming.html
  81. +61 −0 draft/unix.html
  82. +210 −0 draft/validation.html
  83. BIN  draft/validation/01.png
  84. +575 −0 draft/views.html
  85. BIN  draft/views/01.png
  86. BIN  draft/views/02.png
  87. BIN  draft/views/03.png
  88. BIN  draft/views/04.png
  89. +159 −0 draft/why.html
  90. BIN  draft/why/01.png
  91. BIN  draft/why/02.png
  92. BIN  draft/why/03.png
  93. +21 −0 draft/windows.html
  94. BIN  image/couchdb_book.jpg
  95. BIN  image/couchg_bg.png
  96. BIN  image/couchg_bg_dark.png
  97. BIN  image/couchg_bg_footer.png
  98. +153 −1 index.html
  99. +263 −0 script.js
  100. +100 −0 style.css
  101. +562 −0 style/custom.css
  102. +21 −0 style/ie.css
  103. +21 −0 style/print.css
  104. +16 −0 todo.txt
61 README.md
View
@@ -0,0 +1,61 @@
+# CouchDB: The Definitive Guide
+
+This is the hope of the open source book “CouchDB: The Definitive Guide”
+
+
+## Organisation
+
+`draft/` is the always work in progress version of the next edition.
+
+`editions/` has a list of all editions of the book. `editions/1/` is the current edition of the book, when we do a second edition, it will be under `editions/2` and so on.
+
+`editions/1/en` is the main book content. Translations can be found under `editions/1/..`. Each language can be found under its respective language code (de, fr, jp…; see <http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes>)
+
+
+## Contributions
+
+This book is open source under the Creative Commons Attribution 3.0 Unported license (http://creativecommons.org/licenses/by/3.0/).
+
+The authors encourage you to fork, improve and publish our work under the terms of the license. Big thanks to O’Reilly for allowing this!
+
+If you feel like giving back, please use GitHub pull-requests [TODO: link] to notify us of new content. We’re equally happy about issues [TODO: link] you raise to get our attention.
+
+Chris, Jan and Noah retain editorial control over any changes anyone submits.
+
+
+## Publication
+
+Every once in a while, O’Reilly will take the current `en` version and turn it into a printed book. When that’s done, we’ll create a new directory under `editions/` to hold a stable snapshot of that release.
+
+
+## Translations
+
+We’d like to encourage you to start translating the book into your native language (or language of choice, really). We’re continuously publishing our work and all translations, so our readers always get the most up to date information.
+
+If you like to see a translation in a particular language, please first check if one exists already. If not, follow the instructions below.
+
+
+### Starting a Translation
+
+Only ever start a translation from the `en/` directory under `editions/<number>`. Do not try to translate `draft/` as it is constantly changing. Do not try to translate from any other language as `en/` is most likely the most complete source.
+
+Here’s how you would make a german (de) translation.
+
+ cd editions/1
+ cp -r en de
+ git add de
+ git commit -m 'Start German translation' de
+
+
+#### Styles
+
+If you need custom CSS rules for your tranlsations, please create a new file `style.css` in `editions/1/de/` and add another `<link rel="stylesheet" href="../style.css">` line to your HTML files.
+
+
+### Publishing a Translation
+
+You’re free to publish a translation under the aforementioned license. O’Reilly voiced interest in publishing translations as well, but no definite plans have been made. We’re happy to put you in touch with our editor to discuss printed editions of translations further.
+
+## Relax
+
+
5 buy.html
View
@@ -0,0 +1,5 @@
+<meta charset="utf-8">
+
+<link rel="stylesheet" href="style.css">
+
+<script src="http://catalog.oreilly.com/catalog/9780596158163/widgets/buy_buttons.js"></script>
488 draft/api.html
View
@@ -0,0 +1,488 @@
+<title>The Core API</title>
+
+<meta charset="utf-8">
+
+<link rel="stylesheet" href="../style.css">
+
+<link rel="prev" href="tour.html">
+
+<link rel="next" href="design.html">
+
+<h2 id="api">The Core API</h2>
+
+<p>This chapter explores the CouchDB in minute detail. It shows all the nitty-gritty and clever bits. We show you best practices and guide you around common pitfalls.
+
+<p>We start out by revisiting the basic operations we ran in the last chapter, looking behind the scenes. We also show what Futon needs to do behind its user interface to give us the nice features we saw earlier.
+
+<p>This chapter is both an introduction to the core CouchDB API as well as a reference. If you can’t remember how to run a particular request or why some parameters are needed, you can always come back here and look things up (we are probably the heaviest users of this chapter).
+
+<p>While explaining the API bits and pieces, we sometimes need to take a larger detour to explain the reasoning for a particular request. This is a good opportunity for us to tell you why CouchDB works the way it does.
+
+<p>The API can be subdivided into the following sections. We’ll explore them individually:
+
+<ul>
+
+<li>Server</li>
+<li>Databases</li>
+<li>Documents</li>
+<li>Replication</li>
+
+</ul>
+
+<h3 id="server">Server</h3>
+
+<p>This one is basic and simple. It can serve as a sanity check to see if CouchDB is running at all. It can also act as a safety guard for libraries that require a certain version of CouchDB. We’re using the <code>curl</code> utility again:
+
+<pre>
+curl http://127.0.0.1:5984/
+</pre>
+
+<p>CouchDB replies, all excited to get going:
+
+<pre>
+{"couchdb":"Welcome","version":"0.10.1"}
+</pre>
+
+<p>You get back a JSON string, that, if parsed into a native object or data structure of your programming language, gives you access to the welcome string and version information.
+
+<p>This is not terribly useful, but it illustrates nicely the way CouchDB behaves. You send an HTTP request and you receive a JSON string in the HTTP response as a result.
+
+<h3 id="databases">Databases</h3>
+
+<p>Now let’s do something a little more useful: create databases. For the strict, CouchDB is a <em>database management system</em> (DMS). That means it can hold multiple <em>databases</em>. A database is a bucket that holds “related data.” We’ll explore later what that means exactly. In practice, the terminology is overlapping—often people refer to a DMS as “a database” and also a database within the DMS as “a database.” We might follow that slight oddity, so don’t get confused by it. In general, it should be clear from the context if we are talking about the whole of CouchDB or a single database within CouchDB.
+
+<p>Now let’s make one! We want to store our favorite music albums, and we creatively give our database the name <code>albums</code>. Note that we’re now using the <code>-X</code> option again to tell <code>curl</code> to send a <code>PUT</code> request instead of the default <code>GET</code> request:
+
+<pre>
+curl -X PUT http://127.0.0.1:5984/albums
+</pre>
+
+<p>CouchDB replies:
+
+<pre>
+{"ok":true}
+</pre>
+
+<p>That’s it. You created a database and CouchDB told you that all went well. What happens if you try to create a database that already exists? Let’s try to create that database again:
+
+<pre>
+curl -X PUT http://127.0.0.1:5984/albums
+</pre>
+
+<p>CouchDB replies:
+
+<pre>
+{"error":"file_exists","reason":"The database could not be created, the file already exists."}
+</pre>
+
+<p>We get back an error. This is pretty convenient. We also learn a little bit about how CouchDB works. CouchDB stores each database in a single file. Very simple. This has some consequences down the road, but we’ll skip the details for now and explore the underlying storage system in <a href="btree.html">Appendix F, The Power of B-trees</a>.
+
+<p>Let’s create another database, this time with <code>curl</code>’s <code>-v</code> (for “verbose”) option. The verbose option tells <code>curl</code> to show us not only the essentials—the HTTP response body—but all the underlying request and response details:
+
+<pre>
+curl -vX PUT http://127.0.0.1:5984/albums-backup
+</pre>
+
+<p><code>curl</code> elaborates:
+
+<pre>
+* About to connect() to 127.0.0.1 port 5984 (#0)
+* Trying 127.0.0.1... connected
+* Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0)
+&gt; PUT /albums-backup HTTP/1.1
+&gt; User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
+&gt; Host: 127.0.0.1:5984
+&gt; Accept: */*
+&gt;
+&lt; HTTP/1.1 201 Created
+&lt; Server: CouchDB/0.9.0 (Erlang OTP/R12B)
+&lt; Date: Sun, 05 Jul 2009 22:48:28 GMT
+&lt; Content-Type: text/plain;charset=utf-8
+&lt; Content-Length: 12
+&lt; Cache-Control: must-revalidate
+&lt;
+{"ok":true}
+* Connection #0 to host 127.0.0.1 left intact
+* Closing connection #0
+</pre>
+
+<p>What a mouthful. Let’s step through this line by line to understand what’s going on and find out what’s important. Once you’ve seen this output a few times, you’ll be able to spot the important bits more easily.
+
+<pre>
+* About to connect() to 127.0.0.1 port 5984 (#0)
+</pre>
+
+<p>This is <code>curl</code> telling us that it is going to establish a <em>TCP</em> connection to the CouchDB server we specified in our request URI. Not at all important, except when debugging networking issues.
+
+<pre>
+* Trying 127.0.0.1... connected
+* Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0)
+</pre>
+
+<p><code>curl</code> tells us it successfully connected to CouchDB. Again, not important if you aren’t trying to find problems with your network.
+
+<p>The following lines are prefixed with <code>&gt;</code> and <code>&lt;</code> characters. <code>&gt;</code> means the line was sent to CouchDB verbatim (without the actual <code>&gt;</code>). <code>&lt;</code> means the line was sent back to <code>curl</code> by CouchDB.
+
+<pre>
+&gt; PUT /albums-backup HTTP/1.1
+</pre>
+
+<p>This initiates an HTTP request. Its <em>method</em> is <code>PUT</code>, the <em>URI</em> is <code>/albums-backup</code>, and the HTTP version is <code>HTTP/1.1</code>. There is also <code>HTTP/1.0</code>, which is simpler in some cases, but for all practical reasons you should be using <code>HTTP/1.1</code>.
+
+<p>Next, we see a number of <em>request headers</em>. These are used to provide additional details about the request to CouchDB.
+
+<pre>
+&gt; User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
+</pre>
+
+<p>The <code>User-Agent</code> header tell CouchDB which piece of client software is doing the HTTP request. We don’t learn anything new: it’s <code>curl</code>. This header is often useful in web development when there are known errors in client implementations that a server might want to prepare the response for. It also helps to determine which platform a user is on. This information can be used for technical and statistical reasons. For CouchDB, the <code>User-Agent</code> header is irrelevant.
+
+<pre>
+&gt; Host: 127.0.0.1:5984
+</pre>
+
+<p>The <code>Host</code> header is required by HTTP 1.1. It tells the server the hostname that came with the request.
+
+<pre>
+&gt; Accept: */*
+</pre>
+
+<p>The <code>Accept</code> header tells CouchDB that <code>curl</code> accepts any media type. We’ll look into why this is useful a little later.
+
+<pre>
+&gt;
+</pre>
+
+<p>An empty line denotes that the request headers are now finished and the rest of the request contains data we’re sending to the server. In this case, we’re not sending any data, so the rest of the curl output is dedicated to the HTTP response.
+
+<pre>
+&lt; HTTP/1.1 201 Created
+</pre>
+
+<p>The first line of CouchDB’s HTTP response includes the HTTP version information (again, to acknowledge that the requested version could be processed), an HTTP <em>status code</em>, and a <em>status code message</em>. Different requests trigger different response codes. There’s a whole range of them telling the client (<code>curl</code> in our case) what effect the request had on the server. Or, if an error occurred, what kind of error. RFC 2616 (the HTTP 1.1 specification) defines clear behavior for response codes. CouchDB fully follows the RFC.
+
+<p>The <em>201 Created</em> status code tells the client that the resource the request was made against was successfully created. No surprise here, but if you remember that we got an error message when we tried to create this database twice, you now know that this response could include a different response code. Acting upon responses based on response codes is a common practice. For example, all response codes of 400 or larger tell you that some error occurred. If you want to shortcut your logic and immediately deal with the error, you could just check a <code>&gt;= 400</code> response code.
+
+<pre>
+&lt; Server: CouchDB/0.10.1 (Erlang OTP/R13B)
+</pre>
+
+<p>The <code>Server</code> header is good for diagnostics. It tells us which CouchDB version and which underlying Erlang version we are talking to. In general, you can ignore this header, but it is good to know it’s there if you need it.
+
+<pre>
+&lt; Date: Sun, 05 Jul 2009 22:48:28 GMT
+</pre>
+
+<p>The <code>Date</code> header tells you the time of the server. Since client and server time are not necessary synchronized, this header is purely informational. You shouldn’t build any critical application logic on top of this!
+
+<pre>
+&lt; Content-Type: text/plain;charset=utf-8
+</pre>
+
+<p>The <code>Content-Type</code> header tells you which MIME type the HTTP response body is and its encoding. We already know CouchDB returns JSON strings. The appropriate <code>Content-Type</code> header is <code>application/json</code>. Why do we see <code>text/plain</code>? This is where pragmatism wins over purity. Sending an <code>application/json</code> <code>Content-Type</code> header will make a browser offer you the returned JSON for download instead of just displaying it. Since it is extremely useful to be able to test CouchDB from a browser, CouchDB sends a <code>text/plain</code> content type, so all browsers will display the JSON as text.
+
+<p>There are some browser extensions that make your browser JSON-aware, but they are not installed by default.
+
+<p>Do you remember the <code>Accept</code> request header and how it is set to <code>\*/\* -&gt; */*</code> to express interest in any MIME type? If you send <code>Accept: application/json</code> in your request, CouchDB knows that you can deal with a pure JSON response with the proper <code>Content-Type</code> header and will use it instead of <code>text/plain</code>.
+
+<pre>
+&lt; Content-Length: 12
+</pre>
+
+<p>The <code>Content-Length</code> header simply tells us how many bytes the response body has.
+
+<pre>
+&lt; Cache-Control: must-revalidate
+</pre>
+
+<p>This <code>Cache-Control</code> header tells you, or any proxy server between CouchDB and you, not to cache this response.
+
+<pre>
+&lt;
+</pre>
+
+<p>This empty line tells us we’re done with the response headers and what follows now is the response body.
+
+<pre>
+{"ok":true}
+</pre>
+
+<p>We’ve seen this before.
+
+<pre>
+* Connection #0 to host 127.0.0.1 left intact
+* Closing connection #0
+</pre>
+
+<p>The last two lines are <code>curl</code> telling us that it kept the TCP connection it opened in the beginning open for a moment, but then closed it after it received the entire response.
+
+<p>Throughout the book, we’ll show more requests with the <code>-v</code> option, but we’ll omit some of the headers we’ve seen here and include only those that are important for the particular request.
+
+<p>Creating databases is all fine, but how do we get rid of one? Easy—just change the HTTP method:
+
+<pre>
+&gt; curl -vX DELETE http://127.0.0.1:5984/albums-backup
+</pre>
+
+<p>This deletes a CouchDB database. The request will remove the file that the database contents are stored in. There is no “Are you sure?” safety net or any “Empty the trash” magic you’ve got to do to delete a database. Use this command with care. Your data will be deleted without a chance to bring it back easily if you don’t have a backup copy.
+
+<p>This section went knee-deep into HTTP and set the stage for discussing the rest of the core CouchDB API. Next stop: documents.
+
+<h3 id="documents">Documents</h3>
+
+<p>Documents are CouchDB’s central data structure. The idea behind a document is, unsurprisingly, that of a real-world document—a sheet of paper such as an invoice, a recipe, or a business card. We already learned that CouchDB uses the JSON format to store documents. Let’s see how this storing works at the lowest level.
+
+<p>Each document in CouchDB has an <em>ID</em>. This ID is unique per database. You are free to choose any string to be the ID, but for best results we recommend a UUID (or GUID), i.e., a Universally (or Globally) Unique IDentifier. UUIDs are random numbers that have such a low collision probability that everybody can make thousands of UUIDs a minute for millions of years without ever creating a duplicate. This is a great way to ensure two independent people cannot create two different documents with the same ID. Why should you care what somebody else is doing? For one, that somebody else could be you at a later time or on a different computer; secondly, CouchDB replication lets you share documents with others and using UUIDs ensures that it all works. But more on that later; let’s make some documents:
+
+<pre>
+curl -X PUT http://127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af -d '{"title":"There is Nothing Left to Lose","artist":"Foo Fighters"}'
+</pre>
+
+<p>CouchDB replies:
+
+<pre>
+{"ok":true,"id":"6e1295ed6c29495e54cc05947f18c8af","rev":"1-2902191555"}
+</pre>
+
+<p>The <code>curl</code> command appears complex, but let’s break it down. First, <code>-X PUT</code> tells <code>curl</code> to make a PUT request. It is followed by the URL that specifies your CouchDB IP address and port. The resource part of the URL <code>/albums/6e1295ed6c29495e54cc05947f18c8af</code> specifies the location of a document inside our <code>albums</code> database. The wild collection of numbers and characters is a UUID. This UUID is your document’s ID. Finally, the <code>-d</code> flag tells <code>curl</code> to use the following string as the body for the PUT request. The string is a simple JSON structure including <code>title</code> and <code>artist</code> attributes with their respective values.
+
+<div class="aside note">
+
+<p>If you don’t have a UUID handy, you can ask CouchDB to give you one (in fact, that is what we did just now without showing you). Simply send a GET request to <code>/_uuids</code>:
+
+<pre>
+curl -X GET http://127.0.0.1:5984/_uuids
+</pre>
+
+<p>CouchDB replies:
+
+<pre>
+{"uuids":["6e1295ed6c29495e54cc05947f18c8af"]}
+</pre>
+
+<p>Voilá, a UUID. If you need more than one, you can pass in the <code>?count=10</code> HTTP parameter to request 10 UUIDs, or really, any number you need.
+
+</div>
+
+<p>To double-check that CouchDB isn’t lying about having saved your document (it usually doesn’t), try to retrieve it by sending a GET request:
+
+<pre>
+curl -X GET http://127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af
+</pre>
+
+<p>We hope you see a pattern here. Everything in CouchDB has an address, a URI, and you use the different HTTP methods to operate on these URIs.
+
+<p>CouchDB replies:
+
+<pre>
+{"_id":"6e1295ed6c29495e54cc05947f18c8af","_rev":"1-2902191555","title":"There is Nothing Left to Lose","artist":"Foo Fighters"}
+</pre>
+
+<p>This looks a lot like the document you asked CouchDB to save, which is good. But you should notice that CouchDB added two fields to your JSON structure. The first is <code>_id</code>, which holds the UUID we asked CouchDB to save our document under. We always know the ID of a document if it is included, which is very convenient.
+
+<p>The second field is <code>_rev</code>. It stands for <em>revision</em>.
+
+<h4 id="revisions">Revisions</h4>
+
+<p>If you want to change a document in CouchDB, you don’t tell it to go and find a field in a specific document and insert a new value. Instead, you load the full document out of CouchDB, make your changes in the JSON structure (or object, when you are doing actual programming), and save the entire new revision (or version) of that document back into CouchDB. Each revision is identified by a new <code>_rev</code> value.
+
+<p>If you want to update or delete a document, CouchDB expects you to include the <code>_rev</code> field of the revision you wish to change. When CouchDB accepts the change, it will generate a new revision number. This mechanism ensures that, in case somebody else made a change unbeknownst to you before you got to request the document update, CouchDB will not accept your update because you are likely to overwrite data you didn’t know existed. Or simplified: whoever saves a change to a document first, wins. Let’s see what happens if we don’t provide a <code>_rev</code> field (which is equivalent to providing a outdated value):
+
+<pre>
+curl -X PUT http://127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af -d '{"title":"There is Nothing Left to Lose","artist":"Foo Fighters","year":"1997"}'
+</pre>
+
+<p>CouchDB replies:
+
+<pre>
+{"error":"conflict","reason":"Document update conflict."}
+</pre>
+
+<p>If you see this, add the latest revision number of your document to the JSON structure:
+
+<pre>
+curl -X PUT http://127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af -d '{"_rev":"1-2902191555","title":"There is Nothing Left to Lose", "artist":"Foo Fighters","year":"1997"}'
+</pre>
+
+<p>Now you see why it was handy that CouchDB returned that <code>_rev</code> when we made the initial request. CouchDB replies:
+
+<pre>
+{"ok":true,"id":"6e1295ed6c29495e54cc05947f18c8af","rev":"2-2739352689"}
+</pre>
+
+<p>CouchDB accepted your write and also generated a new revision number. The revision number is the md5 hash of the transport representation of a document with an <code>N-</code> prefix denoting the number of times a document got updated. This is useful for replication. See <a href="conflicts.html">Chapter 17, Conflict Management</a> for more information.
+
+<p>There are multiple reasons why CouchDB uses this revision system, which is also called Multi-Version Concurrency Control (MVCC). They all work hand-in-hand, and this is a good opportunity to explain some of them.
+
+<p>One of the aspects of the HTTP protocol that CouchDB uses is that it is <em>stateless</em>. What does that mean? When talking to CouchDB you need to <em>make requests</em>. Making a request includes opening a network connection to CouchDB, exchanging bytes, and closing the connection. This is done every time you make a request. Other protocols allow you to open a connection, exchange bytes, keep the connection open, exchange more bytes later—maybe depending on the bytes you exchanged at the beginning—and eventually close the connection. Holding a connection open for later use requires the server to do extra work. One common pattern is that for the lifetime of a connection, the client has a consistent and static view of the data on the server. Managing huge amounts of parallel connections is a significant amount of work. HTTP connections are usually short-lived, and making the same guarantees is a lot easier. As a result, CouchDB can handle many more concurrent connections.
+
+<p>Another reason CouchDB uses MVCC is that this model is simpler conceptually and, as a consequence, easier to program. CouchDB uses less code to make this work, and less code is always good because the ratio of defects per lines of code is static.
+
+<p>The revision system also has positive effects on replication and storage mechanisms, but we’ll explore these later in the book.
+
+<div class="aside warning">
+
+<p>The terms <em>version</em> and <em>revision</em> might sound familiar (if you are programming without version control, drop this book right now and start learning one of the popular systems). Using new versions for document changes works a lot like version control, but there’s an important difference: CouchDB does <em>not</em> guarantee that older versions are kept around.
+
+</div>
+
+<h4 id="detail">Documents in Detail</h4>
+
+<p>Now let’s have a closer look at our document creation requests with the <code>curl</code> <code>-v</code> flag that was helpful when we explored the database API earlier. This is also a good opportunity to create more documents that we can use in later examples.
+
+<p>We’ll add some more of our favorite music albums. Get a fresh UUID from the <code>/_uuids</code> resource. If you don’t remember how that works, you can look it up a few pages back.
+
+<pre>
+curl -vX PUT http://127.0.0.1:5984/albums/70b50bfa0a4b3aed1f8aff9e92dc16a0 -d '{"title":"Blackened Sky","artist":"Biffy Clyro","year":2002}'
+</pre>
+
+<div class="aside note">
+
+<p>By the way, if you happen to know more information about your favorite albums, don’t hesitate to add more properties. And don’t worry about not knowing all the information for all the albums. CouchDB’s schema-less documents can contain whatever you know. After all, you should relax and not worry about data.
+
+</aside>
+
+<p>Now with the <code>-v</code> option, CouchDB’s reply (with only the important bits shown) looks like this:
+
+<pre>
+&gt; PUT /albums/70b50bfa0a4b3aed1f8aff9e92dc16a0 HTTP/1.1
+&gt;
+&lt; HTTP/1.1 201 Created
+&lt; Location: http://127.0.0.1:5984/albums/70b50bfa0a4b3aed1f8aff9e92dc16a0
+&lt; Etag: "1-2248288203"
+&lt;
+{"ok":true,"id":"70b50bfa0a4b3aed1f8aff9e92dc16a0","rev":"1-2248288203"}
+</pre>
+
+<p>We’re getting back the <code>201 Created</code> HTTP status code in the response headers, as we saw earlier when we created a database. The <code>Location</code> header gives us a full URL to our newly created document. And there’s a new header. An Etag in HTTP-speak identifies a specific version of a resource. In this case, it identifies a specific version (the first one) of our new document. Sound familiar? Yes, conceptually, an Etag is the same as a CouchDB document revision number, and it shouldn’t come as a surprise that CouchDB uses revision numbers for Etags. Etags are useful for caching infrastructures. We’ll learn how to use them in <a href="show.html">Chapter 8, Show Functions</a>.
+
+<h5 id="attachments">Attachments</h5>
+
+<p>CouchDB documents can have attachments just like an email message can have attachments. An attachment is identified by a name and includes its MIME type (or Content-Type) and the number of bytes the attachment contains. Attachments can be any data. It is easiest to think about attachments as files attached to a document. These files can be text, images, Word documents, music, or movie files. Let’s make one.
+
+<p>Attachments get their own URL where you can upload data. Say we want to add the album artwork to the <code>6e1295ed6c29495e54cc05947f18c8af</code> document (“There is Nothing Left to Lose”), and let’s also say the artwork is in a file <code>artwork.jpg</code> in the current directory:
+
+<pre>
+&gt; curl -vX PUT http://127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af/ artwork.jpg?rev=2-2739352689 --data-binary @artwork.jpg -H "Content-Type: image/jpg"
+</pre>
+
+<p>The <code>-d@</code> option tells <code>curl</code> to read a file’s contents into the HTTP request body. We’re using the <code>-H</code> option to tell CouchDB that we’re uploading a JPEG file. CouchDB will keep this information around and will send the appropriate header when requesting this attachment; in case of an image like this, a browser will render the image instead of offering you the data for download. This will come in handy later. Note that you need to provide the current revision number of the document you’re attaching the artwork to, just as if you would update the document. Because, after all, attaching some data is changing the document.
+
+<p>You should now see your artwork image if you point your browser to <code>http://127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af/artwork.jpg</code>.
+
+<p>If you request the document again, you’ll see a new member:
+
+<pre>
+curl http://127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af
+</pre>
+
+<p>CouchDB replies:
+
+<pre>
+{"_id":"6e1295ed6c29495e54cc05947f18c8af","_rev":"3-131533518","title": "There is Nothing Left to Lose","artist":"Foo Fighters","year":"1997","_attachments":{"artwork.jpg":{"stub":true,"content_type":"image/jpg","length":52450}}}
+</pre>
+
+<p><code>_attachments</code> is a list of keys and values where the values are JSON objects containing the attachment metadata. <code>stub=true</code> tells us that this entry is just the metadata. If we use the <code>?attachments=true</code> HTTP option when requesting this document, we’d get a Base64-encoded string containing the attachment data.
+
+<p>We’ll have a look at more document request options later as we explore more features of CouchDB, such as replication, which is the next topic.
+
+<h3 id="replication">Replication</h3>
+
+<p>CouchDB replication is a mechanism to synchronize databases. Much like <code>rsync</code> synchronizes two directories locally or over a network, replication synchronizes two databases locally or remotely.
+
+<p>In a simple POST request, you tell CouchDB the <em>source</em> and the <em>target</em> of a replication and CouchDB will figure out which documents and new document revisions are on <em>source</em> that are not yet on <em>target</em>, and will proceed to move the missing documents and revisions over.
+
+<p>We’ll take an in-depth look at replication later in the book; in this chapter, we’ll just show you how to use it.
+
+<p>First, we’ll create a target database. Note that CouchDB won’t automatically create a target database for you, and will return a replication failure if the target doesn’t exist (likewise for the source, but that mistake isn’t as easy to make):
+
+<pre>
+curl -X PUT http://127.0.0.1:5984/albums-replica
+</pre>
+
+<p>Now we can use the database <code>albums-replica</code> as a replication target:
+
+<pre>
+curl -vX POST http://127.0.0.1:5984/_replicate -d '{"source":"albums","target":"albums-replica"}'
+</pre>
+
+<div class="aside note">
+
+<p>As of version 0.11, CouchDB supports the option <code>"create_target":true</code> placed in the JSON POSTed to the <code>_replicate</code> URL. It implicitly creates the target database if it doesn’t exist.
+
+</div>
+
+<p>CouchDB replies (this time we formatted the output so you can read it more easily):
+
+<pre>
+{
+ "history": [
+ {
+ "start_last_seq": 0,
+ "missing_found": 2,
+ "docs_read": 2,
+ "end_last_seq": 5,
+ "missing_checked": 2,
+ "docs_written": 2,
+ "doc_write_failures": 0,
+ "end_time": "Sat, 11 Jul 2009 17:36:21 GMT",
+ "start_time": "Sat, 11 Jul 2009 17:36:20 GMT"
+ }
+ ],
+ "source_last_seq": 5,
+ "session_id": "924e75e914392343de89c99d29d06671",
+ "ok": true
+}
+</pre>
+
+<p>CouchDB maintains a <em>session history</em> of replications. The response for a replication request contains the history entry for this <em>replication session</em>. It is also worth noting that the request for replication will stay <em>open</em> until replication closes. If you have a lot of documents, it’ll take a while until they are all replicated and you won’t get back the replication response until all documents are replicated. It is important to note that replication replicates the database only as it was at the point in time when replication was started. So, any additions, modifications, or deletions subsequent to the start of replication will not be replicated.
+
+<p>We’ll punt on the details again—the <code>"ok": true</code> at the end tells us all went well. If you now have a look at the <code>albums-replica</code> database, you should see all the documents that you created in the <code>albums</code> database. Neat, eh?
+
+<p>What you just did is called <em>local replication</em> in CouchDB terms. You created a local copy of a database. This is useful for backups or to keep snapshots of a specific state of your data around for later. You might want to do this if you are developing your applications but want to be able to roll back to a stable version of your code and data.
+
+<p>There are more types of replication useful in other situations. The <code>source</code> and <code>target</code> members of our replication request are actually links (like in HTML) and so far we’ve seen links relative to the server we’re working on (hence <em>local</em>). You can also specify a remote database as the target:
+
+<pre>
+curl -vX POST http://127.0.0.1:5984/_replicate -d '{"source":"albums","target":"http://127.0.0.1:5984/albums-replica"}'
+</pre>
+
+<p>Using a local <code>source</code> and a remote <code>target</code> database is called <em>push replication</em>. We’re pushing changes to a remote server.
+
+<div class="aside note">
+
+<p>Since we don’t have a second CouchDB server around just yet, we’ll just use the absolute address of our single server, but you should be able to infer from this that you can put any remote server in there.
+
+</div>
+
+<p>This is great for sharing local changes with remote servers or buddies next door.
+
+<p>You can also use a remote <code>source</code> and a local <code>target</code> to do a <em>pull replication</em>. This is great for getting the latest changes from a server that is used by others:
+
+<pre>
+curl -vX POST http://127.0.0.1:5984/_replicate -d '{"source":"http://127.0.0.1:5984/albums-replica","target":"albums"}'
+</pre>
+
+<p>Finally, you can run <em>remote replication</em>, which is mostly useful for management operations:
+
+<pre>
+curl -vX POST http://127.0.0.1:5984/_replicate -d '{"source":"http://127.0.0.1:5984/albums","target":"http://127.0.0.1:5984/albums-replica"}'
+</pre>
+
+<div class="aside note">
+
+<p><strong>CouchDB and REST</strong>
+
+<p>CouchDB prides itself on having a <em>RESTful API</em>, but these replication requests don’t look very RESTy to the trained eye. What’s up with that? While CouchDB’s core database, document, and attachment API are RESTful, not all of CouchDB’s API is. The replication API is one example. There are more, as we’ll see later in the book.
+
+<p>Why are there RESTful and non-RESTful APIs mixed up here? Have the developers been too lazy to go REST all the way? Remember, REST is an architectural style that lends itself to certain architectures (such as the CouchDB document API). But it is not a one-size-fits-all. Triggering an event like replication does not make a whole lot of sense in the REST world. It is more like a traditional remote procedure call. And there is nothing wrong with this.
+
+<p>We very much believe in the “use the right tool for the job” philosophy, and REST does not fit every job. For support, we refer to Leonard Richardson and Sam Ruby who wrote <a href="http://oreilly.com/catalog/9780596529260">RESTful Web Services</a> (O’Reilly), as they share our view.
+
+</div>
+
+<h3 id="wrap">Wrapping Up</h3>
+
+<p>This is still not the full CouchDB API, but we discussed the essentials in great detail. We’re going to fill in the blanks as we go. For now, we believe you’re ready to start building CouchDB applications.
+
+<script src="../script.js"></script>
41 draft/balancing.html
View
@@ -0,0 +1,41 @@
+<title>Load Balancing</title>
+
+<meta charset="utf-8">
+
+<link rel="stylesheet" href="../style.css">
+
+<link rel="prev" href="conflicts.html">
+
+<link rel="next" href="clustering.html">
+
+<h2 id="balancing">Load Balancing</h2>
+
+<p>Jill is woken up at 4:30 a.m. by her mobile phone. She receives text message after text message, one every minute. Finally, Joe calls. Joe is furious, and Jill has trouble understanding what Joe is saying. In fact, Jill has a hard time figuring out why Joe would call her in the middle of the night. Then she remembers: Joe is running an online shop selling sports gear on one of her servers, and he is furious because the server went down and now his customers in New Zealand are angry because they can’t get to the online shop.
+
+<p>This is a typical scenario, and you have probably seen many variations of it, being in the role of Jill, Joe, or both. If you are Jill, you want to sleep at night, and if you are Joe, you want your customers to buy from you whenever it pleases them.
+
+<h3 id="backup">Having a Backup</h3>
+
+<p>The problems persist: computers fail, and in many ways. There are hardware problems, power outages, bugs in the operating system or application software, etc. Only CouchDB doesn’t have any bugs. (Well, of course, that’s not true. All software has bugs, with the possible exception of things written by Daniel J. Bernstein and Donald Knuth.)
+
+<p>Whatever the cause is, you want to make sure that the service you are providing (in Jill and Joe’s case, the database for an online store) is resilient against failure. The road to resilience is a road of finding and removing single points of failure. A server’s power supply can fail. To keep the server from turning off during such an event, most come with at least two power supplies. To take this further, you could get a server where everything is duplicated (or more), but that would be a highly specialized (and expensive) piece of hardware. It is much cheaper to get two similar servers where the one can take over if the other has a problem. However, you need to make sure both servers have the same set of data in order to switch them without a user noticing.
+
+<p>Removing all single points of failure will give you a highly available or a fault-tolerant system. The order of tolerance is restrained only by your budget. If you can’t afford to lose a customer’s shopping cart in any event, you need to store it on at least two servers in at least two far apart geographical locations.
+
+<div class="aside note">
+
+<p>Amazon does this for the <a href="http://www.amazon.com">Amazon.com</a> website. If one data center is the victim of an earthquake, a user will still be able to shop.
+
+<p>It is likely, though, that Amazon’s problems are not your problems and that you will have a whole set of new problems when your data center goes away. But you still want to be able to live through a server failure.
+
+</div>
+
+<p>Before we dive into setting up a highly available CouchDB system, let’s look at another situation. Joe calls Jill during regular business hours and relays his customers’ complaints that loading the online shop takes “forever.” Jill takes a quick look at the server and concludes that this is a lucky problem to have, leaving Joe puzzled. Jill explains that Joe’s shop is suddenly attracting many more users who are buying things. Joe chimes in, “I got a great review on that blog. That’s where they must be coming from.” A quick referrer check reveals that indeed many of the new customers are coming from a single site. The blog post already includes comments from unhappy customers voicing their frustration with the slow site. Joe wants to make his customers happy and asks Jill what to do. Jill advises that they set up a second server that can take half of the load of the current server, making sure all requests get answered in a reasonable amount of time. Joe agrees, and Jill begins to set things up.
+
+<p>The solution to the outlined problem looks a lot like the earlier one for providing a fault-tolerant setup: install a second server and synchronize all data. The difference is that with fault tolerance, the second server just sits there and waits for the first one to fail. In the server-overload case, a second server helps answer all incoming requests. This case is not fault-tolerant: if one server crashes, the other will get all the requests and will likely break down, or at least provide very slow service, either of which is not acceptable.
+
+<p>Keep in mind that although the solutions look similar, high availability and fault tolerance are not the same. We’ll get back to the second scenario later on, but first we will take a look at how to set up a fault-tolerant CouchDB system.
+
+<p>We already gave it away in the previous chapters: the solution to synchronizing servers is replication.
+
+<script src="../script.js"></script>
61 draft/btree.html
View
@@ -0,0 +1,61 @@
+<title>The Power of B-trees</title>
+
+<meta charset="utf-8">
+
+<link rel="stylesheet" href="../style.css">
+
+<link rel="prev" href="json.html">
+
+<link rel="next" href="colophon.html">
+
+<h2 id="btree">The Power of B-trees</h2>
+
+<p>CouchDB uses a data structure called a B-tree to index its documents and views. We’ll look at B-trees enough to understand the types of queries they support and how they are a good fit for CouchDB.
+
+<p>This is our first foray into CouchDB internals. To use CouchDB, you don’t need to know what’s going on under the hood, but if you understand how CouchDB performs its magic, you’ll be able to pull tricks of your own. Additionally, if you understand the consequences of the ways you are using CouchDB, you will end up with smarter systems.
+
+<p>If you weren’t looking closely, CouchDB would appear to be a B-tree manager with an HTTP interface.
+
+<div class="aside note">
+
+<p>CouchDB is actually using a B+ tree, which is a slight variation of the B-tree that trades a bit of (disk) space for speed. When we say <em>B-tree</em>, we mean CouchDB’s <em>B+ tree</em>.
+
+</div>
+
+<p>A B-tree is an excellent data structure for storing huge amounts of data for fast retrieval. When there are millions and billions of items in a B-tree, that’s when they get fun. B-trees are usually a shallow but wide data structure. While other trees can grow very high, a typical B-tree has a single-digit height, even with millions of entries. This is particularly interesting for CouchDB, where the leaves of the tree are stored on a slow medium such as a hard drive. Accessing any part of the tree for reading or writing requires visiting only a few nodes, which translates to a few head seeks (which are what make a hard drive slow), and because the operating system is likely to cache the upper tree nodes anyway, only the seek to the final leaf node is needed.
+
+<blockquote>
+
+<p>From a practical point of view, B-trees, therefore, guarantee an access time of less than 10 ms even for extremely large datasets.
+
+<p class="attribution">&mdash;Dr. Rudolf Bayer, inventor of the B-tree
+
+</blockquote>
+
+<p>CouchDB’s B-tree implementation is a bit different from the original. While it maintains all of the important properties, it adds Multi-Version Concurrency Control (MVCC) and an append-only design. B-trees are used to store the main database file as well as view indexes. One database is one B-tree, and one view index is one B-tree.
+
+<p>MVCC allows concurrent reads and writes without using a locking system. Writes are serialized, allowing only one write operation at any point in time for any single database. Write operations do not block reads, and there can be any number of read operations at any time. Each read operation is guaranteed a consistent view of the database. How this is accomplished is at the core of CouchDB’s storage model.
+
+<p>The short answer is that because CouchDB uses append-only files, the B-tree root node must be rewritten every time the file is updated. However, old portions of the file will never change, so every old B-tree root, should you happen to have a pointer to it, will also point to a consistent snapshot of the database.
+
+<p>Early in the book we explained how the MVCC system uses the document’s <code>_rev</code> value to ensure that only one person can change a document version. The B-tree is used to look up the existing <code>_rev</code> value for comparison. By the time a write is accepted, the B-tree can expect it to be an authoritative version.
+
+<p>Since old versions of documents are not overwritten or deleted when new versions come in, requests that are reading a particular version do not care if new ones are written at the same time. With an often changing document, there could be readers reading three different versions at the same time. Each version was the latest one when a particular client started reading it, but new versions were being written. From the point when a new version is <em>committed</em>, new readers will read the new version while old readers keep reading the old version.
+
+<p>In a B-tree, data is kept only in leaf nodes. CouchDB B-trees append data only to the database file that keeps the B-tree on disk and grows only at the end. Add a new document? The file grows at the end. Delete a document? That gets recorded at the end of the file. The consequence is a robust database file. Computers fail for plenty of reasons, such as power loss or failing hardware. Since CouchDB does not overwrite any existing data, it cannot corrupt anything that has been written and <em>committed</em> to disk already. See <a href="#figure/1">Figure 1, “Flat B-tree and append-only”</a>.
+
+<p>Committing is the process of updating the database file to reflect changes. This is done in the file footer, which is the last 4k of the database file. The footer is 2k in size and written twice in succession. First, CouchDB appends any changes to the file and then records the file’s new length in the first database footer. It then force-flushes all changes to disk. It then copies the first footer over to the second 2k of the file and force-flushes again.
+
+<div class="figure" id="figure/1">
+
+<img src="btree/01.png">
+
+<p class="caption">Figure 1. Flat B-tree and append-only
+
+</div>
+
+<p>If anywhere in this process a problem occurs—say, power is cut off and CouchDB is restarted later—the database file is in a consistent state and doesn’t need a checkup. CouchDB starts reading the database file backward. When it finds a footer pair, it makes some checks: if the first 2k are corrupt (a footer includes a checksum), CouchDB replaces it with the second footer and all is well. If the second footer is corrupt, CouchDB copies the first 2k over and all is well again. Only once both footers are flushed to disk successfully will CouchDB acknowledge that a write operation was successful. Data is never lost, and data on disk is never corrupted. This design is the reason for CouchDB having no <em>off</em> switch. You just terminate it when you are done.
+
+<p>There’s a lot more to say about B-trees in general, and if and how SSDs change the runtime behavior. The Wikipedia article on <a href="http://en.wikipedia.org/wiki/B-tree">B-trees</a> is a good starting point for further investigations. Scholarpedia includes <a href="http://www.scholarpedia.org/article/B-tree_and_UB-tree">notes</a> by Dr. Rudolf Bayer, inventor of the B-tree.
+
+<script src="../script.js"></script>
BIN  draft/btree/01.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
113 draft/clustering.html
View
@@ -0,0 +1,113 @@
+<title>Clustering</title>
+
+<meta charset="utf-8">
+
+<link rel="stylesheet" href="../style.css">
+
+<link rel="prev" href="balancing.html">
+
+<link rel="next" href="notifications.html">
+
+<h2 id="clustering">Clustering</h2>
+
+<p>OK, you’ve made it this far. I’m assuming you more or less understand what CouchDB is and how the application API works. Maybe you’ve deployed an application or two, and now you’re dealing with enough traffic that you need to think about scaling. “Scaling” is an imprecise word, but in this chapter we’ll be dealing with the aspect of putting together a partitioned or sharded cluster that will have to grow at an increasing rate over time from day one.
+
+<p>We’ll look at request and response dispatch in a CouchDB cluster with stable nodes. Then we’ll cover how to add redundant hot-failover twin nodes, so you don’t have to worry about losing machines. In a large cluster, you should plan for 5–10% of your machines to experience some sort of failure or reduced performance, so cluster design must prevent node failures from affecting reliability. Finally, we’ll look at adjusting cluster layout dynamically by splitting or merging nodes using replication.
+
+<h3 id="lounge">Introducing CouchDB Lounge</h3>
+
+<p><a href="http://tilgovi.github.com/couchdb-lounge/">CouchDB Lounge</a> is a proxy-based partitioning and clustering application, originally developed for <a href="http://www.meebo.com">Meebo</a>, a web-based instant messaging service. Lounge comes with two major components: one that handles simple GET and PUT requests for documents, and another that distributes view requests.
+
+<p>The <em>dumbproxy</em> handles simple requests for anything that isn’t a CouchDB view. This comes as a module for <a href="http://nginx.net/">nginx</a>, a high-performance reverse HTTP proxy. Because of the way reverse HTTP proxies work, this automatically allows configurable security, encryption, load distribution, compression, and, of course, aggressive caching of your database resources.
+
+<p>The <em>smartproxy</em> handles only CouchDB view requests, and dispatches them to all the other nodes in the cluster so as to distribute the work, making view performance a function of the cluster’s cumulative processing power. This comes as a daemon for Twisted, a popular and high-performance event-driven network programming framework for Python.
+
+<h3 id="hashing">Consistent Hashing</h3>
+
+<p>CouchDB’s storage model uses unique IDs to save and retrieve documents. Sitting at the core of Lounge is a simple method of hashing your document IDs. Lounge then uses the first few characters of this hash to determine which shard to dispatch the request to. You can configure this behavior by writing a <em>shard map</em> for Lounge, which is just a simple text configuration file.
+
+<p>Because Lounge allocates a portion of the hash (known as a keyspace) to each node, you can add as many nodes as you like. Because the hash function produces hexidecimal strings that bare no apparent relation to your DocIDs, and because we dispatch requests based on the first few characters, we ensure that all nodes see roughly equal load. And because the hash function is consistent, Lounge will take any arbitrary DocID from an HTTP request URI and point it to the same node each time.
+
+<p>This idea of splitting a collection of shards based on a keyspace is commonly illustrated as a ring, with the hash wrapped around the outside. Each tic mark designates the boundaries in the keyspace between two partitions. The hash function maps from document IDs to positions on the ring. The ring is continuous so that you can always add more nodes by splitting a single partition into pieces. With four physical servers, you allocate the keyspace into 16 independent partitions by distributing them across the servers like so:
+
+<table>
+
+<tbody>
+
+<tr><td>A</td><td>0,1,2,3</td></tr>
+
+<tr><td>B</td><td>4,5,6,7</td></tr>
+
+<tr><td>C</td><td>8,9,a,b</td></tr>
+
+<tr><td>D</td><td>c,d,e,f</td></tr>
+
+</tbody>
+
+</table>
+
+<p>If the hash of your DocID starts with <code>0</code>, it would be dispatched to shard <code>A</code>. Similarly for <code>1</code>, <code>2</code>, or <code>3</code>. Whereas, if the hash started with <code>c</code>, <code>d</code>, <code>e</code>, or <code>f</code>, it would be dispatched to shard <code>D</code>. As a full example, the hash <code>71db329b58378c8fa8876f0ec04c72e5</code> is mapped to the node <code>B</code>, database <code>7</code> in the table just shown. This could map to <code>http://B.couches.local/db-7/</code> on your backend cluster. In this way, the hash table is just a mapping from hashes to backend database URIs. Don’t worry if this all sounds very complex; all you have to do is provide a mapping of shards to nodes and Lounge will build the hash ring appropriately—so no need to get your hands dirty if you don’t want to.
+
+<p>To frame the same concept with web architecture, because CouchDB uses HTTP, the proxy can partition documents according to the request URL, without inspecting the body. This is a core principle behind REST and is one of the many benefits using HTTP affords us. In practice, this is accomplished by running the hash function against the request URI and comparing the result to find the portion of the keyspace allocated. Lounge then looks up the associated shard for the hash in a configuration table, forwarding the HTTP request to the backend CouchDB server.
+
+<p>Consistent hashing is a simple way to ensure that you can always find the documents you saved, while balancing storage load evenly across partitions. Because the hash function is simple (it is based on CRC32), you are free to implement your own HTTP intermediaries or clients that can similarly resolve requests to the correct physical location of your data.
+
+<h4 id="storage">Redundant Storage</h4>
+
+<p>Consistent hashing solves the problem of how to break up a single logical database evenly across a set of partitions, which can then be distributed across multiple servers. It does not address the problem of how to ensure that data you’ve stored is safe from loss due to hardware or software failure. If you are serious about your data, you can’t consider it saved until you have at least two copies of it, preferably in different geographical locations.
+
+<p>CouchDB replication makes maintaining hot-failover redundant slaves or load-balanced multi-master databases relatively painless. The specifics of how to manage replication are covered in <a href="replication.html">Chapter 16, Replication</a>. What is important in this context is to understand that maintaining redundant copies is orthogonal to the harder task of ensuring that the cluster consistently chooses the same partition for a particular document ID.
+
+<p>For data safety, you’ll want to have at least two or three copies of everything. However, if you encapsulate redundancy, the higher layers of the cluster can treat each partition as a single unit and let the logical partitions themselves manage redundancy and failover.
+
+<h4 id="proxies">Redundant Proxies</h4>
+
+<p>Just as we can’t accept the possibility of hardware failure leading to data loss, we’ll need to run multiple instances of the proxy nodes to avoid the chance that a proxy node crash could leave portions of the cluster unavailable. By running redundant proxy instances, and load balancing across them, we can increase cluster throughput as well as reliability.
+
+<h4 id="merging">View Merging</h4>
+
+<p>Consistent hashing leaves documents on the proper node, but documents can still <code>emit()</code> any key. The point of incremental MapReduce is to bring the function to the data, so we shoudn’t redistribute the emitted keys; instead, we send the queries to the CouchDB nodes via HTTP proxy, and merge the results using the Twisted Python Smartproxy.
+
+<p>Smartproxy sends each view request to every node, so it needs to merge the responses before returning them to the client. Thankfully, this operation is not resource-intensive, as merging can be done in constant memory space no matter how many rows are returned. The Smartproxy receives the first row from each cluster node and compares them. We sort the nodes according to their row key using CouchDB’s collation rules. Smartproxy pops the top row from the first sorted node and returns it to the client.
+
+<p>This process can be repeated as long as the clients continue to send rows, but if a limit is imposed by the client, Smartproxy must end the response early, discarding any extra rows sent by the nodes.
+
+<p>This layout is simple and loosely coupled. It has the advantage that it’s simple, which helps in understanding topology and diagnosing failures. There is work underway to move the behavior to Erlang, which ought to make managing dynamic clusters possible as well as let us integrate cluster control into the CouchDB runtime.
+
+<h3 id="growing">Growing the Cluster</h3>
+
+<p>Using CouchDB at web scale likely requires CouchDB clusters that can be scaled dynamically. Growing sites must continuously add more storage capacity, so we need a strategy to increase the size of our cluster without taking it down. Some workloads can result in temporary growth in data size, in which case we’ll also need a process for shrinking the cluster without an interruption in service.
+
+<p>In this section, we’ll see how we can use CouchDB’s replication filters to split one database into several partitions, and how to use that technique to grow the cluster without downtime. There are simple steps you can take to avoid partitioning databases while growing the cluster.
+
+<p><em>Oversharding</em> is a technique where you partition the cluster so that there are multiple shards on each physical machine. Moving a partition from one machine to another is simpler than splitting it into smaller partitions, as the configuration map of the cluster used by the proxy only needs to change to point to shards at their new homes, rather than adding new logical shards. It’s also less resource-intensive to move a partition than to split it into many.
+
+<p>One question we need to answer is, “How much should we overshard?” The answer depends on your application and deployment, but there are some forces that push us in one direction over another. If we get the number of shards right, we’ll end up with a cluster that can grow optimally.
+
+<p>In <a href="#merging">the section called “View Merging”</a>, we discussed how merges can be accomplished in constant space, no matter the number of rows returned. The memory space and network resources required to merge views, as well as to map from document IDs to partitions, does, however, grow linearly with the number of partitions under a given proxy. For this reason, we’ll want to limit the number of partitions for each proxy. However, we can’t accept an upper limit on cluster size. The solution is to use a tree of proxies, where the root proxy partitions to some number of intermediate proxies, which then proxy to database nodes.
+
+<p>The factors that come into play when deciding how many partitions each proxy should manage are: the storage available to each individual server node, the projected growth rate of the data, the network and memory resources available to proxies, and the acceptable latency for requests against the cluster.
+
+<p>Assuming a conservative 64 shards per proxy, and 1 TB of data storage per node (including room for compaction, these nodes will need roughly 2 TB of drive space), we can see that with a single proxy in front of CouchDB data nodes, we’ll be able to store at maximum 64 TB of data (on 128 or perhaps 192 server nodes, depending on the level of redundancy required by the system) before we have to increase the number of partitions.
+
+<p>By replacing database nodes with another proxy, and repartitioning each of the 64 partitions into another 64 partitions, we end up with 4,096 partitions and a tree depth of 2. Just as the initial system can hold 64 partitions on just a few nodes, we can transition to the 2-layer tree without needing thousands of machines. If we assume each proxy must be run on its own node, and that at first database nodes can hold 16 partitions, we’ll see that we need 65 proxies and 256 database machines (not including redundancy factors, which should typically multiply the cluster size by two or three times). To get started with a cluster that can grow smoothly from 64 TB to 4 PB, we can begin with roughly 600 to 1,000 server nodes, adding new ones as data size grows and we move partitions to other machines.
+
+<p>We’ve seen that even a cluster with a depth of 2 can hold a vast amount of data. Basic arithmetic shows us that by applying the same process to create a cluster with three layers of proxies, we can manage 262 petabytes on thousands of machines. Conservative estimates for the latency introduced by each layer is about 100 ms, so even without performance tuning we should see overall response times of 300 ms even with a tree depth of 3, and we should be able to manage queries over exabyte datasets in less than a second.
+
+<p>By using oversharding and iteratively replacing full shards (database nodes that host only one partition) with proxy nodes that point to another set of oversharded partitions, we can grow the cluster to very large sizes while incurring a minimum of latency.
+
+<p>Now we need to look at the mechanics of the two processes that allow the cluster to grow: moving a partition from an overcrowded node to an empty node, and splitting a large partition into many subpartitions. Moving partitions is simpler, which is why it makes sense to use it when possible, running the more resource-intensive repartition process only when partitions get large enough that only one or two can fit on each database server.
+
+<h4 id="moving">Moving Partitions</h4>
+
+<p>As we mentioned earlier, each partition is made up of <em>N</em> redundant CouchDB databases, each stored on different physical servers. To keep things easy to conceptualize, any operations should be applied to all redundant copies automatically. For the sake of discussion, we’ll just talk about the abstract partition, but be aware that the redundant nodes will all be the same size and so should require the same operations during cluster growth.
+
+<p>The simplest way to move a partition from one node to another is to create an empty database on the target node and use CouchDB replication to fill the new node with data from the old node. When the new copy of the partition is up-to-date with the original, the proxy node can be reconfigured to point to the new machine. Once the proxy points to the new partition location, one final round of replication will bring it up-to-date, and the old partition can be retired, freeing space on the original machine.
+
+<p>Another method for moving partition databases is to rsync the files on disk from the old node to the new one. Depending on how recently the partition was compacted, this should result in efficient, low-CPU initialization of a new node. Replication can then be used to bring the rsynced file up-to-date. See more about rsync and replication in <a href="replication.html">Chapter 16, Replication</a>.
+
+<h4 id="splitting">Splitting Partitions</h4>
+
+<p>The last major thing we need to run a CouchDB cluster is the capability to split an oversized partition into smaller pieces. In <a href="replication.html">Chapter 16, Replication</a>, we discussed how to do continuous replication using the <code>_changes API</code>. The <code>_changes API</code> can use filters (see <a href="notifications.html">Chapter 20, Change Notifications</a>), and replication can be configured to use a filter function to replicate only a subset of a total database. Splitting partitions is accomplished by creating the target partitions and configuring them with the range of hash keys they are interested in. They then apply filtered replication to the source partition database, requesting only documents that meet their hash criteria. The result is multiple partial copies of the source database, so that each new partition has an equal share of the data. In total, they have a complete copy of the original data. Once the replication is complete and the new partitions have also brought their redundant backups up-to-date, a proxy for the new set of partitions is brought online and the top-level proxy is pointed at it instead of the old partition. Just like with moving a partition, we should do one final round of replication after the old partition is no longer reachable by the cluster, so that any last second updates are not lost. Once that is done, we can retire the old partition so that its hardware can be reused elsewhere in the cluster.
+
+<script src="../script.js"></script>
19 draft/colophon.html
View
@@ -0,0 +1,19 @@
+<title>Colophon</title>
+
+<meta charset="utf-8">
+
+<link rel="stylesheet" href="../style.css">
+
+<link rel="prev" href="btree.html">
+
+<h2 id="colophon">Colophon</h2>
+
+<p>The animal on the cover of <em>CouchDB: The Definitive Guide</em> is a Pomeranian dog (<em>Canis familiaris</em>), a small variety of the generally larger German Spitz breed, named for the Baltic region of Pomerania (today spilt between northeastern Germany and northern Poland) where it was first bred.
+
+<p>Originally, Pomeranians were closer in size to their German Spitz relatives—weighing 30–50 pounds—and were bred as herding dogs because of their intelligence, energy, and loyalty. From the late 19th century, however, breeders began to favor increasingly smaller dogs, a move caused in large part by Queen Victoria’s affinity for that variety. Today, Pomeranians are classed as “toy dogs,” weighing only 4–7 pounds, and are particularly kept as small pets and show dogs.
+
+<p>The Pomeranian exhibits many of the physical and behavioral characteristics of its larger ancestors and relatives. It has a short, pointed muzzle, upright and pointed ears, a large bushy tail carried curled over the back, and is especially spirited and friendly. Pomeranians are also particularly noted for their double coat—a soft and dense undercoat and a long, straight and harshly textured outer coat—and come in a wide variety of colors, including white, black, brown, red, orange, sable, spotted, or any combination thereof. Because of their small size, Pomeranians are able to exercise sufficiently in small indoor spaces if taken for a daily walk, and consequently make excellent apartment pets.
+
+<p>The cover image is from Lydekker’s <em>Royal Natural History</em>. The cover font is Adobe ITC Garamond. The text font is Linotype Birka; the heading font is Adobe Myriad Condensed; and the code font is LucasFont’s TheSansMonoCondensed.
+
+<script src="../script.js"></script>
266 draft/conflicts.html
View
@@ -0,0 +1,266 @@
+<title>Conflict Management</title>
+
+<meta charset="utf-8">
+
+<link rel="stylesheet" href="../style.css">
+
+<link rel="prev" href="replication.html">
+
+<link rel="next" href="balancing.html">
+
+<h2 id="conflicts">Conflict Management</h2>
+
+<p>Suppose you are sitting in a coffee shop working on your book. J. Chris comes over and tells you about his new phone. The new phone came with a new number, and you have J. Chris dictate it while you change it using your laptop’s address book application.
+
+<p>Luckily, your address book is built on CouchDB, so when you come home, all you need to do to get your home computer up-to-date with J. Chris’s number is replicate your address book from your laptop. Neat, eh? What’s more, CouchDB has a mechanism to maintain continuous replication, so you can keep a whole set of computers in sync with the same data, whenever a network connection is available.
+
+<p>Let’s change the scenario a little bit. Since J. Chris didn’t anticipate meeting you at the coffee shop, he also sent you an email with the new number. At the time you weren’t using WiFi because you wanted concentrate on your work, so you didn’t read his email until you got home. But it was a long day and by then you had forgotten that you changed the number in the address book on your laptop. When you read the email at home, you simply copy-and-pasted the number into the address book on your home computer. Now—and here’s the twist—it turns out you entered the wrong number in your laptop’s address book.
+
+<p>You now have a document in each of the databases that has different information. This situation is called a <em>conflict</em>. Conflicts occur in distributed systems. They are a natural state of your data. How does CouchDB’s replication system deal with conflicts?
+
+<p>When you replicate two databases in CouchDB and you have conflicting changes, CouchDB will detect this and will flag the affected document with the special attribute <code>"_conflicts":true</code>. Next, CouchDB determines which of the changes will be stored as the latest revision (remember, documents in CouchDB are versioned). The version that gets picked to be the latest revision is the <em>winning revision</em>. The <em>losing revision</em> gets stored as the previous revision.
+
+<p>CouchDB does not attempt to merge the conflicting revision. Your application dictates how the merging should be done. The choice of picking the winning revision is arbitrary. In the case of the phone number, there is no way for a computer to decide on the <em>right</em> revision. This is not specific to CouchDB; no other software can do this (ever had your phone’s sync-contacts tool ask you which contact from which source to take?).
+
+<p>Replication guarantees that conflicts are detected and that each instance of CouchDB makes the same choice regarding winners and losers, independent of all the other instances. There is no group decision made; instead, a deterministic algorithm determines the order of the conflicting revision. After replication, all instances taking part have the same data. The data set is said to be in a <em>consistent state</em>. If you ask any instance for a document, you will get the same answer regardless which one you ask.
+
+<p>Whether or not CouchDB picked the version that your application needs, you need to go and resolve the conflict, just as you need to resolve a conflict in a version control system like Subversion. Simply create a version that you want to be the latest by either picking the latest, or the previous, or both (by merging them) and save it as the now latest revision. Done. Replicate again and your resolution will populate over to all other instances of CouchDB. Your conflict resolving on one node could lead to further conflicts, all of which will need to be addressed, but eventually, you will end up with a conflict-free database on all nodes.
+
+<h3 id="brain">The Split Brain</h3>
+
+<p>This is an interesting conflicts scenario in that we helped a BBC build a solution for it that is now in production. The basic setup is this: to guarantee that the company’s website is online 24/7, even in the event of the loss of a data center, it has multiple data centers backing up the website. The “loss” of a data center is a rare occasion, but it can be as simple as a network outage, where the data center is still alive and well but can’t be reached by anyone.
+
+<p>The “split brain” scenario is where two (for simplicity’s sake we’ll stick to two) data centers are up and well connected to end users, but the connection between the data centers—which is most likely not the same connection that end users use to talk to the computers in the data center—fails.
+
+<p>The inter data center connection is used to keep both centers <em>in sync</em> so that either one can take over for the other in case of a failure. If that link goes down, you end up with two halves of a system that act independently—the split brain.
+
+<p>As long as all end users can get to their data, the split brain is not scary. Resolving the split brain situation by bringing up the connection that links the data centers and starting synchronization again is where it gets hairy. Arbitrary conflict resolution, like CouchDB does by default, can lead to unwanted effects on the user’s side. Data could revert to an earlier stage and leave the impression that changes weren’t reliably saved, when in fact they were.
+
+<h3 id="resolution">Conflict Resolution by Example</h3>
+
+<p>Let’s go through an illustrated example of how conflicts emerge and how to solve them in super slow motion. <a href="#figure/1">Figure 1, “Conflict management by example: step 1”</a> illustrates the basic setup: we have two CouchDB databases, and we are replicating from database A to database B. To keep this simple, we assume triggered replication and not continuous replication, and we don’t replicate back from database B to A. All other replication scenarios can be reduced to this setup, so this explains everything we need to know.
+
+<div class="figure" id="figure/1">
+
+<img src="conflicts/01.png">
+
+<p class="caption">Figure 1. Conflict management by example: step 1
+
+</div>
+
+<p>We start out by creating a document in database A (<a href="#figure/2">Figure 2, “Conflict management by example: step 2”</a>). Note the clever use of imagery to identify a specific revision of a document. Since we are not using continuous replication, database B won’t know about the new document for now.
+
+<div class="figure" id="figure/2">
+
+<img src="conflicts/02.png">
+
+<p class="caption">Figure 2. Conflict management by example: step 2
+
+</div>
+
+<p>We now trigger replication and tell it to use database A as the source and database B as the target (<a href="#figure/3">Figure 3, “Conflict management by example: step 3”</a>). Our document gets copied over to database B. To be precise, the latest revision of our document gets copied over.
+
+<div class="figure" id="figure/3">
+
+<img src="conflicts/03.png">
+
+<p class="caption">Figure 3. Conflict management by example: step 3
+
+</div>
+
+<p>Now we go to database B and update the document (<a href="#figure/4">Figure 4, “Conflict management by example: step 4”</a>). We change some values and upon change, CouchDB generates a new revision for us. Note that this revision has a new image. Node A is ignorant of any activity.
+
+<div class="figure" id="figure/4">
+
+<img src="conflicts/04.png">
+
+<p class="caption">Figure 4. Conflict management by example: step 4
+
+</div>
+
+<p>Now we make a change to our document in database A by changing some other values (<a href="#figure/5">Figure 5, “Conflict management by example: step 5”</a>). See how it makes a different image for us to see the difference? It is important to note that this is still the same document. It’s just that there are two different revisions of that same document in each database.
+
+<div class="figure" id="figure/5">
+
+<img src="conflicts/05.png">
+
+<p class="caption">Figure 5. Conflict management by example: step 5
+
+</div>
+
+<p>Now we trigger replication again from database A to database B as before (<a href="#figure/6">Figure 6, “Conflict management by example: step 6”</a>). By the way, it doesn’t make a difference if the two databases live in the same CouchDB server or on different servers connected over a network.
+
+<div class="figure" id="figure/6">
+
+<img src="conflicts/06.png">
+
+<p class="caption">Figure 6. Conflict management by example: step 6
+
+</div>
+
+<p>When replicating, CouchDB detects that there are two different revisions for the same document, and it creates a conflict (<a href="#figure/7">Figure 7, “Conflict management by example: step 7”</a>). A document conflict means that there are now two latest revisions for this document.
+
+<div class="figure" id="figure/7">
+
+<img src="conflicts/07.png">
+
+<p class="caption">Figure 7. Conflict management by example: step 7
+
+</div>
+
+<p>Finally, we tell CouchDB which version we would like to be the latest revision by resolving the conflict (<a href="#figure/8">Figure 8, “Conflict management by example: step 8”</a>). Now both databases have the same data.
+
+<div class="figure" id="figure/8">
+
+<img src="conflicts/08.png">
+
+<p class="caption">Figure 8. Conflict management by example: step 8
+
+</div>
+
+<p>Other possible outcomes include choosing the other revision and replicating that decision back to database A, or creating yet another revision in database B that includes parts of both conflicting revisions (a <em>merge</em>) and replicating that back to database A.
+
+<h3 id="working">Working with Conflicts</h3>
+
+<p>Now that we’ve walked through replication with pretty pictures, let’s get our hands dirty and see what the API calls and responses for this and other scenarios look like. We’ll be continuing <a href="api.html">Chapter 4, The Core API</a> by using <code>curl</code> on the command line to make raw API requests.
+
+<p>First, we create two databases that we can use for replication. These live on the same CouchDB instance, but they might as well live on a remote instance—CouchDB doesn’t care. To save us some typing, we create a shell variable for our CouchDB base URL that we want to talk to. We then create two databases, <code>db</code> and <code>db-replica</code>:
+
+<pre>
+HOST="http://127.0.0.1:5984"
+
+&gt; curl -X PUT $HOST/db
+{"ok":true}
+
+&gt; curl -X PUT $HOST/db-replica
+{"ok":true}
+</pre>
+
+<p>In the next step, we create a simple document <code>{"count":1}</code> in <code>db</code> and trigger replication to <code>db-replica</code>:
+
+<pre>
+&gt; curl -X PUT $HOST/db/foo -d '{"count":1}'
+{"ok":true,"id":"foo","rev":"1-74620ecf527d29daaab9c2b465fbce66"}
+
+&gt; curl -X POST $HOST/_replicate -d '{"source":"db","target":"http://127.0.0.1:5984/db-replica"}'
+{"ok":true,...,"docs_written":1,"doc_write_failures":0}]}
+</pre>
+
+<p>We skip a bit of the output of the replication session (see <a href="replication.html">Chapter 16, Replication</a> for details). If you see <code>"docs_written":1</code> and <code>"doc_write_failures":0</code>, our document made it over to <code>db-replica</code>. We now update the document to <code>{"count":2}</code> in <code>db-replica</code>. Note that we now need to include the correct <code>_rev</code> property.
+
+<pre>
+&gt; curl -X PUT $HOST/db-replica/foo -d '{"count":2,"_rev":"1-74620ecf527d29daaab9c2b465fbce66"}'
+{"ok":true,"id":"foo","rev":"2-de0ea16f8621cbac506d23a0fbbde08a"}
+</pre>
+
+<p>Next, we create the conflict! We change our document on <code>db</code> to <code>{"count":3}</code>. Our document is now logically in conflict, but CouchDB doesn’t know about it until we replicate again:
+
+<pre>
+&gt; curl -X PUT $HOST/db/foo -d '{"count":3,"_rev":"1-74620ecf527d29daaab9c2b465fbce66"}'
+{"ok":true,"id":"foo","rev":"2-7c971bb974251ae8541b8fe045964219"}
+
+&gt; curl -X POST $HOST/_replicate -d '{"source":"db","target":"http://127.0.0.1:5984/db-replica"}'
+{"ok":true,..."docs_written":1,"doc_write_failures":0}]}
+</pre>
+
+<p>To see that we have a conflict, we create a simple view in <code>db-replica</code>. The map function looks like this:
+
+<pre>
+function(doc) {
+ if(doc._conflicts) {
+ emit(doc._conflicts, null);
+ }
+}
+</pre>
+
+<p>When we query this view, we get this result:
+
+<pre>
+{"total_rows":1,"offset":0,"rows":[
+{"id":"foo","key":["2-7c971bb974251ae8541b8fe045964219"],"value":null}
+]}
+</pre>
+
+<p>The <code>key</code> here corresponds to the <code>doc._conflicts</code> property of our document in <code>db-replica</code>. It is an array listing all <em>conflicting revisions</em>. We see that the revision we wrote on <code>db</code> (<code>{"count":3}</code>) is in conflict. CouchDB’s automatic promotion of one revision to be the winning revision chose our first change (<code>{"count":2}</code>). To verify that, we just request that document from <code>db-replica</code>:
+
+<pre>
+&gt; curl -X GET $HOST/db-replica/foo
+{"_id":"foo","_rev":"2-de0ea16f8621cbac506d23a0fbbde08a","count":2}
+</pre>
+
+<p>To resolve the conflict, we need to determine which one we want to keep.
+
+<div class="aside note">
+
+<p><strong>How Does CouchDB Decide Which Revision to Use?</strong>
+
+<p>CouchDB guarantees that each instance that sees the same conflict comes up with the same winning and losing revisions. It does so by running a deterministic algorithm to pick the winner. The application should not rely on the details of this algorithm and must always resolve conflicts. We’ll tell you how it works anyway.
+
+<p>Each revision includes a list of previous revisions. The revision with the longest revision history list becomes the winning revision. If they are the same, the <code>_rev</code> values are compared in ASCII sort order, and the highest wins. So, in our example, <code>2-de0ea16f8621cbac506d23a0fbbde08a</code> beats <code>2-7c971bb974251ae8541b8fe045964219</code>.
+
+<p>One advantage of this algorithm is that CouchDB nodes do not have to talk to each other to agree on winning revisions. We already learned that the network is prone to errors and avoiding it for conflict resolution makes CouchDB very robust.
+
+</div>
+
+<p>Let’s say we want to keep the highest value. This means we don’t agree with CouchDB’s automatic choice. To do this, we first overwrite the target document with our value and then simply delete the revision we don’t like:
+
+<pre>
+&gt; curl -X DELETE $HOST/db-replica/foo?rev=2-de0ea16f8621cbac506d23a0fbbde08a
+{"ok":true,"id":"foo","rev":"3-bfe83a296b0445c4d526ef35ef62ac14"}
+
+&gt; curl -X PUT $HOST/db-replica/foo -d '{"count":3,"_rev":"2-7c971bb974251ae8541b8fe045964219"}'
+{"ok":true,"id":"foo","rev":"3-5d0319b075a21b095719bc561def7122"}
+</pre>
+
+<p>CouchDB creates yet another revision that reflects our decision. Note that the <code>3-</code> didn’t get incremented this time. We didn’t create a new version of the document body; we just deleted a conflicting revision. To see that all is well, we check whether our revision ended up in the document.
+
+<pre>
+&gt; curl -X GET $HOST/db-replica/foo
+{"_id":"foo","_rev":"3-5d0319b075a21b095719bc561def7122","count":3}
+</pre>
+
+<p>We also verify that our document is no longer in conflict by querying our conflicts view again, and we see that there are no more conflicts:
+
+<pre>
+{"total_rows":0,"offset":0,"rows":[
+]}
+</pre>
+
+<p>Finally, we replicate from <code>db-replica</code> back to <code>db</code> by simply swapping <code>source</code> and <code>target</code> in our request to <code>_replicate</code>:
+
+<pre>
+&gt; curl -X POST $HOST/_replicate -d '{"target":"db","source":"http://127.0.0.1:5984/db-replica"}'
+</pre>
+
+<p>We see that our revision ends up in <code>db</code>, too:
+
+<pre>
+&gt; curl -X GET $HOST/db/foo
+{"_id":"foo","_rev":"3-5d0319b075a21b095719bc561def7122","count":3}
+</pre>
+
+<p>And we’re done.
+
+<h3 id="deterministic">Deterministic Revision IDs</h3>
+
+<p>Let’s have a look at this revision ID: <code>3-5d0319b075a21b095719bc561def7122</code>. Parts of the format might look familiar. The first part is an integer followed by a dash (<code>3-</code>). The integer increments for each new revision the document receives. Updates to the same document on multiple instances create their own independent increments. When replicating, CouchDB knows that there are two different revisions (like in our previous example) by looking at the second part.
+
+<p>The second part is an md5-hash over a set of document properties: the JSON body, the attachments, and the <code>_deleted</code> flag. This allows CouchDB to save on replication time in case you make the same change to the same document on two instances. Earlier versions (0.9 and back) used random integers to specify revisions, and making the same change on two instances would result in two different revision IDs, creating a conflict where it was not really necessary. CouchDB 0.10 and above uses deterministic revision IDs using the md5 hash.
+
+<p>For example, let’s create two documents, <code>a</code> and <code>b</code>, with the same contents:
+
+<pre>
+&gt; curl -X PUT $HOST/db/a -d '{"a":1}'
+{"ok":true,"id":"a","rev":"1-23202479633c2b380f79507a776743d5"}
+
+&gt; curl -X PUT $HOST/db/b -d '{"a":1}'
+{"ok":true,"id":"b","rev":"1-23202479633c2b380f79507a776743d5"}
+</pre>
+
+<p>Both revision IDs are the same, a consequence of the deterministic algorithm used by CouchDB.
+
+<h3 id="wrap">Wrapping Up</h3>
+
+<p>This concludes our tour of the conflict management system. You should now be able to create distributed setups that deal with conflicts in a proper way.
+
+<script src="../script.js"></script>
BIN  draft/conflicts/01.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/conflicts/02.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/conflicts/03.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/conflicts/04.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/conflicts/05.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/conflicts/06.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/conflicts/07.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/conflicts/08.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
213 draft/consistency.html
View
@@ -0,0 +1,213 @@
+<title>Eventual Consistency</title>
+
+<meta charset="utf-8">
+
+<link rel="stylesheet" href="../style.css">
+
+<link rel="prev" href="why.html">
+
+<link rel="next" href="tour.html">
+
+<h2 id="consistency">Eventual Consistency</h2>
+
+<p>In the previous chapter, we saw that CouchDB’s flexibility allows us to evolve our data as our applications grow and change. In this chapter, we’ll explore how working “with the grain” of CouchDB promotes simplicity in our applications and helps us naturally build scalable, distributed systems.
+
+<h3 id="grain">Working with the Grain</h3>
+
+<p>A <em>distributed system</em> is a system that operates robustly over a wide network. A particular feature of network computing is that network links can potentially disappear, and there are plenty of strategies for managing this type of network segmentation. CouchDB differs from others by accepting eventual consistency, as opposed to putting absolute consistency ahead of raw availability, like RDBMS or Paxos. What these systems have in common is an awareness that data acts differently when many people are accessing it simultaneously. Their approaches differ when it comes to which aspects of <em>consistency</em>, <em>availability</em>, or <em>partition tolerance</em> they prioritize.
+
+<p>Engineering distributed systems is tricky. Many of the caveats and “gotchas” you will face over time aren’t immediately obvious. We don’t have all the solutions, and CouchDB isn’t a panacea, but when you work with CouchDB’s grain rather than against it, the path of least resistance leads you to naturally scalable applications.
+
+<p>Of course, building a distributed system is only the beginning. A website with a database that is available only half the time is next to worthless. Unfortunately, the traditional relational database approach to consistency makes it very easy for application programmers to rely on global state, global clocks, and other high availability no-nos, without even realizing that they’re doing so. Before examining how CouchDB promotes scalability, we’ll look at the constraints faced by a distributed system. After we’ve seen the problems that arise when parts of your application can’t rely on being in constant contact with each other, we’ll see that CouchDB provides an intuitive and useful way for modeling applications around high availability.
+
+<h3 id="cap">The CAP Theorem</h3>
+
+<p>The CAP theorem describes a few different strategies for distributing application logic across networks. CouchDB’s solution uses replication to propagate application changes across participating nodes. This is a fundamentally different approach from consensus algorithms and relational databases, which operate at different intersections of consistency, availability, and partition tolerance.
+
+<p>The CAP theorem, shown in <a href="#figure/1">Figure 1, “The CAP theorem”</a>, identifies three distinct concerns:
+
+<dl>
+
+<dt>Consistency</dt>
+
+<dd>All database clients see the same data, even with concurrent updates.</dd>
+
+<dt>Availability</dt>
+
+<dd>All database clients are able to access some version of the data.</dd>
+
+<dt>Partition tolerance</dt>
+
+<dd>The database can be split over multiple servers.</dd>
+
+</dl>
+
+<p>Pick two.
+
+<div class="figure" id="figure/1">
+
+<img src="consistency/01.png">
+
+<p class="caption">Figure 1. The CAP theorem
+
+</div>
+
+<p>When a system grows large enough that a single database node is unable to handle the load placed on it, a sensible solution is to add more servers. When we add nodes, we have to start thinking about how to partition data between them. Do we have a few databases that share exactly the same data? Do we put different sets of data on different database servers? Do we let only certain database servers write data and let others handle the reads?
+
+<p>Regardless of which approach we take, the one problem we’ll keep bumping into is that of keeping all these database servers in synchronization. If you write some information to one node, how are you going to make sure that a read request to another database server reflects this newest information? These events might be milliseconds apart. Even with a modest collection of database servers, this problem can become extremely complex.
+
+<p>When it’s absolutely critical that all clients see a consistent view of the database, the users of one node will have to wait for any other nodes to come into agreement before being able to read or write to the database. In this instance, we see that <em>availability</em> takes a backseat to <em>consistency</em>. However, there are situations where availability trumps consistency:
+
+<blockquote>
+
+<p>Each node in a system should be able to make decisions purely based on local state. If you need to do something under high load with failures occurring and you need to reach agreement, you’re lost. If you’re concerned about scalability, any algorithm that forces you to run agreement will eventually become your bottleneck. Take that as a given.
+
+<p class="attribution">&mdash;Werner Vogels, Amazon CTO and Vice President
+
+</blockquote>
+
+<p>If availability is a priority, we can let clients write data to one node of the database without waiting for other nodes to come into agreement. If the database knows how to take care of reconciling these operations between nodes, we achieve a sort of “eventual consistency” in exchange for high availability. This is a surprisingly applicable trade-off for many applications.
+
+<p>Unlike traditional relational databases, where each action performed is necessarily subject to database-wide consistency checks, CouchDB makes it really simple to build applications that sacrifice immediate consistency for the huge performance improvements that come with simple distribution.
+
+<h3 id="consistency">Local Consistency</h3>
+
+<p>Before we attempt to understand how CouchDB operates in a cluster, it’s important that we understand the inner workings of a single CouchDB node. The CouchDB API is designed to provide a convenient but thin wrapper around the database core. By taking a closer look at the structure of the database core, we’ll have a better understanding of the API that surrounds it.
+
+<h4 id="key">The Key to Your Data</h4>
+
+<p>At the heart of CouchDB is a powerful <em>B-tree</em> storage engine. A B-tree is a sorted data structure that allows for searches, insertions, and deletions in logarithmic time. As <a href="#figure/2">Figure 2, “Anatomy of a view request”</a> illustrates, CouchDB uses this B-tree storage engine for all internal data, documents, and views. If we understand one, we will understand them all.
+
+<div class="figure" id="figure/2">
+
+<img src="consistency/02.png">
+
+<p class="caption">Figure 2. Anatomy of a view request
+
+</div>
+
+<p>CouchDB uses MapReduce to compute the results of a view. MapReduce makes use of two functions, “map” and “reduce,” which are applied to each document in isolation. Being able to isolate these operations means that view computation lends itself to parallel and incremental computation. More important, because these functions produce key/value pairs, CouchDB is able to insert them into the B-tree storage engine, sorted by key. Lookups by key, or key range, are extremely efficient operations with a B-tree, described in <em>big O notation</em> as <em>O(log N)</em> and <em>O(log N + K)</em>, respectively.
+
+<p>In CouchDB, we access documents and view results by key or key range. This is a direct mapping to the underlying operations performed on CouchDB’s B-tree storage engine. Along with document inserts and updates, this direct mapping is the reason we describe CouchDB’s API as being a thin wrapper around the database core.
+
+<p>Being able to access results by key alone is a very important restriction because it allows us to make huge performance gains. As well as the massive speed improvements, we can partition our data over multiple nodes, without affecting our ability to query each node in isolation. <em>BigTable</em>, <em>Hadoop</em>, <em>SimpleDB</em>, and <em>memcached</em> restrict object lookups by key for exactly these reasons.
+
+<h4 id="locking">No Locking</h4>
+
+<p>A table in a relational database is a single data structure. If you want to modify a table—say, update a row—the database system must ensure that nobody else is trying to update that row and that nobody can read from that row while it is being updated. The common way to handle this uses what’s known as a <em>lock</em>. If multiple clients want to access a table, the first client gets the lock, making everybody else wait. When the first client’s request is processed, the next client is given access while everybody else waits, and so on. This serial execution of requests, even when they arrived in parallel, wastes a significant amount of your server’s processing power. Under high load, a relational database can spend more time figuring out who is allowed to do what, and in which order, than it does doing any actual work.
+
+<p>Instead of locks, CouchDB uses <em>Multi-Version Concurrency Control (MVCC)</em> to manage concurrent access to the database. <a href="#figure/3">Figure 3, “MVCC means no locking”</a> illustrates the differences between MVCC and traditional locking mechanisms. MVCC means that CouchDB can run at full speed, all the time, even under high load. Requests are run in parallel, making excellent use of every last drop of processing power your server has to offer.
+
+<div class="figure" id="figure/3">
+
+<img src="consistency/03.png">
+
+<p class="caption">Figure 3. MVCC means no locking
+
+</div>
+
+<p>Documents in CouchDB are versioned, much like they would be in a regular version control system such as Subversion. If you want to change a value in a document, you create an entire new version of that document and save it over the old one. After doing this, you end up with two versions of the same document, one old and one new.
+
+<p>How does this offer an improvement over locks? Consider a set of requests wanting to access a document. The first request reads the document. While this is being processed, a second request changes the document. Since the second request includes a completely new version of the document, CouchDB can simply append it to the database without having to wait for the read request to finish.
+
+<p>When a third request wants to read the same document, CouchDB will point it to the new version that has just been written. During this whole process, the first request could still be reading the original version.
+
+<p>A read request will always see the most recent snapshot of your database.
+
+<h4 id="validation">Validation</h4>
+
+<p>As application developers, we have to think about what sort of input we should accept and what we should reject. The expressive power to do this type of validation over complex data <em>within</em> a traditional relational database leaves a lot to be desired. Fortunately, CouchDB provides a powerful way to perform per-document validation from within the database.
+
+<p>CouchDB can validate documents using JavaScript functions similar to those used for MapReduce. Each time you try to modify a document, CouchDB will pass the validation function a copy of the existing document, a copy of the new document, and a collection of additional information, such as user authentication details. The validation function now has the opportunity to approve or deny the update.
+
+<p>By working with the grain and letting CouchDB do this for us, we save ourselves a tremendous amount of CPU cycles that would otherwise have been spent serializing object graphs from SQL, converting them into domain objects, and using those objects to do application-level validation.
+
+<h3 id="distributed">Distributed Consistency</h3>
+
+<p>Maintaining consistency within a single database node is relatively easy for most databases. The real problems start to surface when you try to maintain consistency between multiple database servers. If a client makes a write operation on server <em>A</em>, how do we make sure that this is consistent with server <em>B</em>, or <em>C</em>, or <em>D</em>? For relational databases, this is a very complex problem with entire books devoted to its solution. You could use multi-master, master/slave, partitioning, sharding, write-through caches, and all sorts of other complex techniques.
+
+<h4 id="incremental">Incremental Replication</h4>
+
+<p>Because CouchDB operations take place within the context of a single document, if you want to use two database nodes, you no longer have to worry about them staying in constant communication. CouchDB achieves <em>eventual consistency</em> between databases by using incremental replication, a process where document changes are periodically copied between servers. We are able to build what’s known as a <em>shared nothing</em> cluster of databases where each node is independent and self-sufficient, leaving no single point of contention across the system.
+
+<p>Need to scale out your CouchDB database cluster? Just throw in another server.
+
+<p>As illustrated in <a href="#figure/4">Figure 4, “Incremental replication between CouchDB nodes”</a>, with CouchDB’s incremental replication, you can synchronize your data between any two databases however you like and whenever you like. After replication, each database is able to work independently.
+
+<p>You could use this feature to synchronize database servers within a cluster or between data centers using a job scheduler such as <em>cron</em>, or you could use it to synchronize data with your laptop for offline work as you travel. Each database can be used in the usual fashion, and changes between databases can be synchronized later in both directions.
+
+<div class="figure" id="figure/4">
+
+<img src="consistency/04.png">
+
+<p class="caption">Figure 4. Incremental replication between CouchDB nodes
+
+</div>
+
+<p>What happens when you change the same document in two different databases and want to synchronize these with each other? CouchDB’s replication system comes with automatic conflict detection <em>and</em> resolution. When CouchDB detects that a document has been changed in both databases, it flags this document as being in conflict, much like they would be in a regular version control system.
+
+<p>This isn’t as troublesome as it might first sound. When two versions of a document conflict during replication, the <em>winning</em> version is saved as the most recent version in the document’s history. Instead of throwing the <em>losing</em> version away, as you might expect, CouchDB saves this as a previous version in the document’s history, so that you can access it if you need to. This happens automatically and consistently, so both databases will make exactly the same choice.
+
+<p>It is up to you to handle conflicts in a way that makes sense for your application. You can leave the chosen document versions in place, revert to the older version, or try to merge the two versions and save the result.
+
+<h4 id="study">Case Study</h4>
+
+<p>Greg Borenstein, a friend and coworker, built a small library for converting Songbird playlists to JSON objects and decided to store these in CouchDB as part of a backup application. The completed software uses CouchDB’s MVCC and document revisions to ensure that Songbird playlists are backed up robustly between nodes.
+
+<div class="aside note">
+
+<p>Songbird is a free software media player with an integrated web browser, based on the Mozilla XULRunner platform. Songbird is available for Microsoft Windows, Apple Mac OS X, Solaris, and Linux.
+
+</div>
+
+<p>Let’s examine the workflow of the Songbird backup application, first as a user backing up from a single computer, and then using Songbird to synchronize playlists between multiple computers. We’ll see how document revisions turn what could have been a hairy problem into something that <em>just works</em>.
+
+<p>The first time we use this backup application, we feed our playlists to the application and initiate a backup. Each playlist is converted to a JSON object and handed to a CouchDB database. As illustrated in <a href="#figure/5">Figure 5, “Backing up to a single database”</a>, CouchDB hands back the document ID and revision of each playlist as it’s saved to the database.
+
+<div class="figure" id="figure/5">
+
+<img src="consistency/05.png">
+
+<p class="caption">Figure 5. Backing up to a single database
+
+</div>
+
+<p>After a few days, we find that our playlists have been updated and we want to back up our changes. After we have fed our playlists to the backup application, it fetches the latest versions from CouchDB, along with the corresponding document revisions. When the application hands back the new playlist document, CouchDB requires that the document revision is included in the request.
+
+<p>CouchDB then makes sure that the document revision handed to it in the request matches the current revision held in the database. Because CouchDB updates the revision with every modification, if these two are out of synchronization it suggests that someone else has made changes to the document between the time we requested it from the database and the time we sent our updates. Making changes to a document after someone else has modified it without first inspecting those changes is usually a bad idea.
+
+<p>Forcing clients to hand back the correct document revision is the heart of CouchDB’s optimistic concurrency.
+
+<p>We have a laptop we want to keep synchronized with our desktop computer. With all our playlists on our desktop, the first step is to “restore from backup” onto our laptop. This is the first time we’ve done this, so afterward our laptop should hold an exact replica of our desktop playlist collection.
+
+<p>After editing our Argentine Tango playlist on our laptop to add a few new songs we’ve purchased, we want to save our changes. The backup application replaces the playlist document in our laptop CouchDB database and a new document revision is generated. A few days later, we remember our new songs and want to copy the playlist across to our desktop computer. As illustrated in <a href="#figure/6">Figure 6, “Synchronizing between two databases”</a>, the backup application copies the new document and the new revision to the desktop CouchDB database. Both CouchDB databases now have the same document revision.
+
+<div class="figure" id="figure/6">
+
+<img src="consistency/06.png">
+
+<p class="caption">Figure 6. Synchronizing between two databases
+
+</div>
+
+<p>Because CouchDB tracks document revisions, it ensures that updates like these will work only if they are based on current information. If we had made modifications to the playlist backups between synchronization, things wouldn’t go as smoothly.
+
+<p>We back up some changes on our laptop and forget to synchronize. A few days later, we’re editing playlists on our desktop computer, make a backup, and want to synchronize this to our laptop. As illustrated in <a href="#figure/7">Figure 7, “Synchronization conflicts between two databases”</a>, when our backup application tries to replicate between the two databases, CouchDB sees that the changes being sent from our desktop computer are modifications of out-of-date documents and helpfully informs us that there has been a conflict.
+
+<p>Recovering from this error is easy to accomplish from an application perspective. Just download CouchDB’s version of the playlist and provide an opportunity to merge the changes or save local modifications into a new playlist.
+
+<div class="figure" id="figure/7">
+
+<img src="consistency/07.png">
+
+<p class="caption">Figure 7. Synchronization conflicts between two databases
+
+</div>
+
+<h3 id="wrap">Wrapping Up</h3>
+
+<p>CouchDB’s design borrows heavily from web architecture and the lessons learned deploying massively distributed systems on that architecture. By understanding why this architecture works the way it does, and by learning to spot which parts of your application can be easily distributed and which parts cannot, you’ll enhance your ability to design distributed and scalable applications, with CouchDB or without it.
+
+<p>We’ve covered the main issues surrounding CouchDB’s consistency model and hinted at some of the benefits to be had when you work <em>with</em> CouchDB and not against it. But enough theory—let’s get up and running and see what all the fuss is about!
+
+<script src="../script.js"></script>
BIN  draft/consistency/01.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/consistency/02.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/consistency/03.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/consistency/04.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/consistency/05.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/consistency/06.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
BIN  draft/consistency/07.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
421 draft/cookbook.html
View
@@ -0,0 +1,421 @@
+<title>View Cookbook for SQL Jockeys</title>
+
+<meta charset="utf-8">
+
+<link rel="stylesheet" href="../style.css">
+
+<link rel="prev" href="notifications.html">
+
+<link rel="next" href="security.html">
+
+<h2 id="cookbook">View Cookbook for SQL Jockeys</h2>
+
+<p>This is a collection of some common SQL queries and how to get the same result in CouchDB. The key to remember here is that CouchDB does not work like an SQL database at all and that best practices from the SQL world do not translate well or at all to CouchDB. This chapter’s “cookbook” assumes that you are familiar with the CouchDB basics such as creating and updating databases and documents.
+
+<h3 id="using">Using Views</h3>
+
+<p>How you would do this in SQL:
+
+<pre>
+CREATE TABLE
+</pre>
+
+<p>or:
+
+<pre>
+ALTER TABLE
+</pre>
+
+<p>Using views is a two-step process. First you <em>define</em> a view; then you <em>query</em> it. This is analogous to defining a table structure (with indexes) using <code>CREATE TABLE</code> or <code>ALTER TABLE</code> and querying it using an SQL query.
+
+<h4 id="defining">Defining a View</h4>
+
+<p>Defining a view is done by creating a special document in a CouchDB database. The only real specialness is the <code>_id</code> of the document, which starts with <code>_design/</code>—for example, <code>_design/application</code>. Other than that, it is just a regular CouchDB document. To make sure CouchDB understands that you are defining a view, you need to prepare the contents of that design document in a special format. Here is an example:
+
+<pre>
+{
+ "_id": "_design/application",
+ "_rev": "1-C1687D17",
+ "views": {
+ "viewname": {
+ "map": "function(doc) { ... }",
+ "reduce": "function(keys, values) { ... }"
+ }
+ }
+}
+</pre>
+
+<p>We are defining a view <code>viewname</code>. The definition of the view consists of two functions: the <em>map</em> function and the <em>reduce</em> function. Specifying a reduce function is optional. We’ll look at the nature of the functions later. Note that <code>viewname</code> can be whatever you like: <code>users</code>, <code>by-name</code>, or <code>by-date</code> are just some examples.
+
+<p>A single design document can also include multiple view definitions, each identified by a unique name:
+
+<pre>
+{
+ "_id": "_design/application",
+ "_rev": "1-C1687D17",
+ "views": {
+ "viewname": {
+ "map": "function(doc) { ... }",
+ "reduce": "function(keys, values) { ... }"
+ },
+ "anotherview": {
+ "map": "function(doc) { ... }",
+ "reduce": "function(keys, values) { ... }"
+ }
+ }
+}
+</pre>
+
+<h4 id="querying">Querying a View</h4>
+
+<p>The name of the design document and the name of the view are significant for querying the view. To query the view <code>viewname</code>, you perform an HTTP <code>GET</code> request to the following URI:
+
+<pre>
+/database/_design/application/_view/viewname
+</pre>
+
+<p><code>database</code> is the name of the database you created your design document in. Next up is the design document name, and then the view name prefixed with <code>_view/</code>. To query <code>anotherview</code>, replace <code>viewname</code> in that URI with <code>anotherview</code>. If you want to query a view in a different design document, adjust the design document name.
+
+<h4 id="mapreduce">MapReduce Functions</h4>
+
+<p>MapReduce is a concept that solves problems by applying a two-step process, aptly named the <em>map</em> phase and the <em>reduce</em> phase. The map phase looks at all documents in CouchDB separately one after the other and creates a <em>map result</em>. The map result is an ordered list of key/value pairs. Both <code>key</code> and <code>value</code> can be specified by the user writing the map function. A map function may call the built-in <code>emit(key, value)</code> function 0 to <em>N</em> times per document, creating a row in the map result per invocation.
+
+<p>CouchDB is smart enough to run a map function only once for every document, even on subsequent queries on a view. Only changes to documents or new documents need to be processed anew.
+
+<h5 id="map">Map functions</h5>
+
+<p>Map functions run in isolation for every document. They can’t modify the document, and they can’t talk to the outside world—they can’t have <em>side effects</em>. This is required so that CouchDB can guarantee correct results without having to recalculate a complete result when only one document gets changed.
+
+<p>The map result looks like this:
+
+<pre>
+{"total_rows":3,"offset":0,"rows":[
+{"id":"fc2636bf50556346f1ce46b4bc01fe30","key":"Lena","value":5},
+{"id":"1fb2449f9b9d4e466dbfa47ebe675063","key":"Lisa","value":4},
+{"id":"8ede09f6f6aeb35d948485624b28f149","key":"Sarah","value":6}
+]}
+</pre>
+
+<p>It is a list of rows sorted by the value of <code>key</code>. The <code>id</code> is added automatically and refers back to the document that created this row. The <code>value</code> is the data you’re looking for. For example purposes, it’s the girl’s age.
+
+<p>The map function that produces this result is:
+
+<pre>
+function(doc) {
+ if(doc.name &amp;&amp; doc.age) {
+ emit(doc.name, doc.age);
+ }
+}
+</pre>
+
+<p>It includes the <code>if</code> statement as a sanity check to ensure that we’re operating on the right fields and calls the <code>emit</code> function with the name and age as the key and value.
+
+<h5 id="reduce">Reduce functions</h5>
+
+<p>Reduce functions are explained in <a href="#aggregate">the section called “Aggregate Functions”</a>.
+
+<h3 id="key">Look Up by Key</h3>
+
+<p>How you would do this in SQL:
+
+<pre>
+SELECT <em>field</em> FROM <em>table</em> WHERE value="<em>searchterm</em>"
+</pre>
+<p>Use case: get a <em>result</em> (which can be a record or set of records) associated with a <em>key</em> (<code>"searchterm"</code>).
+
+<p>To look something up quickly, regardless of the storage mechanism, an index is needed. An index is a data structure optimized for quick search and retrieval. CouchDB’s map result is stored in such an index, which happens to be a B+ tree.
+
+<p>To look up a value by <code>"searchterm"</code>, we need to put all values into the key of a view. All we need is a simple map function:
+
+<pre>
+function(doc) {
+ if(doc.value) {
+ emit(doc.value, null);
+ }
+}
+</pre>
+
+<p>This creates a list of documents that have a <code>value</code> field sorted by the data in the <code>value</code> field. To find all the records that match <code>"searchterm"</code>, we query the view and specify the search term as a query parameter:
+
+<pre>
+/database/_design/application/_view/viewname?key="searchterm"
+</pre>
+
+<p>Consider the documents from the previous section, and say we’re indexing on the <code>age</code> field of the documents to find all the five-year-olds:
+
+<pre>
+function(doc) {
+ if(doc.age &amp;&amp; doc.name) {
+ emit(doc.age, doc.name);
+ }
+}
+</pre>
+
+<p>Query:
+
+<pre>
+/ladies/_design/ladies/_view/age?key=5
+</pre>
+
+<p>Result:
+
+<pre>
+{"total_rows":3,"offset":1,"rows":[
+{"id":"fc2636bf50556346f1ce46b4bc01fe30","key":5,"value":"Lena"}
+]}
+</pre>
+
+<p>Easy.
+
+<p>Note that you have to emit a value. The view result includes the associated document ID in every row. We can use it to look up more data from the document itself. We can also use the <code>?include_docs=true</code> parameter to have CouchDB fetch the documents individually for us.
+
+<h3 id="prefix">Look Up by Prefix</h3>
+
+<p>How you would do this in SQL:
+
+<pre>
+SELECT <em>field</em> FROM <em>table</em> WHERE <em>value</em> LIKE "<em>searchterm</em>%"
+</pre>
+
+<p>Use case: find all documents that have a field value that starts with <code>searchterm</code>. For example, say you stored a MIME type (like <code>text/html</code> or <code>image/jpg</code>) for each document and now you want to find all documents that are images according to the MIME type.
+
+<p>The solution is very similar to the previous example: all we need is a map function that is a little more clever than the first one. But first, an example document:
+
+<pre>
+{
+ "_id": "Hugh Laurie",
+ "_rev": "1-9fded7deef52ac373119d05435581edf",
+ "mime-type": "image/jpg",
+ "description": "some dude"
+}
+</pre>
+
+<p>The clue lies in extracting the prefix that we want to search for from our document and putting it into our view index. We use a regular expression to match our prefix:
+
+<pre>
+function(doc) {
+ if(doc["mime-type"]) {
+ // from the start (^) match everything that is not a slash ([^\/]+) until
+ // we find a slash (\/). Slashes needs to be escaped with a backslash (\/)
+ var prefix = doc["mime-type"].match(/^[^\/]+\//);
+ if(prefix) {
+ emit(prefix, null);
+ }
+ }
+}
+</pre>
+
+<p>We can now query this view with our desired MIME type prefix and not only find all images, but also text, video, and all other formats:
+
+<pre>
+/files/_design/finder/_view/by-mime-type?key="image/"
+</pre>
+
+<h3 id="aggregate">Aggregate Functions</h3>
+
+<p>How you would do this in SQL:
+
+<pre>
+SELECT COUNT(<em>field</em>) FROM <em>table</em>
+</pre>
+
+<p>Use case: calculate a derived value from your data.
+
+<p>We haven’t explained reduce functions yet. Reduce functions are similar to aggregate functions in SQL. They compute a value over multiple documents.
+
+<p>To explain the mechanics of reduce functions, we’ll create one that doesn’t make a whole lot of sense. But this example is easy to understand. We’ll explore more useful reductions later.
+
+<p>Reduce functions operate on the output of the map function (also called the <em>map re⁠sult</em> or <em>intermediate result</em>). The reduce function’s job, unsurprisingly, is to reduce the list that the map function produces.
+
+<p>Here’s what our summing reduce function looks like:
+
+<pre>
+function(keys, values) {
+ var sum = 0;
+ for(var idx in values) {
+ sum = sum + values[idx];
+ }
+ return sum;
+}
+</pre>
+
+<p>Here’s an alternate, more idiomatic JavaScript version:
+
+<pre>
+function(keys, values) {
+ var sum = 0;
+ values.forEach(function(element) {
+ sum = sum + element;
+ });
+ return sum;
+}
+</pre>
+
+<p>This reduce function takes two arguments: a list of <code>keys</code> and a list of <code>values</code>. For our summing purposes we can ignore the <code>keys</code>-list and consider only the <code>value</code> list. We’re looping over the list and add each item to a running total that we’re returning at the end of the function.
+
+<p>You’ll see one difference between the map and the reduce function. The map function uses <code>emit()</code> to create its result, whereas the reduce function returns a value.
+
+<p>For example, from a list of integer values that specify the age, calculate the sum of all years of life for the news headline, “786 life years present at event.” A little contrived, but very simple and thus good for demonstration purposes. Consider the documents and the map view we used earlier in this chapter.
+
+<p>The reduce function to calculate the total age of all girls is:
+
+<pre>
+function(keys, values) {
+ return sum(values);
+}
+</pre>
+
+<p>Note that, instead of the two earlier versions, we use CouchDB’s predefined <code>sum()</code> function. It does the same thing as the other two, but it is such a common piece of code that CouchDB has it included.
+
+<p>The result for our reduce view now looks like this:
+
+<pre>
+{"rows":[
+{"key":null,"value":15}
+]}
+</pre>
+
+<p>The total sum of all <code>age</code> fields in all our documents is <code>15</code>. Just what we wanted. The <code>key</code> member of the result object is <code>null</code>, as we can’t know anymore which documents took part in the creation of the reduced result. We’ll cover more advanced reduce cases later on.
+
+<p>As a rule of thumb, the reduce function should reduce a single scalar value. That is, an integer; a string; or a small, fixed-size list or object that includes an aggregated value (or values) from the <code>values</code> argument. It should never just return <code>values</code> or similar. CouchDB will give you a warning if you try to use reduce “the wrong way”:
+
+<pre>
+{"error":"reduce_overflow_error","message":"Reduce output must shrink more rapidly: Current output: ..."}
+</pre>
+
+<h3 id="unique">Get Unique Values</h3>
+
+<p>How you would do this in SQL:
+
+<pre>
+SELECT DISTINCT <em>field</em> FROM <em>table</em>
+</pre>
+
+<p>Getting unique values is not as easy as adding a keyword. But a reduce view and a special query parameter give us the same result. Let’s say you want a list of tags that your users have tagged themselves with and no duplicates.
+
+<p>First, let’s look at the source documents. We punt on <code>_id</code> and <code>_rev</code> attributes here:
+
+<pre>
+{
+ "name":"Chris",
+ "tags":["mustache", "music", "couchdb"]
+}
+
+{
+ "name":"Noah",
+ "tags":["hypertext", "philosophy", "couchdb"]
+}
+
+{
+ "name":"Jan",
+ "tags":["drums", "bike", "couchdb"]
+}
+</pre>
+
+<p>Next, we need a list of all tags. A map function will do the trick:
+
+<pre>
+function(dude) {
+ if(dude.name &amp;&amp; dude.tags) {
+ dude.tags.forEach(function(tag) {
+ emit(tag, null);
+ });
+ }
+}
+</pre>
+
+<p>The result will look like this:
+
+<pre>
+{"total_rows":9,"offset":0,"rows":[
+{"id":"3525ab874bc4965fa3cda7c549e92d30","key":"bike","value":null},
+{"id":"3525ab874bc4965fa3cda7c549e92d30","key":"couchdb","value":null},
+{"id":"53f82b1f0ff49a08ac79a9dff41d7860","key":"couchdb","value":null},
+{"id":"da5ea89448a4506925823f4d985aabbd","key":"couchdb","value":null},
+{"id":"3525ab874bc4965fa3cda7c549e92d30","key":"drums","value":null},
+{"id":"53f82b1f0ff49a08ac79a9dff41d7860","key":"hypertext","value":null},
+{"id":"da5ea89448a4506925823f4d985aabbd","key":"music","value":null},
+{"id":"da5ea89448a4506925823f4d985aabbd","key":"mustache","value":null},
+{"id":"53f82b1f0ff49a08ac79a9dff41d7860","key":"philosophy","value":null}
+]}
+</pre>
+
+<p>As promised, these are all the tags, including duplicates. Since each document gets run through the map function in isolation, it cannot know if the same key has been emitted already. At this stage, we need to live with that. To achieve uniqueness, we need a reduce:
+
+<pre>
+function(keys, values) {
+ return true;
+}
+</pre>
+
+<p>This reduce doesn’t do anything, but it allows us to specify a special query parameter when querying the view:
+
+<pre>
+/dudes/_design/dude-data/_view/tags?group=true
+</pre>
+
+<p>CouchDB replies:
+
+<pre>
+{"rows":[
+{"key":"bike","value":true},
+{"key":"couchdb","value":true},
+{"key":"drums","value":true},
+{"key":"hypertext","value":true},
+{"key":"music","value":true},
+{"key":"mustache","value":true},
+{"key":"philosophy","value":true}
+]}
+</pre>
+
+<p>In this case, we can ignore the value part because it is always true, but the result includes a list of all our tags and no duplicates!
+
+<p>With a small change we can put the reduce to good use, too. Let’s see how many of the non-unique tags are there for each tag. To calculate the tag frequency, we just use the summing up we already learned about. In the map function, we emit a <code>1</code> instead of <code>null</code>:
+
+<pre>
+function(dude) {
+ if(dude.name &amp;&amp; dude.tags) {
+ dude.tags.forEach(function(tag) {
+ emit(tag, 1);
+ });
+ }
+}
+</pre>
+
+<p>In the reduce function, we return the sum of all values:
+
+<pre>
+function(keys, values) {
+ return sum(values);
+}
+</pre>
+
+<p>Now, if we query the view with the <code>?group=true</code> parameter, we get back the count for each tag:
+
+<pre>
+{"rows":[
+{"key":"bike","value":1},
+{"key":"couchdb","value":3},
+{"key":"drums","value":1},
+{"key":"hypertext","value":1},
+{"key":"music","value":1},
+{"key":"mustache","value":1},
+{"key":"philosophy","value":1}
+]}
+</pre>
+
+<h3 id="uniqueness">Enforcing Uniqueness</h3>
+
+<p>How you would do this in SQL:
+
+<pre>
+UNIQUE KEY(<em>column</em>)
+</pre>
+
+<p>Use case: your applications require that a certain value exists only once in a database.
+
+<p>This is an easy one: within a CouchDB database, each document must have a unique <code>_id</code> field. If you require unique values in a database, just assign them to a document’s <code>_id</code> field and CouchDB will enforce uniqueness for you.
+
+<p>There’s one caveat, though: in the distributed case, when you are running more than one CouchDB node that accepts write requests, uniqueness can be guaranteed only per node or outside of CouchDB. CouchDB will allow two identical IDs to be written to two different nodes. On replication, CouchDB will detect a conflict and flag the document accordingly.
+
+<script src="../script.js"></script>
143 draft/design.html
View
@@ -0,0 +1,143 @@
+<title>Design Documents</title>
+
+<meta charset="utf-8">
+
+<link rel="stylesheet" href="../style.css">
+
+<link rel="prev" href="api.html">
+
+<link rel="next" href="views.html">
+
+<h2 id="design">Design Documents</h2>
+
+<p><em>Design documents</em> are a special type of CouchDB document that contains application code. Because it runs inside a database, the application API is highly structured. We’ve seen JavaScript views and other functions in the previous chapters. In this section, we’ll take a look at the function APIs, and talk about how functions in a design document are related within applications.
+
+<p>This part (<a href="index.html#part/ii">Part II, “Developing with CouchDB”</a>, Chapters <a href="design.html">Chapter 5, Design Documents</a> through <a href="transforming.html">Chapter 9, Transforming Views with List Functions</a>) lays the foundation for <a href="index.html#part/iii">Part III, “Example Application”</a>, where we take what we’ve learned and build a small blog application to further develop an understanding of how CouchDB applications are built. The application is called Sofa, and on a few occasions we discuss it this part. If you are unclear on what we are referring to, do not worry, we’ll get to it in <a href="index.html#part/iii">Part III, “Example Application”</a>.
+
+<h3 id="modeling">Document Modeling</h3>
+
+<p>In our experience, there are two main kinds of documents. The first kind is like something a word processor would save, or a user profile. With that sort of data, you want to denormalize as much as you possibly can. Basically, you want to be able to load the document in one request and get something that makes sense enough to display.
+
+<p>A technique exists for creating “virtual” documents by using views to collate data together. You could use this to store each attribute of your user profiles in a different document, but I wouldn’t recommend it. Virtual documents are useful in cases where the presented view will be created by merging the work of different authors; for instance, the reference example, a blog post, and its comments in one query. A blog post titled “<a href="http://www.cmlenz.net/archives/2007/10/couchdb-joins">CouchDB Joins</a>,” by Christopher Lenz, covers this in more detail.
+
+<p>This virtual document idea takes us to the other kind of document—the event log. Use this in cases where you don’t trust user input or where you need to trigger an asynchronous job. This records the user action as an event, so only minimal validation needs to occur at save time. It’s when you load the document for further work that you’d check for complex relational-style constraints.
+
+<p>You can treat documents as state machines, with a combination of user input and background processing managing document state. You’d use a view by state to pull out the relevant document—changing its state would move it in the view.
+
+<p>This approach is also useful for logging—combined with the <code>batch=ok</code> performance hint, CouchDB should make a fine log store, and reduce views are ideal for finding things like average response time or highly active users.
+
+<h3 id="server">The Query Server</h3>
+
+<p>CouchDB’s default query server (the software package that executes design document functions) is written in JavaScript, but there are views servers available for nearly any language you can imagine. Implementing a new language is a matter of handling a few JSON commands from a simple line-based program.
+
+<p>In this section, we’ll review existing functionality like MapReduce views, update validation functions, and show and list transforms. We’ll also briefly describe capabilities available on CouchDB’s roadmap, like replication filters, update handlers for parsing non-JSON input, and a rewrite handler for making application URLs more palatable. Since CouchDB is an open source project, we can’t really say when each planned feature will become available, but it’s our hope that everything described here is available by the time you read this. We’ll make it clear in the text when we’re talking about things that aren’t yet in the CouchDB trunk.
+
+<h3 id="documents">Applications Are Documents</h3>
+
+<p>CouchDB is designed to work best when there is a one-to-one correspondence between applications and design documents.
+
+<p>A <em>design document</em> is a CouchDB document with an <code>id</code> that begins with <code>_design/</code>. For instance, the example blog application, Sofa, is stored in a design document with the ID <code>_design/sofa</code> (see <a href="#figure/1">Figure 1, “Anatomy of our design document”</a>). Design documents are just like any other CouchDB document—they replicate along with the other documents in their database and track edit conflicts with the <code>rev</code> parameter.
+
+<p>As we’ve seen, design documents are normal JSON documents, denoted by the fact that their DocID is prefixed with <code>_design/</code>.
+
+<p>CouchDB looks for views and other application functions here. The static HTML pages of our application are served as attachments to the design document. Views and validations, however, aren’t stored as attachments; rather, they are directly included in the design document’s JSON body.
+
+<div class="figure" id="figure/1">
+
+<img src="design/01.png">
+
+<p class="caption">Figure 1. Anatomy of our design document
+
+</div>
+
+<p>CouchDB’s MapReduce queries are stored in the <code>views</code> field. This is how Futon displays and allows you to edit MapReduce queries. View indexes are stored on a per–design document basis, according to a fingerprint of the function’s text contents. This means that if you edit attachments, validations, or any other non-view (or language) fields on the design document, the views will not be regenerated. However, if you change a map or a reduce function, the view index will be deleted and a new index built for the new view functions.
+
+<p>CouchDB has the capability to render responses in formats other than raw JSON. The design doc fields <code>show</code> and <code>list</code> contain functions used to transform raw JSON into HTML, XML, or other Content-Types. This allows CouchDB to serve Atom feeds without any additional middleware. The <code>show</code> and <code>list</code> functions are a little like “actions” in traditional web frameworks—they run some code based on a request and render a response. However, they differ from actions in that they may not have side effects. This means that they are largely restricted to handling <code>GET</code> requests, but it also means they can be cached by HTTP proxies like Varnish.
+
+<p>Because application logic is contained in a single document, code upgrades can be accomplished with CouchDB replication. This also opens the possibility for a single database to host multiple applications. The interface a newspaper editor needs is vastly different from what a reader desires, although the data is largely the same. They can both be hosted by the same database, in different design documents.
+
+<p>A CouchDB database can contain many design documents. Example design DocIDs are:
+
+<pre>
+_design/calendar
+_design/contacts
+_design/blog
+_design/admin
+</pre>
+
+<p>In the full CouchDB URL structure, you’d be able to <code>GET</code> the design document JSON at URLs like:
+
+<pre>
+http://localhost:5984/mydb/_design/calendar
+http://127.0.0.1:5984/mydb/_design/contacts
+http://127.0.0.1:5984/mydb/_design/blog
+http://127.0.0.1:5984/mydb/_design/admin
+</pre>
+
+<p>We show this to note that design documents have a special case, as they are the only documents whose URLs can be used with a literal slash. We’ve done this because nobody likes to see <code>%2F</code> in their browser’s location bar. In all other cases, a slash in a DocID must be escaped when used in a URL. For instance, the DocID <code>movies/jaws</code> would appear in the URL like this: <code>http://127.0.0.1:5984/mydb/movies%2Fjaws</code>.
+
+<p>We’ll build the first iteration of the example application without using <code>show</code> or <code>list</code>, because writing Ajax queries against the JSON API is a better way to teach CouchDB as a database. The APIs we explore in the first iteration are the same APIs you’d use to analyze log data, archive assets, or manage persistent queues.
+
+<p>In the second iteration, we’ll upgrade our example blog so that it can function with client-side JavaScript turned off. For now, sticking to Ajax queries gives more transparency into how CouchDB’s JSON/HTTP API works. JSON is a subset of JavaScript, so working with it in JavaScript keeps the impedance mismatch low, while the browser’s <em>XMLHttpRequest (XHR)</em> object handles the HTTP details for us.
+
+<p>CouchDB uses the <code>validate_doc_update</code> function to prevent invalid or unauthorized document updates from proceeding. We use it in the example application to ensure that blog posts can be authored only by logged-in users. CouchDB’s validation functions also can’t have any side effects, and they have the opportunity to block not only end user document saves, but also replicated documents from other nodes. We’ll talk about validation in depth in <a href="index.html#part/iii">Part III, “Example Application”</a>.
+
+<p>The raw images, JavaScript, CSS, and HTML assets needed by Sofa are stored in the <code>_attachments</code> field, which is interesting in that by default it shows only the stubs, rather than the full content of the files. Attachments are available on all CouchDB documents, not just design documents, so asset management applications have as much flexibility as they could need. If a set of resources is required for your application to run, they should be attached to the design document. This means that a new user can easily bootstrap your application on an empty database.
+
+<p>The other fields in the design document shown in <a href="#figure/1">Figure 1, “Anatomy of our design document”</a> (and in the design documents we’ll be using) are used by CouchApp’s upload process (see <a href="standalone.html">Chapter 10, Standalone Applications</a> for more information on CouchApp). The <code>signatures</code> field allows us to avoid updating attachments that have not changed between the disk and the database. It does this by comparing file content hashes. The <code>lib</code> field is used to hold additional JavaScript code and JSON data to be inserted at deploy time into view, show, and validation functions. We’ll explain CouchApp in the next chapter.
+
+<h3 id="basic">A Basic Design Document</h3>
+
+<p>In the next section we’ll get into advanced techniques for working with design documents, but before we finish here, let’s look at a very basic design document. All we’ll do is define a single view, but it should be enough to show you how design documents fit into the larger system.
+
+<p>First, add the following text (or something like it) to a text file called <code>mydesign.json</code> using your editor:
+
+<pre>
+{
+ "_id" : "_design/example",
+ "views" : {
+ "foo" : {
+ "map" : "function(doc){ emit(doc._id, doc._rev)}"
+ }
+ }
+}
+</pre>
+
+<p>Now use <code>curl</code> to <code>PUT</code> the file to CouchDB (we’ll create a database first for good measure):
+
+<pre>
+curl -X PUT http://127.0.0.1:5984/basic
+curl -X PUT http://127.0.0.1:5984/basic/_design/example -d @mydesign.json
+</pre>
+
+<p>From the second request, you should see a response like:
+
+<pre>
+{"ok":true,"id":"_design/example","rev":"1-230141dfa7e07c3dbfef0789bf11773a"}
+</pre>
+
+<p>Now we can query the view we’ve defined, but before we do that, we should add a few documents to the database so we have something to view. Running the following command a few times will add empty documents:
+
+<pre>
+curl -X POST http://127.0.0.1:5984/basic -d '{}'
+</pre>
+
+<p>Now to query the view:
+
+<pre>
+curl http://127.0.0.1:5984/basic/_design/example/_view/foo
+</pre>
+
+<p>This should give you a list of all the documents in the database (except the design document). You’ve created and used your first design document!
+
+<h3 id="future">Looking to the Future</h3>
+
+<p>There are other design document functions that are being introduced at the time of this writing, including <code>_update</code> and <code>_filter</code> that we aren’t covering in depth here. Filter functions are covered in <a href="notifications.html">Chapter 20, Change Notifications</a>. Imagine a web service that <code>POSTs</code> an XML blob at a URL of your choosing when particular events occur. PayPal’s instant payment notification is one of these. With an <code>_update</code> handler, you can <code>POST</code> these directly in CouchDB and it can parse the XML into a JSON document and save it. The same goes for CSV, multi-part form, or any other format.
+
+<p>The bigger picture we’re working on is like an app server, but different in one crucial regard: rather than let the developer do whatever he wants (loop a list of DocIDs and make queries, make queries based on the results of other queries, etc.), we’re defining “safe” transformations, such as view, show, list, and update. By safe, we mean that they have well-known performance characteristics and otherwise fit into CouchDB’s architecture in a streamlined way.
+
+<p>The goal here is to provide a way to build standalone apps that can also be easily indexed by search engines and used via screen readers. Hence, the push for plain old HTML. You can pretty much rely on JavaScript getting executed (except when you can’t). Having HTML resources means CouchDB is suitable for public-facing web apps.
+
+<p>On the horizon are a rewrite handler and a database event handler, as they seem to flesh out the application capabilities nicely. A rewrite handler would allow your application to present its own URL space, which would make integration into existing systems a bit easier. An event handler would allow you to run asynchronous processes when the database changes, so that, for instance, a document update can trigger a workflow, multi-document validation, or message queue.
+
+<script src="../script.js"></script>
BIN  draft/design/01.png
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
368 draft/documents.html
View
@@ -0,0 +1,368 @@
+<title>Storing Documents</title>
+
+<meta charset="utf-8">
+
+<link rel="stylesheet" href="../style.css">
+
+<link rel="prev" href="managing.html">
+
+<link rel="next" href="formats.html">
+
+<h2 id="documents">Storing Documents</h2>
+
+<p><em>Documents</em> are CouchDB’s central data structure. To best understand and use CouchDB, you need to <em>think in documents</em>. This chapter walks you though the lifecycle of designing and saving a document. We’ll follow up by reading documents and aggregating and querying them with views. In the next section, you’ll see how CouchDB can also transform documents into other formats.
+
+<p>Documents are self-contained units of data. You might have heard the term <em>record</em> to describe something similar. Your data is usually made up of small native types such as integers and strings. Documents are the first level of abstraction over these native types. They provide some structure and logically group the primitive data. The height of a person might be encoded as an integer (<code>176</code>), but this integer is usually part of a larger structure that contains a label (<code>"height": 176</code>) and related data (<code>{"name":"Chris", "height": 176}</code>).
+
+<p>How many data items you put into your documents depends on your application and a bit on how you want to use views (later), but generally, a document roughly corresponds to an object instance in your programming language. Are you running an online shop? You will have <em>items</em> and <em>sales</em> and <em>comments</em> for your items. They all make good candidates for objects and, subsequently, documents.
+
+<p>Documents differ subtly from garden-variety objects in that they usually have authors and CRUD operations (create, read, update, delete). Document-based software (like the word processors and spreadsheets of yore) builds its storage model around saving documents so that authors get back what they created. Similarly, in a CouchDB application you may find yourself giving greater leeway to the presentation layer. If, instead of adding timestamps to your data in a controller, you allow the user to control them, you get draft status and the ability to publish articles in the future for free (by viewing published documents using an <code>endkey</code> of <em>now</em>).
+
+<p>Validation functions are available so that you don’t have to worry about bad data causing errors in your system. Often in document-based software, the client application edits and manipulates the data, saving it back. As long as you give the user the document she asked you to save, she’ll be happy.
+
+<p>Say your users can comment on the item (“lovely book”); you have the option to store the comments as an array, on the item document. This makes it trivial to find the item’s comments, but, as they say, “it doesn’t scale.” A popular item could have tens of comments, or even hundreds or more.
+
+<p>Instead of storing a list on the item document, in this case it may be better to model comments into a collection of documents. There are patterns for accessing collections, which CouchDB makes easy. You likely want to show only 10 or 20 at a time and provide <em>previous</em> and <em>next</em> links. By handling comments as individual entities, you can group them with views. A group could be the entire collection or slices of 10 or 20, sorted by the item they apply to so that it’s easy to grab the set you need.
+
+<p>A rule of thumb: break up into documents everything that you will be handling separately in your application. Items are single, and comments are single, but you don’t need to break them into smaller pieces. Views are a convenient way to group your documents in meaningful ways.
+
+<p>Let’s go through building our example application to show you in practice how to work with documents.
+
+<h3 id="format">JSON Document Format</h3>
+
+<p>The first step in designing any application (once you know what the program is for and have the user interaction nailed down) is deciding on the format it will use to represent and store data. Our example blog is written in JavaScript. A few lines back we said documents roughly represent your data objects. In this case, there is a an exact correspondence. CouchDB borrowed the JSON data format from JavaScript; this allows us to use documents directly as native objects when programming. This is really convenient and leads to fewer problems down the road (if you ever worked with an ORM system, you might know what we are hinting at).
+
+<p>Let’s draft a JSON format for blog posts. We know we’ll need each post to have an author, a title, and a body. We know we’d like to use document IDs to find documents so that URLs are search engine–friendly, and we’d also like to list them by creation date.
+
+<p>It should be pretty straightforward to see how JSON works. Curly braces (<code>{}</code>) wrap objects, and objects are key/value lists. Keys are strings that are wrapped in double quotes (<code>""</code>). Finally, a value is a string, an integer, an object, or an array (<code>[]</code>). Keys and values are separated by a colon (<code>:</code>), and multiple keys and values by comma (<code>,</code>). That’s it. For a complete description of the JSON format, see <a href="json.html">Appendix E, JSON Primer</a>.
+
+<p><a href="#figure/1">Figure 1, “The JSON post format”</a> shows a document that meets our requirements. The cool thing is we just made it up on the spot. We didn’t go and define a schema, and we didn’t define how things should look. We just created a document with whatever we needed. Now, requirements for objects change all the time during the development of an application. Coming up with a different document that meets new, evolved needs is just as easy.
+
+<div class="figure" id="figure/1">
+
+<img src="documents/01.png">
+
+<p class="caption">Figure 1. The JSON post format
+
+</div>
+
+<blockquote>
+
+<p>Do I really look like a guy with a plan? You know what I am? I’m a dog chasing cars. I wouldn’t know what to do with one if I caught it. You know, I just do things. The mob has plans, the cops have plans, Gordon’s got plans. You know, they’re schemers. Schemers trying to control their little worlds. I’m not a schemer. I try to show the schemers how pathetic their attempts to control things really are.
+
+<p class="attribution">&mdash;The Joker, <em>The Dark Knight</em>
+
+</blockquote>
+
+<p>Let’s examine the document in a little more detail. The first two members (<code>_id</code> and <code>_rev</code>) are for CouchDB’s housekeeping and act as identification for a particular <em>instance</em> of a document. <code>_id</code> is easy: if I store something in CouchDB, it creates the <code>_id</code> and returns it to me. I can use the <code>_id</code> to build the URL where I can get my something back.
+
+<div class="aside note">
+
+<p>Your document’s <code>_id</code> defines the URL the document can be found under. Say you have a database <code>movies</code>. All documents can be found somewhere under the URL <code>/movies</code>, but where exactly?
+
+<p>If you store a document with the <code>_id</code> <code>Jabberwocky</code> (<code>{"_id":"Jabberwocky"}</code>) into your <code>movies</code> database, it will be available under the URL <code>/movies/Jabberwocky</code>. So if you send a GET request to <code>/movies/Jabberwocky</code>, you will get back the JSON that makes up your document (<code>{"_id":"Jabberwocky"}</code>).
+
+</div>
+
+<p>The <code>_rev</code> (or <em>revision ID</em>) describes a version of a document. Each change creates a new document version (that again is self-contained) and updates the <code>_rev</code>. This becomes useful because, when saving a document, you must provide an up-to-date <code>_rev</code> so that CouchDB knows you’ve been working against the latest document version.
+
+<p>We touched on this in <a href="consistency.html">Chapter 2, Eventual Consistency</a>. The revision ID acts as a gatekeeper for writes to a document in CouchDB’s MVCC system. A document is a shared resource; many clients can read and write them at the same time. To make sure two writing clients don’t step on each other’s feet, each client must provide what it believes is the latest revision ID of a document along with the proposed changes. If the on-disk revision ID matches the provided <code>_rev</code>, CouchDB will accept the change. If it doesn’t, the update will be rejected. The client should read the latest version, integrate the changes, and try saving again.
+
+<p>This mechanism ensures two things: a client can only overwrite a version it knows, and it can’t trip over changes made by other clients. This works without CouchDB having to manage explicit <em>locks</em> on any document. This ensures that no client has to wait for another client to complete any work. Updates are serialized, so CouchDB will never attempt to write documents faster than your disk can spin, and it also means that two mutually conflicting writes can’t be written at the same time.
+
+<h3 id="data">Beyond _id and _rev: Your Document Data</h3>
+
+<p>Now that you thoroughly understand the role of <code>_id</code> and <code>_rev</code> on a document, let’s look at everything else we’re storing.
+
+<pre>
+{
+ "_id":"Hello-Sofa",
+ "_rev":"2-2143609722",
+ "type":"post",
+</pre>
+
+<p>The first thing is the type of the document. Note that this is an application-level parameter, not anything particular to CouchDB. The type is just an arbitrarily named key/value pair as far as CouchDB is concerned. For us, as we’re adding blog posts to Sofa, it has a little deeper meaning. Sofa uses the <code>type</code> field to determine which validations to apply. It can then rely on documents of that type being valid in the views and the user interface. This removes the need to check for every field and nested JSON value before using it. This is purely by convention, and you can make up your own or infer the type of a document by its structure (“has an array with three elements”—a.k.a. <em>duck typing</em>). We just thought this was easy to follow and hope you agree.
+
+<pre>
+ "author":"jchris",
+ "title":"Hello Sofa",
+</pre>
+
+<p>The <code>author</code> and <code>title</code> fields are set when the post is created. The <code>title</code> field can be changed, but the <code>author</code> field is locked by the validation function for security. Only the author may edit the post.
+
+<pre>
+ "tags":["example","blog post","json"],
+</pre>
+
+<p>Sofa’s tag system just stores them as an array on the document. This kind of denormalization is a particularly good fit for CouchDB.
+
+<pre>
+ "format":"markdown",
+ "body":"some markdown text",
+ "html":"&lt;p&gt;the html text&lt;/p&gt;",
+</pre>
+
+<p>Blog posts are composed in the <a href="http://daringfireball.net/projects/markdown/">Markdown HTML format</a> to make them easy to author. The Markdown format as typed by the user is stored in the <code>body</code> field. Before the blog post is saved, Sofa converts it to HTML in the client’s browser. There is an interface for previewing the Markdown conversion, so you can be sure it will display as you like.
+
+<pre>
+ "created_at":"2009/05/25 06:10:40 +0000"
+}
+</pre>
+
+<p>The <code>created_at</code> field is used to order blog posts in the Atom feed and on the HTML index page.
+
+<h3 id="edit">The Edit Page</h3>
+
+<p>The first page we need to build in order to get one of these blog entries into our post is the interface for creating and editing posts.
+
+<p>Editing is more complex than just rendering posts for visitors to read, but that means once you’ve read this chapter, you’ll have seen most of the techniques we touch on in the other chapters.
+
+<p>The first thing to look at is the <em>show function</em> used to render the HTML page. If you haven’t already, read <a href="show.html">Chapter 8, Show Functions</a> to learn about the details of the API. We’ll just look at this code in the context of Sofa, so you can see how it all fits together.
+
+<pre>
+function(doc, req) {
+ // !json templates.edit
+ // !json blog
+ // !code vendor/couchapp/path.js
+ // !code vendor/couchapp/template.js
+</pre>
+
+<p>Sofa’s edit page show function is very straightforward. In the previous section, we showed the important templates and libraries we’ll use. The important line is the <code>!json</code> macro, which loads the <code>edit.html</code> template from the <code>templates</code> directory. These macros are run by CouchApp, as Sofa is being deployed to CouchDB. For more information about the macros, see <a href="formats.html">Chapter 13, Showing Documents in Custom Formats</a>.
+
+<pre>
+ // we only show html
+ return template(templates.edit, {
+ doc : doc,
+ docid : toJSON((doc &amp;&amp; doc._id) || null),
+ blog : blog,
+ assets : assetPath(),
+ index : listPath('index','recent-posts',{descending:true,limit:8})
+ });
+}
+</pre>
+
+<p>The rest of the function is simple. We’re just rendering the HTML template with data culled from the document. In the case where the document does not yet exist, we make sure to set the <code>docid</code> to <code>null</code>. This allows us to use the same template both for creating new blog posts as well as editing existing ones.
+
+<h4 id="scaffold">The HTML Scaffold</h4>
+
+<p>The only missing piece of this puzzle is the HTML that it takes to save a document like this.
+
+<p>In your browser, visit <code>http://127.0.0.1:5984/blog/_design/sofa/_show/edit</code> and, using your text editor, open the source file <code>templates/edit.html</code> (or view source in your browser). Everything is ready to go; all we have to do is wire up CouchDB using in-page JavaScript. See <a href="#figure/2">Figure 2, “HTML listing for edit.html”</a>.
+
+<p>Just like any web application, the important part of the HTML is the form for accepting edits. The edit form captures a few basic data items: the post title, the body (in Markdown format), and any tags the author would like to apply.
+
+<pre>
+&lt;!-- form to create a Post --&gt;
+&lt;form id="new-post" action="new.html" method="post"&gt;
+ &lt;h1&gt;Create a new post&lt;/h1&gt;
+ &lt;p&gt;&lt;label&gt;Title&lt;/label&gt;
+ &lt;input type="text" size="50" name="title"&gt;&lt;/p&gt;
+ &lt;p&gt;&lt;label for="body"&gt;Body&lt;/label&gt;
+ &lt;textarea name="body" rows="28" cols="80"&gt;
+ &lt;/textarea&gt;&lt;/p&gt;
+ &lt;p&gt;&lt;input id="preview" type="button" value="Preview"/&gt;
+ &lt;input type="submit" value="Save &amp;rarr;"/&gt;&lt;/p&gt;
+&lt;/form&gt;
+</pre>
+
+<p>We start with just a raw HTML document, containing a normal HTML form. We use JavaScript to convert user input into a JSON document and save it to CouchDB. In the spirit of focusing on CouchDB, we won’t dwell on the JavaScript here. It’s a combination of Sofa-specific application code, CouchApp’s JavaScript helpers, and jQuery for interface elements. The basic story is that it watches for the user to click “Save,” and then applies some callbacks to the document before sending it to CouchDB.
+
+<div class="figure" id="figure/2">
+
+<img src="documents/02.png">
+
+<p class="caption">Figure 2. HTML listing for edit.html
+
+</div>
+
+<h3 id="saving">Saving a Document</h3>
+
+<p>The JavaScript that drives blog post creation and editing centers around the HTML form from <a href="#figure/2">Figure 2, “HTML listing for edit.html”</a>. The CouchApp jQuery plug-in provides some abstraction, so we don’t have to concern ourselves with the details of how the form is converted to a JSON document when the user hits the submit button. <code>$.CouchApp</code> also ensures that the user is logged in and makes her information available to the application. See <a href="#figure/3">Figure 3, “JavaScript callbacks for edit.html”</a>.
+
+<pre>
+$.CouchApp(function(app) {
+ app.loggedInNow(function(login) {
+</pre>
+
+<p>The first thing we do is ask the CouchApp library to make sure the user is logged in. Assuming the answer is yes, we’ll proceed to set up the page as an editor. This means we apply a JavaScript event handler to the form and specify callbacks we’d like to run on the document, both when it is loaded and when it saved.
+
+<div class="figure" id="figure/3">
+
+<img src="documents/03.png">
+
+<p class="caption">Figure 3. JavaScript callbacks for edit.html
+
+</div>
+
+<pre>
+ // w00t, we're logged in (according to the cookie)
+ $("#header").prepend('&lt;span id="login"&gt;'+login+'&lt;/span&gt;');
+ // setup CouchApp document/form system, adding app-specific callbacks
+ var B = new Blog(app);
+</pre>
+
+<p>Now that we know the user is logged in, we can render his username at the top of the page. The variable <code>B</code> is just a shortcut to some of the Sofa-specific blog rendering code. It contains methods for converting blog post bodies from Markdown to HTML, as well as a few other odds and ends. We pulled these functions into <code>blog.js</code> so we could keep them out of the way of main code.
+
+<pre>
+ var postForm = app.docForm("form#new-post", {
+ id : &lt;%= docid %&gt;,
+ fields : ["title", "body", "tags"],
+ template : {
+ type : "post",
+ format : "markdown",
+ author : login
+ },
+</pre>
+
+<p>CouchApp’s <code>app.docForm()</code> helper is a function to set up and maintain a correspondence between a CouchDB document and an HTML form. Let’s look at the first three arguments passed to it by Sofa. The <code>id</code> argument tells <code>docForm()</code> where to save the document. This can be null in the case of a new document. We set <code>fields</code> to an array of form elements that will correspond directly to JSON fields in the CouchDB document. Finally, the <code>template</code> argument is given a JavaScript object that will be used as the starting point, in the case of a new document. In this case, we ensure that the document has a type equal to “post,” and that the default format is Markdown. We also set the author to be the login name of the current user.
+
+<pre>
+ onLoad : function(doc) {
+ if (doc._id) {
+ B.editing(doc._id);
+ $('h1').html('Editing &lt;a href="../post/'+doc._id+'"&gt;'+doc._id+'&lt;/a&gt;');
+ $('#preview').before('&lt;input type="button" id="delete"
+ value="Delete Post"/&gt; ');
+ $("#delete").click(function() {
+ postForm.deleteDoc({
+ success: function(resp) {
+ $("h1").text("Deleted "+resp.id);
+ $('form#new-post input').attr('disabled', true);
+ }
+ });
+ return false;
+ });
+ }
+ $('label[for=body]').append(' &lt;em&gt;with '+(doc.format||'html')+'&lt;/em&gt;');
+</pre>
+
+<p>The <code>onLoad</code> callback is run when the document is loaded from CouchDB. It is useful for decorating the document before passing it to the form, or for setting up other user interface elements. In this case, we check to see if the document already has an ID. If it does, that means it’s been saved, so we create a button for deleting it and set up the callback to the delete function. It may look like a lot of code, but it’s pretty standard for Ajax applications. If there is one criticism to make of this section, it’s that the logic for creating the delete button could be moved to the <code>blog.js</code> file so we can keep more user-interface details out of the main flow.
+
+<pre>
+ },
+ beforeSave : function(doc) {
+ doc.html = B.formatBody(doc.body, doc.format);
+ if (!doc.created_at) {
+ doc.created_at = new Date();
+ }
+ if (!doc.slug) {
+ doc.slug = app.slugifyString(doc.title);
+ doc._id = doc.slug;
+ }
+ if(doc.tags) {
+ doc.tags = doc.tags.split(",");
+ for(var idx in doc.tags) {
+ doc.tags[idx] = $.trim(doc.tags[idx]);
+ }
+ }
+ },
+</pre>
+
+<p>The <code>beforeSave()</code> callback to <code>docForm</code> is run after the user clicks the submit button. In Sofa’s case, it manages setting the blog post’s timestamp, transforming the title into an acceptable document ID (for prettier URLs), and processing the document tags from a string into an array. It also runs the Markdown-to-HTML conversion in the browser so that once the document is saved, the rest of the application has direct access to the HTML.
+
+<pre>
+ success : function(resp) {
+ $("#saved").text("Saved _rev: "+resp.rev).fadeIn(500).fadeOut(3000);
+ B.editing(resp.id);
+ }
+ });
+</pre>
+
+<p>The last callback we use in Sofa is the <code>success</code> callback. It is fired when the document is successfully saved. In our case, we use it to flash a message to the user that lets her know she’s succeeded, as well as to add a link to the blog post so that when you create a blog post for the first time, you can click through to see its permalink page.
+
+<p>That’s it for the <code>docForm()</code> callbacks.
+
+<pre>
+ $("#preview").click(function() {
+ var doc = postForm.localDoc();
+ var html = B.formatBody(doc.body, doc.format);
+ $('#show-preview').html(html);
+ // scroll down
+ $('body').scrollTo('#show-preview', {duration: 500});
+ });
+</pre>
+
+<p>Sofa has a function to preview blog posts before saving them. Since this doesn’t affect how the document is saved, the code that watches for events from the “preview” button is not applied within the <code>docForm()</code> callbacks.
+
+<pre>
+ }, function() {
+ app.go('&lt;%= assets %&gt;/account.html#'+document.location);
+ });
+});
+</pre>
+
+<p>The last bit of code here is triggered when the user is not logged in. All it does is redirect him to the account page so that he can log in and try editing again.
+
+<h4 id="validation">Validation</h4>
+