Add documentation about transactions.

doc/07.transactions.rst

@@ -0,0 +1,294 @@
+.. _transactions:
+
+Transactions
+============
+
+HyperDex Warp is a superset of HyperDex that provides transactions.  For this
+chapter, you'll need to install the hyperdex-warp demo package available at
+http://hyperdex.org/
+
+By the end of this chapter you'll be familiar with HyperDex Warp's transactions.
+
+Setup
+-----
+
+As in the previous chapter, the first step is to deploy the cluster and connect
+a client.   First we launch and initialize the coordinator:
+
+.. sourcecode:: console
+
+   $ hyperdex coordinator -f -l 127.0.0.1 -p 1982
+
+Next, let's launch a few daemon processes to store data.  Execute the following
+commands (note that each instance binds to a different port and has a different ``/path/to/data``):
+
+.. sourcecode:: console
+
+   $ hyperdex daemon -f --listen=127.0.0.1 --listen-port=2012 \
+                        --coordinator=127.0.0.1 --coordinator-port=1982 --data=/path/to/data1
+   $ hyperdex daemon -f --listen=127.0.0.1 --listen-port=2013 \
+                        --coordinator=127.0.0.1 --coordinator-port=1982 --data=/path/to/data2
+   $ hyperdex daemon -f --listen=127.0.0.1 --listen-port=2014 \
+                        --coordinator=127.0.0.1 --coordinator-port=1982 --data=/path/to/data3
+
+
+This brings up three different daemons ready to serve in the HyperDex cluster.
+Finally, we create a space which makes use of all three systems in the cluster.
+In this example, let's create a space that may be suitable for storing virtual
+currency in a banking application:
+
+.. sourcecode:: console
+
+   >>> import hyperclient
+   >>> c = hyperclient.Client('127.0.0.1', 1982)
+   >>> c.add_space('''
+   ... space accounts
+   ... key id
+   ... attribute
+   ...    int balance
+   ... ''')
+
+Let's create some accounts with some starting balances:
+
+.. sourcecode:: pycon
+
+   >>> c.put('accounts', 'joe', {'balance': 100})
+   True
+   >>> c.put('accounts', 'brian', {'balance': 25})
+   True
+
+Using Transactions
+------------------
+
+The prototypical example of where transactions come in handy is a standard bank
+debit/credit scenario.  If Joe wanted to transfer currency to Brian, it would be
+bad for Joe and Brian if the money were to leave Joe's account and not find its
+way into Brian's.  It would be bad for the bank if money were to be created out
+of thin air by a deposit in Brian's account without a corresponding withdrawal
+from Joe's -- manipulating markets in such a fashion is reserved for those who
+are too big to fail.
+
+In the prototypical, "Hello World," example involving transactions, we transfer
+$10 from Joe to Brian:
+
+.. sourcecode:: pycon
+
+   >>> x = c.begin_transaction()
+   >>> brian = x.get('accounts', 'brian')['balance']
+   >>> joe = x.get('accounts', 'joe')['balance']
+   >>> x.put('accounts', 'brian', {'balance': brian + 10})
+   True
+   >>> x.put('accounts', 'joe', {'balance': joe - 10})
+   True
+   >>> x.commit()
+   True
+   >>> print "Brian's balance =", c.get('accounts', 'brian')['balance']
+   Brian's balance = 35
+   >>> print "Joe's balance =", c.get('accounts', 'joe')['balance']
+   Joe's balance = 90
+
+This transaction is relatively straight-forward.  There are three critical
+properties that will distinguish this transaction.  One, either both of these
+operations execute, or neither do.  It is impossible for half of a transaction
+to be lost.
+
+Two, there is no eventual consistency.  All transaction results are immediately
+visible to all future events.  There is no need to reconcile or maintain
+multiple versions.  There is no inconsistency.
+
+Finally, HyperDex Warp is fault tolerant.  A user-configurable fault tolerance
+threshold, ``f`` allows HyperDex Warp to remain available in the presence of
+concurrent failures.
+
+For a real bank application, we'd likely want to charge Joe an (excessive)
+overdraft fee if he didn't have the money to cover the transfer.  With
+transactions, implementing this case is trivial:
+
+.. sourcecode:: pycon
+
+   >>> x = c.begin_transaction()
+   >>> brian = x.get('accounts', 'brian')['balance']
+   >>> joe = x.get('accounts', 'joe')['balance']
+   >>> x.put('accounts', 'brian', {'balance': brian + 10})
+   True
+   >>> if joe < 10:
+   ...     # assess an overdraft fee on Joe
+   ...     joe -= 35
+   ...
+   >>> x.put('accounts', 'joe', {'balance': joe - 10})
+   True
+   >>> x.commit()
+   True
+   >>> print "Brian's balance =", c.get('accounts', 'brian')['balance']
+   Brian's balance = 45
+   >>> print "Joe's balance =", c.get('accounts', 'joe')['balance']
+   Joe's balance = 80
+
+Here, we've successfully transferred money from Joe to Brian and cover the case
+where the bank wishes to charge Joe a fee for not having enough money in the
+first place.
+
+The astute reader will notice that the example above is very different from
+doing the following:
+
+.. sourcecode:: pycon
+
+   >>> # an example of broken code
+   >>> c.atomic_add('accounts', 'brian', {'balance': 10})
+   True
+   >>> c.atomic_sub('accounts', 'joe', {'balance': 10})
+   True
+   >>> print "Brian's balance =", c.get('accounts', 'brian')['balance']
+   Brian's balance = 55
+   >>> print "Joe's balance =", c.get('accounts', 'joe')['balance']
+   Joe's balance = 70
+
+This example is broken in multiple ways.  First, even though each individual
+operation is atomic, the two operations are not indivisible, and will be
+interspersed with all other operations on the accounts.  Second, the client is a
+single point of failure for this transaction.  Imagine if the server executing
+these operations failed between the two atomic operations.  No matter what the
+order of operations, someone would lose money.
+
+Transactions are especially useful when multiple competing processes are
+executing transactions concurrently.  HyperDex Warp guarantees *one-copy
+serializablility*.  Any number of transactions may execute simultaneously.
+The final state of the database will always be identical to executing the
+committed transactions sequentially without concurrency.
+
+Let's illustrate HyperDex's behavior with concurrent transactions.
+In the same terminal, let's begin a new transaction to pay Joe his yearly
+5% interest payment:
+
+.. sourcecode:: pycon
+
+   >>> x = c.begin_transaction()
+   >>> balance = x.get('accounts', 'joe')['balance']
+   >>> balance = int(balance * 1.05)
+   >>> x.put('accounts', 'joe', {'balance': balance})
+   True
+
+At this point, transaction ``x`` has not committed.  Any modifications performed
+within the context of a transaction are visible only within the transaction.  No
+other clients or transactions may observe the state.  Verify this for yourself
+with:
+
+.. sourcecode:: pycon
+
+   >>> print "Joe's balance =", c.get('accounts', 'joe')['balance']
+   Joe's balance = 70
+   >>> print "Joe's balance =", x.get('accounts', 'joe')['balance']
+   Joe's balance = 73
+
+Joe's balance is still the same as before.  Now, open another window and start a
+second, concurrent transaction where Joe deposits cash at a branch location:
+
+.. sourcecode:: pycon
+
+   >>> import hyperclient
+   >>> c2 = hyperclient.Client('127.0.0.1', 1982)
+   >>> y = c2.begin_transaction()
+   >>> balance = y.get('accounts', 'joe')['balance']
+   >>> balance += 386
+   >>> y.put('accounts', 'joe', {'balance': balance})
+   True
+
+At this point, both ``x`` and ``y`` are ready to commit, but neither has
+actually altered the account balance.  In fact, there is no way for ``x`` and
+``y`` to commit that doesn't violate consistency.  If ``x`` commits before
+``y``, the ending balance will be $466, shorting Joe of the $4 in interest he
+is due.  If, however, ``y`` commits before ``x``, then the ending balance of $84
+will cause Joe's cash deposit of $386 to be lost into the ether.  In this
+situation, one transaction must abort.  If ``y`` commits first, then ``x`` will
+abort:
+
+In the second terminal, commit ``y``:
+
+.. sourcecode:: pycon
+
+   >>> y.commit()
+   True
+   >>> print "Joe's balance =", c2.get('accounts', 'joe')['balance']
+   Joe's balance = 456
+
+Trying to commit ``x`` in the first terminal will result in the transaction
+being aborted by HyperDex Warp:
+
+.. sourcecode:: pycon
+
+   >>> x.commit()
+   Traceback (most recent call last):
+   HyperClientException: HyperClient(HYPERCLIENT_ABORTED, Transaction was aborted)
+
+As expected, transaction ``x`` will not alter Joe's balance:
+
+.. sourcecode:: pycon
+
+   >>> print "Joe's balance =", c2.get('accounts', 'joe')['balance']
+   Joe's balance = 456
+
+A transaction will only abort if it was executed simultaneously with another
+transation that operates on the same data.  The typical process for handling
+aborted transactions is to repeatedly try the transaction until it succeeds:
+
+.. sourcecode:: pycon
+
+   >>> committed = False
+   >>> while not committed:
+   ...     try:
+   ...         x = c.begin_transaction()
+   ...         balance = x.get('accounts', 'joe')['balance']
+   ...         balance = int(balance * 1.05)
+   ...         x.put('accounts', 'joe', {'balance': balance})
+   ...         committed = x.commit()
+   ...     except hyperclient.HyperClientException as e:
+   ...         if e.status != hyperclient.HYPERCLIENT_ABORTED:
+   ...             raise e
+   ...
+   True
+
+Rich API
+--------
+
+HyperDex Warp Transactions expose the full set of atomic and asynchronous
+operations from the HyperDex API.  All operations performed under a transaction
+are fully transactional.  The following example shows asynchronous atomic math
+operations combined with an asynchronous commit:
+
+.. sourcecode:: pycon
+
+   >>> t = c.begin_transaction()
+   >>> d1 = t.async_atomic_add('accounts', 'brian', {'balance': 10})
+   >>> d2 = t.async_atomic_sub('accounts', 'joe', {'balance': 10})
+   >>> d1.wait()
+   True
+   >>> d2.wait()
+   True
+   >>> d3 = t.async_commit()
+   >>> d3.wait()
+   True
+
+All transactions operate on the most recent copy of the data.  When a
+transaction commits, the transaction is cosistently applied.  HyperDex Warp
+really means it when it says a transaction has committed.  There are no
+conflicting operations to reconcile later (after the system commits), and there
+is no eventual consistency.  All effects are immediate and permanent.
+
+Summary
+-------
+
+HyperDex Warp provides decentralized, distributed ACID transactions.  Committed
+transactions are one-copy serializable, making it extremely easy to reason about
+the concurrent operations.  The rich API coupled with transactional guarantees
+provide a rare combination of ease-of-use, correctness, and performance that
+other NoSQL systems cannot provide.
+
+Get your copy of `HyperDex Warp`_ today.
+
+.. _HyperDex Warp: http://hyperdex.org/Warp
+
+.. todo::
+
+   .. sourcecode:: pycon
+
+      >>> c.rm_space('accounts')

doc/index.rst

@@ -12,6 +12,7 @@ Welcome to the HyperDex Reference Manual
    04.datatypes
    05.asynchronous
    06.faults
+   07.transactions
    api/python
    api/c
    glossary