Browse files

Add notes to help suss out details and preemptively document

  • Loading branch information...
rzezeski committed Jun 22, 2012
1 parent b08856f commit 628b5027eaf7c63cbaae6f70b48cdef1f8bf6260
Showing with 33 additions and 0 deletions.
  1. +20 −0 notes/
  2. +13 −0 notes/
@@ -0,0 +1,20 @@
+Each incoming KV object is converted into a Solr document with 3
+1. `id` - The unique identifier for this document (since partitioning
+ is per-index the key (from bucket-key) is unique.
+2. `value` - The value of the `riak_object`.
+3. `_entropy` - The corresponding entropy data in the form of a
+ triplet `<iso8601> <partition> <id> <base64-obj-hash>`.
+Using this data a Merkle Tree can be built representing representing
+the data stored for a given partition before a given point in time.
+This Merkle Tree can then be compared with a tree built from a KV
+partition to detect entropy. You could also build trees to compare
+Solr partitions to each other to verify data is not divergent but not
+for detecting missing data since partitions only partially overlap.
@@ -0,0 +1,13 @@
+Yokozuna uses _doc-based_ partitioning, however a best effort will be
+made to code partitioning in such a way that it might be easily
+experimented with for future purposes. The big reason to use
+doc-based is because it makes entropy detection easiest. Under this
+scheme the document and it's entropy data can be atomically comitted.
+This guarentees that the entropy data stored matches the document data
+it represents. This is discussed more in the _anti-entropy.md_

0 comments on commit 628b502

Please sign in to comment.