WriteOnceBucketIndexingBug
RIAK-1937 encapsulates a bug in Riak Search (Yokozuna) whereby Yokozuna indexing is not happening properly with Write Once buckets. Puts occur unusually quickly when Write Once buckets are configured with a search index, and data put into these buckets is not immediately available for query. If AAE is turned on, then data will eventually get indexed in Solr, but via the AAE mechanism, which is less than optimal and can cause nodes to become unresponsive or crash.
This issue was initially reported in https://github.com/basho/yokozuna/issues/512
Write Once buckets were added in Riak 2.1 to provide a fast way to write data to a Riak back-end, effectively by circumventing the Put FSM and sending write requests directly to the Vnodes in the pref-list for the entry. Unfortunately, circumventing the Put FSM also circumvented indexing, which is a bug.
The fix requires a change to Yokozuna and to Riak K/V.
In Yokozuna, we the function yz_kv:index_binary/5, which takes a Bucket, Key, and Riak Object in binary form, as well as the same parameters as those in yz_kv:index/3. The index_binary function will decode the Riak Object (requiring a Bucket and Key) using riak_object:from_binary/3, and otherwise follows the same logic as yz_kv:index/3.
The Yokozuna changes can be found at https://github.com/basho/yokozuna/pull/529/files
Riak/KV is modified to call yz_kv:index_binary/5 in the VNode during Write Once puts.
The Riak/KV changes can be found at https://github.com/basho/riak_kv/pull/1159/files
The yz_pb system test has been modified to verify that data written to write once buckets can be queried after being written. Without the changes described above, this part of the test fails.
Basho Bench has been run against:
- 5 node cluster
- CentOS6 Virtual Machines (VMWare Fusion 7.1.1, over MacOS 10.10.3)
- Apple Macbook Pro (Early 2011) 2.0 Ghz i7; 16GB RAM; 500GB OWC Mercury EXTREME Pro 6G SSD
Basho Bench driver run from separate machine, using load-fruit.config as described in https://github.com/basho/yokozuna/blob/develop/docs/BENCHMARKING.md
Baseline measurements were run against the Riak 2.1 branch (as of 8/5/15), starting with an empty Riak database, and using the load-fruit configuration described above. This test ran on 2015.08.05 from 16:11-16:58 EDT.
The throughput and latency summary is illustrated in the following charts:
The Riak cluster was stopped, data in the cluster was cleared out, and the riak_kv and yokozuna applications were modified with the branch versions containing the bug fixes described above. This test ran on 2015.08.05 from 17:21-19:01 EDT.
The throughput and latency summary with these changes is illustrated in the following charts:
I had Collectd running on these nodes, and here are some measurements taken on node 1. For these graphs, please ignore the measurements taken at approximately Wed midnight and Wed noon; pay attention only to the measurements taken between 4PM and 7PM. Note that the baseline measurements were taken from roughly 4-5PM, and the measurements with the patch in place were taken from roughly 5:20-7PM.
The following graph shows the throughput for write once puts, as measured through Riak stats. Except for a noticeable drop in throughput, the baseline measurements were about 2x the patched measurements:
This is expected, because we are not doing any indexing in the baseline case. So write-once puts are clearly expected to be "fast" in this case (even though they will eventually have a devastating affect on AAE).
The following graph shows the indexing throughput, as measured through Riak.
As expected, we only hit the indexing code with the patch installed, so no stats were generated for indexing in the baseline case.
And here are the latency measurements:
Search Index Latency Histogram
Again, the only time we see any measurements for search index is when the patch is applied.
TODO GC stats