Add read_before_last_manifest_write option to avoid sibling explosion #1011
Conversation
Add read_before_last_manifest_write option to help avoid sibling explosion for use cases involving high churn and concurrency on a fixed set of keys. When sibling explosion occurs the objects stored in Riak can become very large and severely impair the functioning of the system. The trade-off in enabling this option is latency penalty of doing an extra read before the final write of an object's manifest to Riak; however, for use cases matching the description the minor latency penalty is preferable to the system to the consequences of sibling explosion. This option is not needed for Riak CS installations using a Riak version that is >= 2.0.0 with DVVs enabled.
|
Testing with this applied reduces sibling generation on moderate concurrency write loads significantly -- I went from sustaining 8-12 writers across a test cluster 4 up to 24 without unbounded upward sibling growth. GC seems to have a nontrivial impact on sibling counts under these conditions as well, although I'm not certain there's a great strategy available for addressing that. |
|
Are there any objections to amending the branch to default to |
I am not opposed to this. I'll push a change to that effect shortly. |
|
FYI, the CI failure is due to a dialyzer bug with Erlang R15B01. The test machine is still using that version and I will work with Paul to get that updated soon. |
|
Reviewed — looks good once the CI failure is resolved. |
|
@borshop merge |
…-write Add read_before_last_manifest_write option to avoid sibling explosion Reviewed-by: kellymclaughlin
…-write Add read_before_last_manifest_write option to avoid sibling explosion Reviewed-by: kellymclaughlin
Add read_before_last_manifest_write option to help avoid sibling
explosion for use cases involving high churn and concurrency on a
fixed set of keys. When sibling explosion occurs the objects stored in
Riak can become very large and severely impair the functioning of the
system. The trade-off in enabling this option is latency penalty of
doing an extra read before the final write of an object's manifest to
Riak; however, for use cases matching the description the minor
latency penalty is preferable to the system to the consequences of
sibling explosion. This option is not needed for Riak CS installations
using a Riak version that is >= 2.0.0 with DVVs enabled.