Read repairs should ignore sibling and object limits like handoffs [JIRA: RIAK-2045]

[jonmeredith]

During read repair, if the reconciled object exceeds sibling limits or object size it cannot be read repaired.

Read repair should be treated as handoff and permit the large objects to be merged and written so that the object can be reconciled.

The workaround is for a client to read with R=all which will force the return of a vclock that covers all the replicas so it can be resolved.

ObjSize = app_helper:get_env(riak_kv, max_object_size) div 2.
APL = riak_core_apl:get_apl_ann(riak_core_util:chash_key({<<"b">>,<<"k">>}), 3, riak_core_node_watcher:nodes(riak_kv)).
[rpc:call(Node, riak_kv_vnode, local_put, [Idx, riak_object:apply_updates(riak_object:update_last_modified(riak_object:set_vclock(riak_object:new(<<"b">>,<<"k">>,crypto:rand_bytes(ObjSize)), vclock:fresh(<<(Idx bsr 100):64>>, 1))))]) || {{Idx,Node},Type} <- APL].  
[rpc:call(Node, riak_kv_vnode, local_get, [Idx, {<<"b">>,<<"k">>}]) ||  {{Idx,Node},Type} <- APL]. 
{ok, C} = riak:local_client(),
C:get(<<"b">>,<<"k">>).

%% Ohnoes! Error messages
%17:08:05.344 [warning] Read large object <<"b">>/<<"k">> (26214489 bytes)
%17:08:05.345 [warning] Read large object <<"b">>/<<"k">> (26214489 bytes)
%17:08:05.364 [warning] Read large object <<"b">>/<<"k">> (26214488 bytes)
%17:08:05.800 [error] Put failure: object too large to write <<"b">>/<<"k">> 78643432 bytes
%17:08:05.820 [error] Put failure: object too large to write <<"b">>/<<"k">> 78643432 bytes
%17:08:06.015 [error] Put failure: object too large to write <<"b">>/<<"k">> 78643432 bytes

%% Here's the workaround
{ok, Obj} = C:get(<<"b">>,<<"k">>, [{r,all}]).
C:put(riak_object:update_value(Obj, crypto:rand_bytes(ObjSize))).
17:14:10.065 [warning] Read large object <<"b">>/<<"k">> (26214489 bytes)
17:14:10.089 [warning] Writing very large object (26214590 bytes) to <<"b">>/<<"k">>
17:14:10.212 [warning] Read large object <<"b">>/<<"k">> (26214488 bytes)
17:14:10.213 [warning] Read large object <<"b">>/<<"k">> (26214489 bytes)
17:14:10.238 [warning] Writing very large object (26214590 bytes) to <<"b">>/<<"k">>
17:14:10.265 [warning] Writing very large object (26214590 bytes) to <<"b">>/<<"k">>
ok

[Basho-JIRA]

#1363

_[posted via JIRA by Brian Sparrow]_

[Basho-JIRA]

basho/riak_test#1007

_[posted via JIRA by Brian Sparrow]_

[Basho-JIRA]

basho/riak_test#1009 replaces 1007

_[posted via JIRA by Brian Sparrow]_