RObject#store randomly fails

[skippy]

We are sending 100ks of requests to riak and 4-10 of them randomly fail with protocol buffer errors. here is a recent example:

Beefcake::Message::WrongTypeError: Wrong type7given for (content). Expected 2

/gems/beefcake-0.3.7/lib/beefcake.rb:154 in "decode"
/gems/riak-client-1.4.2/lib/riak/client/beefcake_protobuffs_backend.rb:261 in "decode_response"
/gems/riak-client-1.4.2/lib/riak/client/beefcake_protobuffs_backend.rb:66 in "store_object"
/gems/riak-client-1.4.2/lib/riak/client.rb:551 in "block in store_object"
/gems/riak-client-1.4.2/lib/riak/client.rb:470 in "block in recover_from"
/gems/innertube-1.0.2/lib/innertube.rb:127 in "take"
/gems/riak-client-1.4.2/lib/riak/client.rb:468 in "recover_from"
/gems/riak-client-1.4.2/lib/riak/client.rb:414 in "protobuffs"
/gems/riak-client-1.4.2/lib/riak/client.rb:140 in "backend"
/gems/riak-client-1.4.2/lib/riak/client.rb:550 in "store_object"
/gems/riak-client-1.4.2/lib/riak/robject.rb:130 in "store"

recover_from doesn't retry these sorts of errors. We've never been able to recreate in any meaningful way. On staging we can recreate but there is no rhyme or reason, and the same request will succeed if retried.

Any thoughts on what is going on? In the short term we are patching the recover_from to retry these sort of Beefcake exceptions. I'm not very familiar with beefcake; if you can point me in the right direction (esp. if it is client-side) I can run from there and submit a PR.

[skippy]

There is another missing protocol buffer error; it seems the server is returning an unexpected protocolBuffer msg, which triggers the following exception:

Beefcake::Message::RequiredFieldNotSetError: Field errcode is required but nil

between this and the above, we are seeing an explosion of riak errors as our usage goes up. It is definitely becoming a stability issue. We hacked riak/client.rb to retry these issues, and that seems to help. but I wanted to pass this along as an FYI.

thanks,
adam

[bkerley]

The unexpected protobuffs message is being handled as a ruby-client 2.0 improvement. The low-probability failures are another story; we've had other users report it, but not enough information to trace down their cause :(

[bkerley]

Closing for now; I don't believe this is a client issue.