Sporadic 'TypeError: wrong argument type Fixnum' in datastore client

[timanovsky]

This is single occurrence exception, recovered itself on retry:

Unexpected error: TypeError: wrong argument type Fixnum (expected Google::Protobuf::Map)
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:62:in `decode' Context
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:62:in `call' Context
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/rpc_desc.rb:62:in `block in unmarshal_proc' Context
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:231:in `remote_read' Context
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:309:in `request_response' Context
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:167:in `request_response' Context
/app/vendor/bundle/ruby/2.3.0/gems/grpc-1.0.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:184:in `block (3 levels) in rpc_stub_class' Context
/app/vendor/bundle/ruby/2.3.0/gems/google-cloud-datastore-0.20.1/lib/google/cloud/datastore/service.rb:83:in `block in lookup' Context
/app/vendor/bundle/ruby/2.3.0/gems/google-cloud-datastore-0.20.1/lib/google/cloud/datastore/service.rb:151:in `block in execute' Context

grpc (1.0.0)
google-cloud-datastore (0.20.1)

I'm afraid it resembles memory corruption error from the past, which manifested in similar protocol errors.

[apolcyn]

Do you have an example or type of the protobuf object that's getting decoded, e.g., the expected protobuf request type?

The last similar error happened frequently when doing lots of protobuf unmarshals of a certain protobuf message used in datastore, while turning GC on and off, but that doesn't seem to be working as easily here.

[apolcyn]

actually do you know which version of the google-protobuf gem was installed when this happened?

I think the fix of the similar-looking issue in protocolbuffers/protobuf#2004 appears in 3.3.4 or later.

[timanovsky]

@apolcyn google-protobuf (3.0.2) how do you know it was fixed in 3.3.4? As far as I remember 3.0.2 was supposed to include the fix, but I can not find a confirmation now

Update. Actually: https://github.com/google/protobuf/releases/tag/v3.0.2

[apolcyn]

thanks I see, so that fix should have been there.

But I've unfortunately been unable to reproduce.
Are there any proto definitions, copies of the actual ruby protobuf object, or grpc call type (unary, streaming) known to trigger this?

[timanovsky]

@apolcyn Proto definitions and call types - it is google-cloud-datastore (0.20.1) client. From the stack trace I see it was Datastore lookup call. Also this happened only once so far, contrary to pretty frequent occurrences with pre-3.0.2 version.

[apolcyn]

Realize this issue was never clearly resolved, but do we know if this has been seen since? Just by the type in the stack trace, I suspect it might have been fixed by one of the protobuf bug fixes since the OP (e.g. protocolbuffers/protobuf#3560)

[stale]

This issue/PR has been automatically marked as stale because it has not had any update (including commits, comments, labels, milestones, etc) for 180 days. It will be closed automatically if no further update occurs in 1 day. Thank you for your contributions!