[grpc-js] Error: 13 INTERNAL: Received RST_STREAM with code 0

[dnalborczyk]

Problem description

grpc-js client is throwing the following exception under high load:

ERROR Error: 13 INTERNAL: Received RST_STREAM with code 0
    at Object.callErrorFromStatus (/repro/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/repro/node_modules/@grpc/grpc-js/build/src/client.js:176:52)
    at Object.onReceiveStatus (/repro/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:342:141)
    at Object.onReceiveStatus (/repro/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:305:181)
    at Http2CallStream.outputStatus (/repro/node_modules/@grpc/grpc-js/build/src/call-stream.js:117:74)
    at Http2CallStream.maybeOutputStatus (/repro/node_modules/@grpc/grpc-js/build/src/call-stream.js:156:22)
    at Http2CallStream.endCall (/repro/node_modules/@grpc/grpc-js/build/src/call-stream.js:142:18)
    at ClientHttp2Stream.<anonymous> (/repro/node_modules/@grpc/grpc-js/build/src/call-stream.js:420:22)
    at ClientHttp2Stream.emit (events.js:314:20)
    at emitCloseNT (internal/streams/destroy.js:81:10) {
  code: 13,
  details: 'Received RST_STREAM with code 0',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

Reproduction steps

I have a repro repository in the works and will update with the link.

Environment

• OS name, version and architecture: macOS Big Sur v11.0 [20A5343j]
• Node version: v14.7.0 (seems to be irrelevant, also reproduces with v10.22.0 and v12.18.3
• Node installation method: nvs
• If applicable, compiler version: n/a
• Package name and version: grpc-js v1.1.3

Additional context

[racerxdl]

Same issue here. With nodejs v10, and with no load.

[murgatroid99]

You say grpc-js throws this error. This is a standard request status error, which you should be handling. If your method has a response stream, the error will be emitted in the error event (handled with call.on('error', ...)).

Regarding the error itself, the message indicates that the server unexpectedly closed the stream before the client considered it to be complete. The code 0 indicates that the server did consider the stream to be complete. I have a guess about what that means, and an reproduction code you can provide should help verify that.

[dnalborczyk]

thank you @murgatroid99

I'll have a repro repo up in a bit. the exception is not being thrown when one switches out the client to use the native deprecated grpc module.

[civilizeddev]

I faced the same problem.

You need to make sure your call method path starts with '/' just like below.

/<package>.<service>/<method>

Having only <method> in the path might cause this problem.

[SpencerHehl]

any further progress on this? we have an issue with this error code as well.

[murgatroid99]

As I said before, reproduction code for this bug would help figure out what's happening. If you can share some that would be great.

[chroth7]

if it helps, we had these errors when running on GKE (google cloud's kubernetes engine), when under high load and resulting memory pressure.
Essentially, our upper limit on the memory was too low, which caused the workloads to use more memory than its limit, which in turn put memory pressure on the nodes, which in turn made it fall apart...
In the specific case, we had a PubSub listener with surges at times.

[SpencerHehl]

@chroth7 you basically just described our setup as well. We are running on GKE, with relatively high load, and subscribing to a long standing stream. I'll go back and check resource monitoring, but I was considering running just on a high-spec GCE to see if it might be a resource thing. Thanks for the thoughts and I'll post back in here if that helps out for us as well.

[eli-stewart]

I am seeing this as well. I am running a Typescript React App with a node.js server. The server is throwing this error fairly regularly. Something like once an hour. I also get this similar error: @firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 14 Message: 14 UNAVAILABLE: Stream refused by server but about 20% as frequently.

I had a crash on my site on Monday where I was not able to read or write the database and I am trying to figure out if its related.

These are the 3 types of errors I am receiving of this type:
@firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 13 Message: 13 INTERNAL: Received RST_STREAM with code 2

@firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 14 Message: 14 UNAVAILABLE: Stream refused by server

@firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 13 Message: 13 INTERNAL: Received RST_STREAM with code 0

[castarco]

Does anybody know where to lookup for what each code means?

[dnalborczyk]

@castarco https://grpc.github.io/grpc/core/md_doc_statuscodes.html

[davebarkerKount]

error text Error: 14 UNAVAILABLE: Stream refused by server at Object.callErrorFromStatus (<local repo>/node_modules/@grpc/grpc-js/src/call.ts:81:24) at Object.onReceiveStatus (<local repo>/node_modules/@grpc/grpc-js/src/client.ts:343:36) at Object.onReceiveStatus (<local repo>/node_modules/@grpc/grpc-js/src/client-interceptors.ts:430:34) at Object.onReceiveStatus (<local repo>/node_modules/@grpc/grpc-js/src/client-interceptors.ts:392:48) at <local repo>/node_modules/@grpc/grpc-js/src/call-stream.ts:299:24 at processTicksAndRejections (node:internal/process/task_queues:78:11) { code: 14, details: 'Stream refused by server', metadata: Metadata { internalRepr: Map(0) {}, options: {} }

happens calling a gRPC/go micro-service intermittently after a wait. So use case is like this:

1. setup client.
2. make several calls to service to setup integration test.
3. hard wait 90 secs
4. make another call to server via client for test assert. At this step approx. one out of 10 times will get one of the errors described above or this one.

grpc-js: 1.4.2