Request hangs with no feedback if server connection is refused

[caseylucas]

If the server is not listening for requests and the client attempts to make a request, the client will hang with no feedback on the failed request. Reproduce by:

1. make one basic grpc request to verify all is ok
2. kill / stop grpc server (envoy, node proxy, etc.)
3. attempt to make another grpc request

In the console (at least under chrome) you'll see something similar to:

POST http://SOME_HOST/XXX/SOME_METHOD_NAME net::ERR_CONNECTION_REFUSED

The on('error'...) handler is not invoked and you have no way of knowing that the request failed.

I believe this is occurring because when XHR state is being inspected in the COMPLETE handler, the XHR status of 0 is not being checked. XHR status of 0 means that there was some problem that prevented the request from being sent (https://stackoverflow.com/questions/3825581/does-an-http-status-code-of-0-have-any-meaning/#answer-26451773)

Relevant code:
https://github.com/grpc/grpc-web/blob/master/javascript/net/grpc/web/grpcwebclientreadablestream.js#L196-L214

I added a check for 0 and was then notified of the failed requests:

--- a/javascript/net/grpc/web/grpcwebclientreadablestream.js
+++ b/javascript/net/grpc/web/grpcwebclientreadablestream.js
@@ -196,7 +196,7 @@ const GrpcWebClientReadableStream = function(genericTransportInterface) {
   events.listen(this.xhr_, EventType.COMPLETE, function(e) {
     if (!self.onErrorCallback_) return;
     var lastErrorCode = self.xhr_.getLastErrorCode();
-    if (lastErrorCode != ErrorCode.NO_ERROR) {
+    if (lastErrorCode != ErrorCode.NO_ERROR || (self.xhr_.getStatus() == 0)) {
       self.onErrorCallback_({
         code: StatusCode.UNAVAILABLE,
         message: ErrorCode.getDebugMessage(lastErrorCode)

Please let me know if you want a PR or have thoughts on why my findings are invalid.

[stanley-cheung]

Thanks for the report. We had a bit of back and forth about this in the past. I will have to look this up again.

[caseylucas]

Can I provide any help? Do you have any references to past discussions or findings?

[theguruofreason]

I'm having a similar problem: if I make a client request for a server-side stream and the server is down, I get a massive spam of POST ERR_CONNECTION_REFUSED errors in the console.

[caseylucas]

@theguruofreason, If you patch your local copy of grpcwebclientreadablestream.js (in node_modules) does it fix the problem?

[theguruofreason]

@caseylucas I'm actually using Improbable Engineering's typescript implementation, though I believe the underlying library is the same (not sure on this). This is the only lead I have after hours of scouring the web and grpc codebase for what could be causing this error spam. I have a observable for the rpc that I complete and the POST error spam continues, bafflingly. I've tried stream.cancel() to no avail as well. My guess is that this XHR error is slipping by and the underlying mechanism keeps trying to get stream rpc responses as a result, but the code path is pretty hard for me to follow and I don't see a clear culprit.

[ktm-labs]

I've tried this out, using the Echo example. I then stopped the server by running the docker stop command. When attempting to make another grpc request, I did get an error message in the browser: Error code: 14 "upstream connect error or disconnect/reset before headers. reset reason: connection failure". And nothing in the console (Chrome). @theguruofreason, @caseylucas I wonder if you can confirm if this has been fixed for you, or if you are still having the same issue?

[zsluedem]

I met some similar problem like this.

I send a request to server which is down I can only get console error and not trigger the error hanler.

I think it is possible that grpc-web is only handling error from the grpc and eat all the errors in http request to the proxy.

[hackerwins]

@stanley-cheung

I met the same problem. I think that below XHR logic should handle the error.

grpc-web/javascript/net/grpc/web/grpcwebclientbase.js

Line 74 in 3763612

GrpcWebClientBase.prototype.rpcCall = function(

• https://developers.google.com/closure/library/docs/xhrio#listening-for-request-events
• https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequestEventTarget/onerror

[stanley-cheung]

This should be fixed recently. This is a unit test to show that if there's an underlying xhr error, it should be returned as the err of the main callback. For streaming, it will be return as .on('error', ...).