Error: Received RST_STREAM with error code 2

[lafama]

Environment details

• OS: darwin x64
• Node.js version:8.5.0
• npm version:5.4.2
• google-cloud/pubsub version:0.14.1

Steps to reproduce

1. require google-cloud/pubsub

subscription=pubsub.topic('awesomeTopic).subscription('awesomeTopicSubcription',{reuseExisting: true})
3.

subscription.on('error',function(error){
    logger.error('subscription',error);
  })

4. This (3) will log the following

{"message":"subscription",
"stack":"Error: Received RST_STREAM with error code 2\n    at ClientDuplexStream.<anonymous> 
(/app/node_modules/@google-cloud/pubsub/src/connection-pool.js:218:21)\n    
at emitOne (events.js:115:13)\n    at ClientDuplexStream.emit (events.js:210:7)\n    
at ClientDuplexStream._emitStatusIfDone (/app/node_modules/grpc/src/node/src/client.js:260:10)\n    
at ClientDuplexStream._receiveStatus (/app/node_modules/grpc/src/node/src/client.js:233:8)\n    
at /app/node_modules/grpc/src/node/src/client.js:757:12","code":13,"timestamp":"2017-09-19T04:57:56.510Z",
"level":"error"}

[callmehiphop]

@lafama thanks for reporting. I believe this already being investigated, but this message is almost always received when the PubSub api closes a stream. However we usually retry on that specific error, so I'm intrigued as to why you're seeing it.

I'm unable to reproduce with the steps you've provided, so if you could provide us with a gist/repo/etc. that reproduces the error you're seeing, that would help tremendously.

[lafama]

What would make PubSub api close a stream? My worker process hangs after this error and i have to keep restarting it. The challenge reproducing this is that it occurs infrequently and at random time in production. It also only stated when upgrade to 0.14.0 from 0.13.0

[callmehiphop]

The streaming API is new, hence why this only occurs in 0.14.x. Prior to this release we manually polled the servers.

Usually the PubSub API closes the stream after about 30 minutes. However, the only time we would emit an error like this would be if the client was unable to re-establish a connection within 5 minutes of the previous connections closing. So it might be that the server closes the connection and won't allow for any new connections. @jganetsk for thoughts here.

If that turns out to be the case, then you might want to consider re-creating the subscription object when that error fires, possibly with a backoff of some kind.

[jganetsk]

Is there some way to confirm that the user is experiencing a prolonged absence of connectivity, and that's why the error is occurring? Maybe the client library's error handling code has a bug.

[callmehiphop]

@jganetsk I think you're right, after investigating it seems that the metadata event does not always fire, which is what I was using to see if the stream connected. I'm thinking instead we should just check that the channel is connected and assume that when creating a stream it'll just work. Does that sound correct to you?

[jganetsk]

I don't know these details. I trust you and @lukesneeringer can work them out.

[lafama]

This error seems to occur exactly 34 minutes into processing. So i have to restart my worker process every 34th Minute..

[callmehiphop]

@lafama I'm working on resolving this, however in the mean time I would suggest downgrading to 0.13.

[lafama]

Downgraded to 0.13.1. Works for now, @callmehiphop update me, when the issue is fixed

[callmehiphop]

@lafama I've opened a PR (#2627) that I believe will address the issue you are seeing. Would you mind testing it?

[lafama]

@callmehiphop this PR seems to work fine, so far in tests no errors are being logged and the process does not hung 👍

[lafama]

Just realised that after a while no new messages are being pulled but no errors are logged.

[callmehiphop]

Are you running your code in a specific environment (GKE?) and do you know after about how long do the messages stop coming?

[lafama]

No am running the code on my comp (OS: darwin x64). Will update on the exact time later

[callmehiphop]

@lafama we issued a release today that I believe will address this issue, so I'm going to close it. If you think I closed it prematurely, please feel free to re-open.