pubsub: ResourceExhausted is considered 'retryable'

[sthomp]

Hello,

We recently upgrade our gcloud library to use the new streaming pubsub client. We've had a few encounters where the client appears to be idle even with a large number of Undelivered messages showing in stackdriver. After some investigation it appears the root cause is a resource limit:

rpc error: code = ResourceExhausted desc = Your project has exceeded a limit: (type="StreamingPull connections", current=1010, maximum=1000).

We've already reached out for a limit increase but I wanted to make sure from the client perspective that this error should be considered 'retryable'.

func isRetryable(err error) bool {
	s, ok := status.FromError(err)
	if !ok { // includes io.EOF, normal stream close, which causes us to reopen
		return true
	}
	switch s.Code() {
	case codes.DeadlineExceeded, codes.Internal, codes.Canceled, codes.ResourceExhausted:
		return true
	case codes.Unavailable:
		return !strings.Contains(s.Message(), "Server shutdownNow invoked")
	default:
		return false
	}
}

Or perhaps this error should be logged to make it apparent?

[jba]

It is indeed retryable, as the code you pasted demonstrates.

I suggest adding a log line to your local copy. We're hesitant to add logging to these clients to avoid spam. Something like this would probably be better surfaced on a metrics dashboard anyway.

[sthomp]

Ok will do. It wasn't apparent at first that this is a resource we need to monitor which is what caused problems for us initially. Also, I think we had some misunderstanding about what NumGoRoutines is as each of those go routines seems to take a connection.

[jba]

Yes, each goroutine creates a new stream. If you don't need the throughput, stick to NumGoroutines = 1 and use MaxOutstandingMessages/Bytes to adjust your throughput.