Cause and mitigation for "Client has been stopped. Can't make any more RPCs." error

[alex-vasenin]

Most of the times GRPC works reliably in our app, however in rare cases I noticed this error "Client has been stopped. Can't make any more RPCs." after which the client becomes unusable.

What might be causing this error? I'm not stopping the client manually and network errors doesn't seem to cause it (I don't have a way to reproduce).

It seems like I should be ready to recreate GRPC client (and all service clients that wraps it) if it stops for whatever reason. What should be my approach to make sure my service clients are always usable? Is it true that if client is stopped try await client.runConnections() should complete? Do I need to recreate TransportServices as well?

Here is my client init code for reference:

    final class Service {        
        static let grpcClient = {
            let client = GRPCClient(transport: transport,
                                    interceptors: [authInterceptor, loggingInterceptor])
            Task { try! await client.runConnections() } // Runs indefinetly
            return client
        }()
        
        private static let authInterceptor = AuthInterceptor()
        private static let loggingInterceptor = LoggingInterceptor()
        
        private static let transport = try! Transport(target: target, transportSecurity: .tls)
        typealias Transport = HTTP2ClientTransport.TransportServices
        
        private static let target: any ResolvableTarget = .dns(host: AppConfig.grpcHost)
    }

[glbrntt]

It seems like I should be ready to recreate GRPC client (and all service clients that wraps it) if it stops for whatever reason. What should be my approach to make sure my service clients are always usable?

This shouldn't happen.

This error is only thrown in one place. It happens when the client is in the stopping or stopped states. That only happens when:

1. beginGracefulShutdown is called, or
2. runConnections() returns or throws.

Can you confirm you're not calling beginGracefulShutdown or cancelling the task anywhere?

The other option is the connect() function in the HTTP2ClientTransport.TransportServices returning (this is what runConnections() calls). From inspection this also doesn't appear to be the case: it will only return as a result of beginGracefulShutdown being called (or the task being cancelled).

[alex-vasenin-peggy]

Hi George!

I am NOT calling beginGracefulShutdown anywhere. If runConnections() did throw errors I would see a crash, and I don't see anything like this, it probably just exits. I DO cancel calls to GRPC services sometimes and it seems to work fine (client continues to work). Since Task { try! await client.runConnections() } creates unstructured task, cancellation should not be propagated here.

I can add fatalError to catch the cases where runConnections() exits, but how do I get the cause of the exit?

[glbrntt]

Thanks for confirming, sometimes people post snippets which don't actually match the code they're running so it's good to double check!

I don't think the fatal error will provide much insight here: as you note we won't know the cause of the exit. Let me do a deeper dive through the code to check I didn't miss anything.

[alex-vasenin-peggy]

I added fatalError after runConnection() to prove that it exits and it indeed happened once.

[glbrntt]

Okay, there's another codepath that could lead to this: when the resolver throws an error.

Would you be able to try this out to verify this is the problem you're hitting? The following code is for a custom resolver: it'll just wrap the DNS resolver and fatal error if it throws. If that does happen you should be able to workaround the problem of the client shutting down by replacing the fatal error with the commented out line to return an empty result. We'll then need a more robust fix on our side.

To use the resolver you'll need to register it with your transport and use a different target, see the second snippet.

struct NoThrowDNS: NameResolverFactory {
  struct Target: ResolvableTarget {
    var host: String
    var port: Int? = 443
  }

  func resolver(for target: Target) -> NameResolver {
    let dns = NameResolvers.DNS()
    let resolver = dns.resolver(for: .dns(host: target.host, port: target.port))

    return NameResolver(
      names: RPCAsyncSequence(wrapping: ResolutionResults(base: resolver.names)),
      updateMode: resolver.updateMode
    )
  }

  struct ResolutionResults: AsyncSequence {
    typealias Element = NameResolutionResult
    typealias AsyncIterator = Iterator

    var base: RPCAsyncSequence<NameResolutionResult, any Error>

    init(base: RPCAsyncSequence<NameResolutionResult, any Error>) {
      self.base = base
    }

    func makeAsyncIterator() -> Iterator {
      Iterator(base: self.base.makeAsyncIterator())
    }

    struct Iterator: AsyncIteratorProtocol {
      var base: RPCAsyncSequence<NameResolutionResult, any Error>.AsyncIterator

      init(base: RPCAsyncSequence<NameResolutionResult, any Error>.AsyncIterator) {
        self.base = base
      }

      mutating func next() async throws -> Element? {
        do {
          return try await base.next()
        } catch {
          fatalError("Resolver threw error: \(error)")
          // return NameResolutionResult(endpoints: [], serviceConfig: nil)
        }
      }
    }
  }
}

This is the rough shape of how you use the custom resolver:

var registry = NameResolverRegistry.defaults
registry.registerFactory(NoThrowDNS())

let transport = HTTP2ClientTransport.TransportServices(
  target: NoThrowDNS.Target(host: AppConfig.grpcHost), 
  transportSecurity: .tls, 
  resolverRegistry: registry
)

[alex-vasenin-peggy]

I managed to reproduce the issue (on real device with debugger attached) by turning on Network Link Conditioner with 100% Loss profile. I started with NLC disabled, made few successful calls to the server, then turned NLC on and initially it was failing with "unavailable: "Stream unexpectedly closed."" as expected. But then the following error appeared in NoThrowDNS:

internalError: "Couldn't resolve address for staging.grpc.redacted.com:443" (cause: "nodename nor servname provided, or not known")

Since I replaced fatalError with error logging and rethrow, the app continued to work, threw few "unavailable: "Channel isn't ready."" errors, but in a few seconds crashed on fatalError I put after client.runConnection(). I cannot reliably reproduce the issue, but in about an hour playing with NLC it appeared twice. I'm gonna add the workaround you suggested and see if the issue stops appearing in crash reports. We don't have many testers now, so it might take a week or so.

So, overall it looks like a bug. If GRPCClient.runConnections() supposed to run indefinitely it should do so, even if DNS fails at some point.

P.S. I added authority: resolver.authority parameter to NameResolver.init call because otherwise I could not connect to the server due to "bad certificate format" error.

[alex-vasenin-peggy]

@glbrntt I can confirm that the workaround you proposed really works - I didn't have a single exit on .runConnections() for two weeks. When can we expect more permanent solution?

[glbrntt]

Thanks for confirming; I was hoping to get to it soon but things have been a little busy recently. (Sorry!)