Configure a way call the refresh api AkkaDiscoveryNameResolver preiodically or on event changes in the DNS

[Marcus-Rosti]

Short description

gRPC likes to cache all the dns lookups from AkkaDiscoveryNameResolver. On scale up events in kubernetes for instance, the client has no way then to discover those new endpoints until a cache gets invalidated and it reresolves them.

There should be a configuration to add a way to call the refresh endpoint in AkkaDiscoveryNameResolver periodically.

Details

Could just work via a tick in akka streams or it could be event driven but that may be too endpoint specific and more of an akka-management thing:

hand waving akka stream specifics

Source.tick(clientSettings.refreshDnsInterval.seconds, 0.seconds, ())
  .map(akkaDiscoveryNameResolver.refresh())
  .run()

Could also be solved via what @jrudolph mentions in akka/akka-http#3505

[danieltahara]

^ This would be very helpful (or at least some default). We just figured out this workaround after a painful incident.

[johanandren]

Is this for a specific underlying discovery impl (either of the DNS ones?) If so I'd expect that the current behavior is that it stays in the cache of that until the TTL of the DNS entry runs out. That would mean that this feature request is basically about a way to specifically tell it to throw away the cached DNS entry for a service earlier (periodically or up to a given application to trigger).

I see that the Netty client API for name resolution (io.grpc.NameResolver#refresh) has a refresh method, but right now that will just hit the cached DNS value, not force-evict it. Not entirely clear to me exactly when Netty will call that (on transient connection failures).

If it is DNS I think we'd need to add support for that in the underlying resolver in Akka first, and then possibly add the capability of periodic forced-evict-re-refresh here. The evict would need to be followed by a call to the io.grpc.NameResolver.Listener with the fresh lookup results, so it's not easy to actually workaround with any public API as far as I can see.

[danieltahara]

This is specifically for akka discovery. There are enough layers involved in the DNS resolver that I agree it would be a taller task. However, switching to use akka discovery (in particular for integration with kubernetes) allows us to hook into akka discovery itself to force refreshes, as suggested by the example above.

Not entirely clear to me exactly when Netty will call that (on transient connection failures).

I believe this is correct, empirically.

[johanandren]

Upstream cache-evict implemented in akka/akka#32260