DNS creates unbounded number of grpc-default-executor threads, and can't be overriden

[lalithsuresh]

What version of gRPC are you using?

1.4.0

What JVM are you using (java -version)?

1.8.0_101

What did you do?

I'm running about fifty processes that use grpc to communicate with each other. The workload involves events that trigger bursts of all-to-all messaging between the processes, and thereby moments of high load. After each instance of this event, I see 5-10 grpc-default-executor threads being created per process.

I have supplied FixedThreadPools through executor() for both the server and client channels.

Why are grpc-defaut-executor threads showing up despite overriding executor()? Is this expected behavior?

[ejona86]

Looking through the code it seems there are two unconditional users of it: DNS resolution and for the OOBChannel in the LoadBalancer. The OOBChannel is only used by GrpcLb at this point, so it probably didn't contribute.

That would seem to imply that all 5-10 threads were used for DNS lookups. Are you creating 5-10 ManagedChannels in a burst?

It is unfortunate to create that many threads just for DNS. I think C core uses a single thread, although that would imply the DNS resolution occurs serially, which may be very noticeable latency for a burst of 10.

I think we are avoiding the application-provided executor pool for DNS because the application could not predict that we could be using it for blocking I/O, which would be very important for a fully-async app or an app using directExecutor.

[lalithsuresh]

That makes sense, I am creating a large bunch of channels in a burst (I'm not sure how to avoid doing that with this workload :\ ).

Two questions:

• I am for the time being just using an IP:port tuple to refer to these processes. Why is DNS resolution being triggered?
• Is it possible to allow users to override the shared thread pool? Or change the default to a fixed size thread pool with a configurable number of threads?

[ejona86]

Why is DNS resolution being triggered?

Because we don't know ahead-of-time whether a DNS query will be made. That may be a possible enhancement though.

Is it possible to allow users to override the shared thread pool?

Theoretically, yes. Practically it is an API concern. We should figure something out though.

Or change the default to a fixed size thread pool with a configurable number of threads?

That's not as great as it sounds, because Java's fixed-size thread pools eagerly create threads. So unless we limit to 1 thread the fixed-size thread pool would frequently create more threads than the current cached thread pool.

CC @zhangkun83

[ejona86]

#6238 allowed specifying the Executor used by DnsNameResolver ("blockingExecutor"). It was later renamed to ("offloadExecutor") on the ManagedChannelBuilder (#6389).