[Improvement] Fast fail when grpc request encounter unable to create new native thread exception

[zuston]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

In online applications, I found the some jobs will hang when encountering the exception as follows

23/10/27 07:54:32 ERROR ManagedChannelImpl: [Channel<27>: (10.67.67.91:21000)] Uncaught exception in the SynchronizationContext. Panic!
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:714)
	at io.grpc.netty.shaded.io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:32)
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$1.execute(ThreadExecutorMap.java:57)
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:975)
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:944)
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:827)
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.lazyExecute(SingleThreadEventExecutor.java:820)
	at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractScheduledEventExecutor.schedule(AbstractScheduledEventExecutor.java:263)
	at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractScheduledEventExecutor.schedule(AbstractScheduledEventExecutor.java:177)
	at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutorGroup.schedule(AbstractEventExecutorGroup.java:50)
	at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutorGroup.schedule(AbstractEventExecutorGroup.java:32)
	at io.grpc.internal.Rescheduler.reschedule(Rescheduler.java:63)
	at io.grpc.internal.ManagedChannelImpl.rescheduleIdleTimer(ManagedChannelImpl.java:447)
	at io.grpc.internal.ManagedChannelImpl.exitIdleMode(ManagedChannelImpl.java:402)
	at io.grpc.internal.ManagedChannelImpl$RealChannel$2.run(ManagedChannelImpl.java:984)
	at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:95)
	at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:127)
	at io.grpc.internal.ManagedChannelImpl$RealChannel.newCall(ManagedChannelImpl.java:981)
	at org.apache.uniffle.client.impl.grpc.RetryInterceptor$1RetryingUnaryRequestClientCall.startCall(RetryInterceptor.java:98)
	at org.apache.uniffle.client.impl.grpc.RetryInterceptor$1RetryingUnaryRequestClientCall.halfClose(RetryInterceptor.java:94)
	at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:318)
	at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:227)
	at org.apache.uniffle.proto.ShuffleServerGrpc$ShuffleServerFutureStub.requireBuffer(ShuffleServerGrpc.java:920)
	at org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.doRequirePreAllocation(ShuffleServerGrpcClient.java:291)
	at org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.requirePreAllocation(ShuffleServerGrpcClient.java:277)
	at org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.lambda$sendShuffleData$3(ShuffleServerGrpcClient.java:405)
	at org.apache.uniffle.common.util.RetryUtils.retry(RetryUtils.java:49)
	at org.apache.uniffle.common.util.RetryUtils.retry(RetryUtils.java:31)
	at org.apache.uniffle.client.impl.grpc.ShuffleServerGrpcClient.sendShuffleData(ShuffleServerGrpcClient.java:400)
	at org.apache.uniffle.client.impl.ShuffleWriteClientImpl.lambda$sendShuffleDataAsync$4(ShuffleWriteClientImpl.java:239)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

And the retry will cost too much time and always fail. So I want to exclude this exception when retrying

How should we improve?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!