HIVE-29048: Increase default pool and queue size of HS2 Background Async Threadpool.

[mdayakar]

HIVE-29048: Increase default pool and queue size of HS2 Background Async Threadpool.

What changes were proposed in this pull request?

To increase the default value for the hive.server2.async.exec.threads and hive.server2.async.exec.wait.queue.size configuration from 100 to 250.

Recently most of the customers reported below error on beeline

0: jdbc:hive2://localhost> show databases;
Error: The background threadpool cannot accept new task for execution, please retry the operation (state=,code=0)

HS2 server logs shows below exception

Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@24363395 rejected from java.util.concurrent.ThreadPoolExecutor@54ef7045[Running, pool size = 100, active threads = 100, queued tasks = 100, completed tasks = 682504]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_342]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_342]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_342]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) ~[?:1.8.0_342]
at org.apache.hive.service.cli.session.SessionManager.submitBackgroundOperation(SessionManager.java:760) ~[hive-service-3.1.3]
at org.apache.hive.service.cli.session.HiveSessionImpl.submitBackgroundOperation(HiveSessionImpl.java:598) ~[hive-service-3.1.3]
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:279) ~[hive-service-3.1.3]
... 50 more

So here for most of the customers load, the default value 100 is not sufficient so their queries are getting rejected.

Why are the changes needed?

The changes required to handle most of the customer's cluster basic load.

Does this PR introduce any user-facing change?

No

How was this patch tested?

NA

[deniskuzZ]

@mdayakar, these configs remain unchanged since Hive 2.3. Typically, customers can override them by modifying hive-site.xml
downstream we use hive-site.xml template with the following values, but it’s not a one-size-fits-all solution

hive.server2.async.exec.threads=200
hive.server2.async.exec.wait.queue.size=1000

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[mdayakar]

@mdayakar, these configs remain unchanged since Hive 2.3. Typically, customers can override them by modifying hive-site.xml downstream we use hive-site.xml template with the following values, but it’s not a one-size-fits-all solution

hive.server2.async.exec.threads=200
hive.server2.async.exec.wait.queue.size=1000

Hi @deniskuzZ ,
I agree that users can modify the value in hive-site.xml or they can set through via beeline also but if they are not configuring it in hive-site.xml then the default value 100 will be considered and for most of the customer's workload this value is not sufficient. So I proposed to increase this value to 250. Anyway if there is no load on the cluster these threads will be terminated based on keepalive timeout(10s) so even if the default value is 250 there will not be any issue. This is mainly for those customers who uses the default value configured in HiveConf.java file and faces the RejectedException due to 100 default value.

[deniskuzZ]

Hi @deniskuzZ , I agree that users can modify the value in hive-site.xml or they can set through via beeline also but if they are not configuring it in hive-site.xml then the default value 100 will be considered and for most of the customer's workload this value is not sufficient. So I proposed to increase this value to 250. Anyway if there is no load on the cluster these threads will be terminated based on keepalive timeout(10s) so even if the default value is 250 there will not be any issue. This is mainly for those customers who uses the default value configured in HiveConf.java file and faces the RejectedException due to 100 default value.

Cluster administrator should manage the configs. None of production setups should rely on default values. Note, I don't have strong opinions about this change

+0