[Bug] Connection hang after too high qps

[cambyzju]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master & 2.0

What's Wrong?

If current running connections bigger than max_mysql_service_task_threads_num, new connection will hang.

The problem is:
AcceptListener.java ==> handleEvent ==> channel.getWorker().execute got rejectedExecution, but do not throw it.

We should:

1. change mysql-nio-pool to use ThreadPoolManager.newDaemonCacheThreadPoolThrowException, make sure when connection rejected, throw RejectedExecutionException;
2. handle RejectedExecutionException inside AcceptListener.java::handleEvent, close the connection with a error message;

What You Expected?

return error instead of hang

How to Reproduce?

1. set max_mysql_service_task_threads_num=2 in fe.conf;
2. run two SQL at the same time: mysql -hxxx -e 'select sleep(300);';
3. run three SQL: mysql -hxxx -e 'select 1;';

In the log we found, the third sql already been rejected, and will not be executed anymore, the third connection hanged.

2024-02-29 11:55:49,299 WARN (doris-mysql-nio I/O-2|193) [ThreadPoolManager$LogDiscardPolicy.rejectedExecution():305] Task org.apache.doris.mysql.AcceptListener$$Lambda$1918/502547814@1a81657b rejected from mysql-nio-pool java.util.concurrent.ThreadPoolExecutor@59bfd0bf[Running, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 24]

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[cjj2010]

I want to try it

[didiaode18]

I executed select sleep(300) twice on my cluster and found that a timeout problem was reported.