[Improvement]: Optimizer has a performance bottleneck

[nicochen]

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

When there are too many optimize tasks, even if the optimizer consumes the optimize task with multiple parallelism, the efficiency is limited. The main reason for this problem is that each optimize only puts 10 tasks into the optimize group at a time.

How should we improve?

Support configuration of this property, or adaptive dynamic adjustment of this property.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[zhoujinsong]

@nicochen I have found the value of OptimizeQueueService#MAX_POOL_TASK_CNT had been setted to 50 in v0.4.0.
But I still agree that we should support setting this value in AMS config files.

@majin1102 @wangtaohz How do you think?

[wangtaohz]

I have found the value of OptimizeQueueService#MAX_POOL_TASK_CNT had been setted to 50 in v0.4.0.
But I still agree that we should support setting this value in AMS config files.

We put up to 10 or 50 tasks into optimize group at a time to avoid a table taking up all quotas, the optimizers may execute 10 or 50 tasks of a table and then turn to execute tasks of other tables.
However, the tasks of a table can't be committed before all tasks are executed and set to prepared, that's why a table takes a long time to finish the Optimize when there are many tasks.

I think we should modify the process above, and execute all tasks of a table at a time, then we can commit these tasks as soon as possible, and then we may not need a config like OptimizeQueueService#MAX_POOL_TASK_CNT.

Also, executing all tasks of a table at a time is consistent with the idea of this issue #946.

[zhoujinsong]

@wangtaohz
Yes, we should limit tasks number when planning if we are afraid of a table taking too much optimizing process resources at a time rather than limiting the tasks put into optimizing group queue.

Of course, we may not need to limit the planning task number for now.

[shidayang]

I agree to remove OptimizeQueueService#MAX_POOL_TASK_CNT. Prioritizing the optimization of a table is the most efficient way

[wangtaohz]

Yes, we should limit tasks number when planning if we are afraid of a table taking too much optimizing process resources at a time rather than limiting the tasks put into optimizing group queue.
Of course, we may not need to limit the planning task number for now.

I agree with this.

[zhoujinsong]

Completed by #993.