[Improvement][master] command slot allocation optimization

[eye-gu]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Currently, all masters occupy the slots allocated by the command. However, when resources are scarce, some masters no longer consume commands, which will result in commands in certain slots being unable to run. When registering a master, both /all and /run lists can be registered at the same time./all offline represents the need for fault tolerance, and /run offline reallocates slots. When the master resource is insufficient, it will be removed from the /run list

目前，所有master都占用command分配的插槽。但是在资源短缺时，一些mater不再消费command, 这将导致某些槽位的command一直无法运行。master在注册时, 可以同时注册/all和/run两个列表,/all下线代表需要容错, /run下线重新分配槽位, master资源不足时将自己从/run列表上下线.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Currently, all masters occupy the slots allocated by the command. However, when resources are scarce, some masters no longer consume commands, which will result in commands in certain slots being unable to run. When registering a master, both /all and /run lists can be registered at the same time./all offline represents the need for fault tolerance, and /run offline reallocates slots. When the master resource is insufficient, it will be removed from the /run list

Currently, all masters occupy slots allocated by command. But when there is a shortage of resources, some maters no longer consume commands, which will cause the commands in some slots to be unable to run all the time. When the master registers, it can register both /all and /run lists at the same time. When /all goes offline, it means that fault tolerance is needed. When /run goes offline, the slots are reassigned. When the master resources are insufficient, it will log itself off the /run list.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

[Radeity]

Hi, @eye-gu , I'm +0 to this for two reasons:

• First, Master is less likely to keep high resource utilization. When we judge overload and sleep for a while, maybe after few seconds, as more and more commands are consumed, Master should have enough CPU and memory to continue finding and consuming other commands. Thus, I think what user should think over whether this machine is suitable to deploy a Master, better not run some other compute-intensive applications on it.
• Second, if you introduce online/offline master path, you have to add another listener to synchronize online Master, which will introduce other overhead, and during this step, all Master can not consume commands. And if online and offline operation is frequent, performance will be worse.

[eye-gu]

Hi, @eye-gu , I'm +0 to this for two reasons:

• First, Master is less likely to keep high resource utilization. When we judge overload and sleep for a while, maybe after few seconds, as more and more commands are consumed, Master should have enough CPU and memory to continue finding and consuming other commands. Thus, I think what user should think over whether this machine is suitable to deploy a Master, better not run some other compute-intensive applications on it.
• Second, if you introduce online/offline master path, you have to add another listener to synchronize online Master, which will introduce other overhead, and during this step, all Master can not consume commands. And if online and offline operation is frequent, performance will be worse.

1. A master node is processing a large workflow, resulting in a consistently high memory usage. Perhaps users can choose the slot allocation method through configuration
2. It can also be solved through configuration, such as offline when the memory usage is 0.9 and online when it is 0.7
   What do you think about it?

[Radeity]

It can also be solved through configuration, such as offline when the memory usage is 0.9 and online when it is 0.7

We already have configuration max-cpu-load-avg and reserved-memory to judge overload, however, aforementioned problem still exists.

[eye-gu]

• And if online and offline operation is frequent

At present, the indicators for online and offline are the same, and when the master happens to load near this indicator, it is easy to encounter the frequent online and offline issues you mentioned. My example is that the master goes offline when the load is greater than 0.9, but instead of immediately going online when the load is less than 0.9, it goes online when the load is less than 0.7 to solve the problem of frequent online and offline

If this interval configuration method is used, the master will occupy slots for a period of time without consuming commands, so this is the problem that my issue will solve.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.