[Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?

[gjzhappy]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

查看doc，目前只提供三种负载均衡（随机/平滑轮询/线性加权），为何不把‘Yarn’当作一种策略，根据Yarn Service API，Yarn是支持如shell脚本在内的很多服务调用的，即xxx on Yarn，所以，这一块的可扩展性还是很大的，请问是否有开发计划？

我的应用场景是这样的，我希望在调用shell脚本的时候，也能提交给yarn。

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Looking at the doc, there are currently only three types of load balancing (random/smooth round robin/linear weighted), why not use 'Yarn' as a strategy, according to the Yarn Service API, Yarn supports many service calls such as shell scripts Yes, that is xxx on Yarn, so the scalability of this area is still very large. Is there any development plan?

My application scenario is like this, I hope that when calling the shell script, it can also be submitted to yarn.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

[Radeity]

Hi, @gjzhappy , i agree with you that we have to extend and support more scheduling strategies, there's still a lot of room for improvement. HOWEVER, i don't think YARN can be regarded as a scheduling strategy for DS's Master. Thus, maybe you can extend shell task and support to submit it on Yarn.

[Radeity]

In addition, use english to describe your issue is better :D

[gjzhappy]

Hi, @Radeity, thanks a lot for your reply.
About the scheduling strategies, what i thought is to replace the strategy with the Yarn api, let Yarn decide where to dispatch the task, but i'm not sure if it is feasible for DS.（this is yarn api doc: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html）

And you mentioned about extend shell task to support it on Yarn, and I'm very interest in this, i want to know how to make it through DS, can you show me some tips? thanks again.

[Radeity]

About the scheduling strategies, what i thought is to replace the strategy with the Yarn api, let Yarn decide where to dispatch the task, but i'm not sure if it is feasible for DS.

In my view, it's not suitable to hand over task dispatching to Yarn.

And you mentioned about extend shell task to support it on Yarn, and I'm very interest in this, i want to know how to make it through DS, can you show me some tips? thanks again.

Whether extend shell task has to be further discussed, ideally, it can achieve the same outcome as K8S Executor in our roadmap. You can try to give some proposal based on your scenario :D

[gjzhappy]

In my view, it's not suitable to hand over task dispatching to Yarn.

Scheduler and App Manager are fundamental functions of Yarn, do I misunderstand something?

Whether extend shell task has to be further discussed, ideally, it can achieve the same outcome as K8S Executor in our roadmap. You can try to give some proposal based on your scenario :D

In my scenario, some of the tasks in the workflow will run as script, bat, sh etc. And the real procedure will triggered by the script, which will cost much more time, cpu, gpu and memory. I wonder if these scripts can be dispatched on Yarn, cause with Yarn, it supports like gpu management. Or, do you have any suggestion on this requirement? Thanks.

[Radeity]

In my view, it's not suitable to hand over task dispatching to Yarn.

DS master is responsible for DISPATCHING tasks to DS worker. DS worker will submit task on Yarn if task (eg. Spark, Flink) support yarn mode. You can use yarn api to extend and support shell task submit to Yarn. But, I'm not familiar with these Yarn API... maybe you have to explore it by yourself.