Skip to content

[proposal] Improve Coscheduling Capabilities #2480

Description

@ZiMengSheng

What is your proposal:

Gang 是 AI 场景下调度器的必备能力,在社区 scheduler 这种 Pod 作为一等公民的框架上增加扩展,使它原生支持Gang 是一件非常有挑战性的事情。Koordinator 已经提供了 Coscheduling 插件和基础能力,但是在用户体验和一些边界情况下仍然有不足之处,需要优化,包括但不限于:

  1. Pod 成员不足的时候会被 Coscheduling PreEnqueue 给拦住,Pod 上就没有任何 Event 了,所以这里需要有一个控制器为这种情况输出有效信息
  2. Koordinator Coscheduling 采用 PreEnqueue 使得 ActiveQ 中只有一个代表 Pod。但是在调度失败时,需要切换已经处于 Waiting 状态的代表 Pod 到当前失败 Pod,这种切换在 PostFilte 阶段完成,会和抢占有所冲突。所以这里需要一种方式使得切换 代表 Pod 的逻辑一定会被执行且不会跟抢占冲突。一种可能的方式是提供一种新的 BeforePostFilter 来放这个逻辑。

Gang is a must-have capability for schedulers in AI scenarios. It is very challenging to add extensions to the community scheduler framework, where Pod is a first-class citizen, to make it natively support Gang. Koordinator has provided Coscheduling plug-ins and basic capabilities, but there are still deficiencies in user experience and some edge cases, which need to be optimized, including but not limited to:

  1. When there are insufficient Pod members, it will be blocked by Coscheduling PreEnqueue, and there will be no Event on the Pod, so there needs to be a controller to output valid information for this situation
  2. Koordinator Coscheduling uses PreEnqueue so that there is only one representative Pod in ActiveQ. However, when scheduling fails, it is necessary to switch the representative Pod that is already in the Waiting state to the current failed Pod. This switch is completed in the PostFilter stage, which will conflict with preemption. Therefore, a way is needed here to ensure that the logic of switching the representative Pod will be executed and will not conflict with preemption. One possible way is to provide a new BeforePostFilter to put this logic.

Why is this needed:

Improve user experience and logical completeness

Is there a suggested solution, if so, please add it:

As described in Proposal

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions