add elastic scheduler design doc

[qiankunli]

design doc for #1884

[qiankunli]

关于优先级这块有一个小细节，机器学习训练job下的pod通常分为master（仅有一个，也可以视为index=0 worker）和worker两种角色，引入弹性机制之后，希望能：优先创建master，删除elastic pod时优先删除worker。

大的思路有两种

1. master/worker 区分label。 需要operator 和 volcano 约定一个label
2. master/worker 设置不同的Priority。

关于Priority 有一个比较麻烦的地方，设置pod.Priority 必须通过PriorityClass 来进行，这个对上游的operator 就不太友好了：这个PriorityClass是用户来创建 还是operator 来创建？前者不方便，后者operator太麻烦。为此，有以下几种方法干预task.Priority

1. kubeflow operator 为每个pod 加一个label training.kubeflow.org/replica-index，这个值用来计算优先级。不过这个label对volcano 不太通用
2. kubeflow 任务的命名规则是 job-name-worker-${index} ，可以比较task优先级时使用name作为依据。目前volcano 在判断task优先级依次根据：pod.priority pod.creationTimestamp 和 pod.uid，可以将uid 改为pod.name
3. operator 和 volcano 可以约定一个新的label ，比如 volcano.sh/task-priority 用来传递task 优先级
   @william-wang @Thor-wl

[k82cn]

LGTM overall :)

[Thor-wl]

/lgtm overall.
Thanks for your contribution. This feature is meaningful.Please format the doc and fix some syntax error

[Thor-wl]

1. master/worker 区分label。 需要operator 和 volcano 约定一个label

Perhaps we can submit this design to kubeflow community.

[Thor-wl]

All these design are based on operator users. How about users makeing use of vcjob directly?

[qiaqiatic]

关于优先级这块有一个小细节，机器学习训练job下的pod通常分为master（仅有一个，也可以视为index=0 worker）和worker两种角色，引入弹性机制之后，希望能：优先创建master，删除elastic pod时优先删除worker。

大的思路有两种

1. master/worker 区分label。 需要operator 和 volcano 约定一个label
2. master/worker 设置不同的Priority。

关于Priority 有一个比较麻烦的地方，设置pod.Priority 必须通过PriorityClass 来进行，这个对上游的operator 就不太友好了：这个PriorityClass是用户来创建 还是operator 来创建？前者不方便，后者operator太麻烦。为此，有以下几种方法干预task.Priority

1. kubeflow operator 为每个pod 加一个label training.kubeflow.org/replica-index，这个值用来计算优先级。不过这个label对volcano 不太通用
2. kubeflow 任务的命名规则是 job-name-worker-${index} ，可以比较task优先级时使用name作为依据。目前volcano 在判断task优先级依次根据：pod.priority pod.creationTimestamp 和 pod.uid，可以将uid 改为pod.name
3. operator 和 volcano 可以约定一个新的label ，比如 volcano.sh/task-priority 用来传递task 优先级
   @william-wang @Thor-wl

Hi @qiankunli ,

Thanks for this great work. I think this is a great contribution that might benefit Volcano users globally. We will really appreciate if we can try communicating in english.

[shinytang6]

rest lgtm )

[shinytang6]

The description here is slightly inaccurate. All pods will be created initially, but minAvailable pods will be scheduled first and then schedule elastic pods if there are free resources.

[qiankunli]

it is more accurate, I have changed the doc

[k82cn]

/approve

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: k82cn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [k82cn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Thor-wl]

/lgtm