Customize scheduling algorithms per queue

[hzxuzhonghu]

Support customized scheduling algorithms per queue, the main use scenario is: different users may need different algorithms, even in the same clusters.

For example, the enqueue is not necessary and may conflict with preempt. So it is better to configure the scheduling according to the user's requirement.

[Subham-Pratap]

Hey @hzxuzhonghu .

I'm interested in working on this project as part of community bridge. I have demonstrated experience working on scheduling algorithms and scheduling in general with projects on the same. I also have decent knowledge of Golang.

Is there any forum for discussion regarding the project which I can join or a timeline?

Thanks.

[Thor-wl]

Hey @hzxuzhonghu .

I'm interested in working on this project as part of community bridge. I have demonstrated experience working on scheduling algorithms and scheduling in general with projects on the same. I also have decent knowledge of Golang.

Is there any forum for discussion regarding the project which I can join or a timeline?

Thanks.

Warmly Welcome! Volcano community has wechat group and weekly meeting. You can search "k8s2222" for a apply to add friend and ask her to join you into volcano group. Our weekly meeting is every 15:00 PM every Friday. The meeting link can be found in README. Volcano feature design and version arrangement are listed volcano/docs. You can find our current ongoing feature and bug fix in the issue list. We are waiting for your PRs and discussions.

[hzxuzhonghu]

@Subham-Pratap Have you applied in the official website? And this project was selected by sresthasrivastava.

EDIT: @sresthas

[Subham-Pratap]

Yes I have applied on the official website and I'm looking at the issue list in the meantime.

[hzxuzhonghu]

@Subham-Pratap Could you select another one?

[Subham-Pratap]

Okay do you want me to select another project within volcano or select a different org. I have also applied on the official website, so should I withdraw it?

[hzxuzhonghu]

I mean you can select another project within volcano

[sresthas]

@k82cn @hzxuzhonghu
As of now the configuration is loaded into scheduler at the initialisation or when scheduling config is changed,
Now scheduler creates session periodically & goes through each of the actions and executes them.
This works assuming same config for all queues, hence once actions are loaded, they can be directly traversed in every period. In this design, in one session, jobs corresponding to different queues can be scheduled/traversed together.

For our use case,

1. we will need to modify this to traverse different actions and use different plugins based on queue, so we will need to have different sessions based on different scheduling configurations, right?
2. Also as of now Execute functions go through each of the jobs, find their corresponding Queues and do the action, instead this should be modified to only consider the jobs corresponding to the Queue in consideration, right?

[sresthas]

We will need to add an optional configmap key in QueueSpec. Currently config map is mounted as a volume to scheduler and that is passed as command like argument to scheduler. This can remain same for default scheduling configmap. But if user has specified a configmap key in QueueSpec, then that configmap should be loaded onto scheduler before opening a scheduling session.

[k82cn]

We will need to add an optional configmap key in QueueSpec. Currently config map is mounted as a volume to scheduler and that is passed as command like argument to scheduler. This can remain same for default scheduling configmap. But if user has specified a configmap key in QueueSpec, then that configmap should be loaded onto scheduler before opening a scheduling session.

It's better to have a design doc for this feature; we can discuss the detail when reviewing the proposal :)

[sresthas]

We will need to add an optional configmap key in QueueSpec. Currently config map is mounted as a volume to scheduler and that is passed as command like argument to scheduler. This can remain same for default scheduling configmap. But if user has specified a configmap key in QueueSpec, then that configmap should be loaded onto scheduler before opening a scheduling session.

It's better to have a design doc for this feature; we can discuss the detail when reviewing the proposal :)

I have created this design doc based on my understanding of the project.
https://docs.google.com/document/d/1pzH8gqqRUWKw5Wdh4sCjY2KlG6JqYtx0N_0X_Qzlw1U/edit?usp=sharing

[stale]

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).