Add blog post for scheduler plugin reusing #381

uniemimu · 2023-03-08T16:00:10Z

No description provided.

Signed-off-by: Ukri Niemimuukko <ukri.niemimuukko@intel.com>

k8s-ci-robot · 2023-03-08T16:00:19Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: uniemimu
Once this PR has been reviewed and has the lgtm label, please assign mrbobbytables for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2023-03-08T16:00:19Z

Welcome @uniemimu!

It looks like this is your first PR to kubernetes/contributor-site 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/contributor-site has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

jberkus · 2023-04-18T13:23:00Z

Apolgies! Somehow this PR never showed up in my notifications, and as such I just noticed it. Thank you so much for your patience. I will review it during/after Kubecon and move it forwards.

jberkus · 2023-05-10T19:54:03Z

This passes spelling and composition check, but I'm waiting for @kubernetes/sig-scheduling-misc review; I've reached out to the SIG.

alculquicondor · 2023-05-10T20:35:34Z

/assign @Huang-Wei

content/en/blog/2023/2023-03-08-scheduler-plugin-reusing/index.md

Signed-off-by: Ukri Niemimuukko <ukri.niemimuukko@intel.com>

jberkus · 2023-06-07T16:22:52Z

OK, this looks good, should we publish?

Huang-Wei · 2023-06-07T17:11:28Z

OK, this looks good, should we publish?

Yup,
/lgtm on scheduling side.

uniemimu · 2023-06-15T13:04:45Z

Should I squash this?

jberkus · 2023-06-16T23:59:55Z

You can if you want. I can also have Tide do it.

However, I need to confer with @sftim on how we fix the date for the blog post.

sftim

Thanks

sftim · 2023-10-24T15:57:59Z

content/en/blog/2023/2023-03-08-scheduler-plugin-reusing/index.md

+layout: blog
+title: "Scheduler plugin reusing"


For this repo

Suggested change

layout: blog

title: "Scheduler plugin reusing"

date: 2023-11-08

layout: blog

author: "Ukri Niemimuukko (Intel)"

title: "Code reuse in Kubernetes scheduler plugins"

and change the path to, eg content/en/blog/2023/scheduler-plugin-code-reuse.md

sftim · 2023-10-24T15:58:37Z

content/en/blog/2023/2023-03-08-scheduler-plugin-reusing/index.md

+title: "Scheduler plugin reusing"
+---
+
+**Author:** Ukri Niemimuukko (Intel)


Suggested change

**Author:** Ukri Niemimuukko (Intel)

This repo puts author information in front matter.

sftim · 2023-10-24T15:59:08Z

content/en/blog/2023/2023-03-08-scheduler-plugin-reusing/index.md

+
+However, the real use-case has a much more simplistic graph:
+
+{{< figure src="diagram2.svg" alt="Plugins NodeResourcesFit and TopologyScorer with connection NodeResourcesFit->TopologyScorer">}}


(nit)

Suggested change

{{< figure src="diagram2.svg" alt="Plugins NodeResourcesFit and TopologyScorer with connection NodeResourcesFit->TopologyScorer">}}

{{< figure src="diagram2.svg" alt="Plugins NodeResourcesFit and TopologyScorer with connection NodeResourcesFit → cycle state → TopologyScorer">}}

sftim · 2023-10-24T16:00:02Z

content/en/blog/2023/2023-03-08-scheduler-plugin-reusing/index.md

+
+Ideally, a new plugin could ask for getting the scores from another plugin, or multiple plugins, without rerunning them. The tricky thing is, that plugins run highly parallel in order to execute faster. Thus proper synchronization would be needed. Also it would be ideal, if no changes to existing plugins would be required, and the framework would provide for plugins in need of reusing other plugins' results.
+
+This got the author thinking, that if the `framework.CycleState` object could contain a channel for the scores, and if the framework would create such a channel when it is needed, then the parallel running plugins which are interested in score reuse, could wait for the scores to appear from such "ScoreChannels".


Suggested change

This got the author thinking, that if the `framework.CycleState` object could contain a channel for the scores, and if the framework would create such a channel when it is needed, then the parallel running plugins which are interested in score reuse, could wait for the scores to appear from such "ScoreChannels".

This got me thinking that if the `framework.CycleState` object could contain a channel for the scores, and if the framework would create such a channel when it is needed, then the parallel running plugins which are interested in score reuse could wait for the scores to appear from such `ScoreChannels`.

even better rewritten as shorter sentences (commas are not enough; a rewrite would use more periods)

sftim · 2023-10-24T16:00:42Z

content/en/blog/2023/2023-03-08-scheduler-plugin-reusing/index.md

+
+A little less than 80 framework lines of PoC-code later the now very dirty scheduler was able to pick up from the arguments of plugins, whether they wanted to use scores from other plugins, and it was creating such "ScoreChannels" into the `framework.CycleState` and wrote scores to them as requested. A couple of experimental plugins later it was possible to verify that indeed, plugin scoring graphs such as the following were entirely possible:
+
+{{< figure src="diagram1.svg" alt="Plugins A, B, C and D with connections A->B, A->C, B->D, C->D">}}


Suggested change

{{< figure src="diagram1.svg" alt="Plugins A, B, C and D with connections A->B, A->C, B->D, C->D">}}

{{< figure src="diagram1.svg" alt="Plugins A, B, C and D with connections A → B, A → C, B → D, C → D. The connections are via a Cycle State element." >}}

sftim · 2023-10-24T16:01:39Z

content/en/blog/2023/2023-03-08-scheduler-plugin-reusing/index.md

+
+## Next steps
+
+As per discussed in the sig-scheduling meetings, the topology resource scoring plugin will proceed with the more conservative composition approach, because it can get the job done without framework changes, associated risks and maintenance burden. However, if you find that it would indeed benefit your use-cases to be able to simply configure scheduler profiles in such a way that certain score plugins are connected, chime in at slack [#sig-scheduling].


Suggested change

As per discussed in the sig-scheduling meetings, the topology resource scoring plugin will proceed with the more conservative composition approach, because it can get the job done without framework changes, associated risks and maintenance burden. However, if you find that it would indeed benefit your use-cases to be able to simply configure scheduler profiles in such a way that certain score plugins are connected, chime in at slack [#sig-scheduling].

As per discussed in SIG Scheduling meetings, the topology resource scoring plugin will proceed with the more conservative composition approach, because it can get the job done without framework changes, associated risks and maintenance burden. However, if you find that it would indeed benefit your use-cases to be able to simply configure scheduler profiles in such a way that certain score plugins are connected, chime in via [`#sig-scheduling`][#sig-scheduling] on Kubernetes Slack (visit https://slack.k8s.io/ for an invitation).

sftim · 2023-10-24T16:01:57Z

content/en/blog/2023/2023-03-08-scheduler-plugin-reusing/index.md

+}
+```
+
+A little less than 80 framework lines of PoC-code later the now very dirty scheduler was able to pick up from the arguments of plugins, whether they wanted to use scores from other plugins, and it was creating such "ScoreChannels" into the `framework.CycleState` and wrote scores to them as requested. A couple of experimental plugins later it was possible to verify that indeed, plugin scoring graphs such as the following were entirely possible:


Suggested change

A little less than 80 framework lines of PoC-code later the now very dirty scheduler was able to pick up from the arguments of plugins, whether they wanted to use scores from other plugins, and it was creating such "ScoreChannels" into the `framework.CycleState` and wrote scores to them as requested. A couple of experimental plugins later it was possible to verify that indeed, plugin scoring graphs such as the following were entirely possible:

A little less than 80 framework lines of PoC-code later the now very dirty scheduler was able to pick up from the arguments of plugins, whether they wanted to use scores from other plugins, and it was creating such `ScoreChannels` into the `framework.CycleState` and wrote scores to them as requested. A couple of experimental plugins later it was possible to verify that indeed, plugin scoring graphs such as the following were entirely possible:

k8s-triage-robot · 2024-01-22T19:32:51Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

natalisucks · 2024-02-08T10:05:35Z

@uniemimu do you have capacity to address Tim's feedback?

uniemimu · 2024-02-09T13:50:53Z

@uniemimu do you have capacity to address Tim's feedback?

Obviously I haven't had, my apologies for that. Since the somewhat related KEP PR (3925) never found any interest, I'm leaning towards closing this PR without publishing. What do you think?

natalisucks · 2024-02-11T00:42:00Z

@uniemimu I think that's completely fine. If the KEP picks up traction again, this PR can always be re-opened and worked on once again. Thanks for answering!

Add blog post for scheduler plugin reusing

fed0eca

Signed-off-by: Ukri Niemimuukko <ukri.niemimuukko@intel.com>

k8s-ci-robot requested review from jberkus and jeefy March 8, 2023 16:00

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 8, 2023

k8s-ci-robot added the sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. label May 10, 2023

k8s-ci-robot assigned Huang-Wei May 10, 2023

Huang-Wei reviewed May 19, 2023

View reviewed changes

fixes for review comments

0d77f3a

Signed-off-by: Ukri Niemimuukko <ukri.niemimuukko@intel.com>

uniemimu requested a review from Huang-Wei June 1, 2023 16:38

sftim reviewed Oct 24, 2023

View reviewed changes

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 22, 2024

uniemimu closed this Feb 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add blog post for scheduler plugin reusing #381

Add blog post for scheduler plugin reusing #381

uniemimu commented Mar 8, 2023

k8s-ci-robot commented Mar 8, 2023

k8s-ci-robot commented Mar 8, 2023

jberkus commented Apr 18, 2023

jberkus commented May 10, 2023

alculquicondor commented May 10, 2023

jberkus commented Jun 7, 2023

Huang-Wei commented Jun 7, 2023

uniemimu commented Jun 15, 2023

jberkus commented Jun 16, 2023

sftim left a comment

sftim Oct 24, 2023

sftim Oct 24, 2023

sftim Oct 24, 2023

sftim Oct 24, 2023

sftim Oct 24, 2023

sftim Oct 24, 2023

sftim Oct 24, 2023

k8s-triage-robot commented Jan 22, 2024

natalisucks commented Feb 8, 2024

uniemimu commented Feb 9, 2024

natalisucks commented Feb 11, 2024


		However, the real use-case has a much more simplistic graph:

		{{< figure src="diagram2.svg" alt="Plugins NodeResourcesFit and TopologyScorer with connection NodeResourcesFit->TopologyScorer">}}


		Ideally, a new plugin could ask for getting the scores from another plugin, or multiple plugins, without rerunning them. The tricky thing is, that plugins run highly parallel in order to execute faster. Thus proper synchronization would be needed. Also it would be ideal, if no changes to existing plugins would be required, and the framework would provide for plugins in need of reusing other plugins' results.

		This got the author thinking, that if the `framework.CycleState` object could contain a channel for the scores, and if the framework would create such a channel when it is needed, then the parallel running plugins which are interested in score reuse, could wait for the scores to appear from such "ScoreChannels".

	This got the author thinking, that if the `framework.CycleState` object could contain a channel for the scores, and if the framework would create such a channel when it is needed, then the parallel running plugins which are interested in score reuse, could wait for the scores to appear from such "ScoreChannels".
	This got me thinking that if the `framework.CycleState` object could contain a channel for the scores, and if the framework would create such a channel when it is needed, then the parallel running plugins which are interested in score reuse could wait for the scores to appear from such `ScoreChannels`.


		A little less than 80 framework lines of PoC-code later the now very dirty scheduler was able to pick up from the arguments of plugins, whether they wanted to use scores from other plugins, and it was creating such "ScoreChannels" into the `framework.CycleState` and wrote scores to them as requested. A couple of experimental plugins later it was possible to verify that indeed, plugin scoring graphs such as the following were entirely possible:

		{{< figure src="diagram1.svg" alt="Plugins A, B, C and D with connections A->B, A->C, B->D, C->D">}}


		## Next steps

		As per discussed in the sig-scheduling meetings, the topology resource scoring plugin will proceed with the more conservative composition approach, because it can get the job done without framework changes, associated risks and maintenance burden. However, if you find that it would indeed benefit your use-cases to be able to simply configure scheduler profiles in such a way that certain score plugins are connected, chime in at slack [#sig-scheduling].

Add blog post for scheduler plugin reusing #381

Add blog post for scheduler plugin reusing #381

Conversation

uniemimu commented Mar 8, 2023

k8s-ci-robot commented Mar 8, 2023

k8s-ci-robot commented Mar 8, 2023

jberkus commented Apr 18, 2023

jberkus commented May 10, 2023

alculquicondor commented May 10, 2023

jberkus commented Jun 7, 2023

Huang-Wei commented Jun 7, 2023

uniemimu commented Jun 15, 2023

jberkus commented Jun 16, 2023

sftim left a comment

Choose a reason for hiding this comment

sftim Oct 24, 2023

Choose a reason for hiding this comment

sftim Oct 24, 2023

Choose a reason for hiding this comment

sftim Oct 24, 2023

Choose a reason for hiding this comment

sftim Oct 24, 2023

Choose a reason for hiding this comment

sftim Oct 24, 2023

Choose a reason for hiding this comment

sftim Oct 24, 2023

Choose a reason for hiding this comment

sftim Oct 24, 2023

Choose a reason for hiding this comment

k8s-triage-robot commented Jan 22, 2024

natalisucks commented Feb 8, 2024

uniemimu commented Feb 9, 2024

natalisucks commented Feb 11, 2024