Decide on how to handle out-of-tree plugins

[knelasevero]

There are a few ways to do this. One way is to follow the kube-scheduler path and just have a new repo hosting all custom plugins. This can bring some problems related to having to maintain everything in that new repo and could be hard to coordinate ownership.

Wanted to raise this issue so we can discuss other options:

• have the same approach as kube-scheduler and host all custom plugins in a new repo
  • this implies building a custom image from scratch when an user wants a plugin included
• have a way to let people create their own repos, so we don't have to maintain a big repo with all custom plugins
  • still implies building a custom image from scratch when an user wants a plugin included
• have plugin over grpc approach and the plugin would be another binary that would allow us to make grpc calls to it
  • similar to https://secrets-store-csi-driver.sigs.k8s.io/providers.html#implementing-a-provider-for-secrets-store-csi-driver
  • or https://github.com/hashicorp/go-plugin
• follow discussions that are happening on the kube-scheduler side and try to use whatever is decided there:
  • Explore go buildmode to simplfy building custom scheduler binary kubernetes/kubernetes#100723
  • support to load custom plugins with plugin package so that users can use custom plugins without building custom scheduler kubernetes/kubernetes#106705
  • Scheduler: introduce WASM runtime to scheduler and load scheduler plugins via WASM runtime kubernetes/kubernetes#112851
  • https://kubernetes.slack.com/archives/C09TP78DV/p1676610114431299

EDIT:

• just remembered another option. In ESO we have a bunch of providers/plugins in-tree, but we classify them as internally maintained when we think maintainers would want to personally give support to them, and as community maintained (with names of people that would be contacts so people can ping them) when we don't worry too much about e2e tests and support -> ESO stability support. There we still think about having out-of-tree plugins, but I think it is worth to have a look.

.

• Also another option:
  • have a combination of above. Having a way to guarantee that an image has everything that is supported is useful, in enterprise situations. Images with supported plugins pre-baked helps service providers to be able to quickly say: 'we support that' or 'we don't support that'. Then thinking about both enterprise compliance and usability: the runtime plugins as options and the custom image with plugins inside as the recommended way scratches both itches.

[knelasevero]

@damemi based on our discussion

[binacs]

/cc @binacs

[damemi]

Mentioned this offline to @knelasevero, but I think the "shared repo of third-party plugins" approach brings a lot of maintenance overhead and defeats the purpose of a development framework.

The scheduler-plugins repo is essentially another repo for the SIG to maintain. For that, the concept of "out-of-tree" applies better because the "in-tree" alternative refers to code living in the main Kubernetes project repo, which is much larger and slower. For us, an out-of-tree repo would be pretty much just a 2nd descheduler repo.

(For this reason, I think it's better to refer to our plugins as "first-party"/"third-party" since we don't have much of a tree to begin with.)

Code ownership becomes a problem too. As developers contribute third-party plugins to a centralized repo, there's nothing binding them to owning that code forever. People change jobs and companies change priorities, which leaves the SIG in the difficult position of enforcing deprecation labels on abandoned plugins.

I think a much more lightweight approach would be to have a simple index repo (similar to how Krew indexes third-party kubectl plugins and how Operatorhub lets people contribute operators). Then there is basically 0 maintenance for the SIG over third-party plugins. Users can host and maintain them at their own pace, and descheduler maintainers don't have to be gatekeepers reviewing every new plugin.

If an index approach is done right, it can be pretty simple to provide Go code generators to build a descheduler based on the indexed plugins. That lets us provide an image people can grab if that's all they want.

[knelasevero]

Where can I see this Kueue index? Do you mean Krew?

[damemi]

@knelasevero ah yes, sorry! I meant krew https://github.com/kubernetes-sigs/krew-index

[a7i]

linking another PR: #1144

[knelasevero]

Hey

https://github.com/kubernetes-sigs/kube-scheduler-wasm-extension

Kube-scheduler is investigating using wasm to extend functionality. I think we could do the same (investigate).

kubernetes/kubernetes#112851 (comment)

We can monitor it and see how it goes.

[damemi]

Looking toward next steps in the framework, I'd like to unblock this. Looking back on it, my previous comment pushing for an index-style plugin repo is maybe a bit lofty for our use case. At a large scale, that makes more sense. But a lot of the issues I was concerned with can be managed.

Kube-scheduler does provide a good precedent for us to follow, even if we don't have to do everything the same way as that repo. So I am +1 for taking that approach. This will be the fastest way for us to unblock more feature development and promote the framework with examples to users.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pravarag]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[binacs]

/remove-lifecycle stale