Even lighter weight CRD-only API Server?

[vorburger]

@smarterclayton I love this project! Long been thinking this is really at the core of all of it.

Re. https://github.com/kcp-dev/kcp#this-sounds-cool-and-i-want-to-help "feedback can have a big impact", here's a thought:

What if one wanted to build an EVEN MORE lightweight "API server" than the resource types that are currently "built in" here? It's what got me interested in playing with and poking around in @detiber's https://github.com/thetirefire, see his KubeCon presentation - before hearing about this today. That currently has ONLY customresourcedefinitions and apiservices built-in - for some scenarios, that may be all one wants.

For example, for something like what @andrewrynhard is building in https://github.com/cosi-project/runtime, see his KubeCon presentation, or in say a similar hypothetical agent for a basic KRM-inspired Machine Management CRD with a purely localhost controller running e.g. in a Static Pod, having the types currently baked-in to kcp may be too much already?

On the fine day when (eventually...) Kubernetes can be configured to run as kcp runs today, perhaps the "minimal required base resources" could be configurable? Or (better..) there simply could be separate binaries with "nothing at all" baked in (~badidea), "some base types" (~kcp), "original fat one" ( ~apiserver)?

Full disclosure: I'm only half way through reading up on this project, and just starting to learn more about what's what in space. Perhaps this is already possible using the "base apiserver library" (?) - but even just better illustrating and documenting how to really do that IMHO could have value for the community (if that turns out to be all that's "technically" required).

[imjasonh]

Thanks for your feedback! I haven't dug into those projects, but I'm definitely excited to learn more.

The badidea example is even more stripped down; CRDs seemingly must be cluster-scoped, presumably because there's no such thing as a namespace 😆 -- and RBAC support seems necessary to be able to do anything useful with it with 2+ users. From the "what's missing" slide in the api-machinery presentation, it looks like badidea + what's-missing ~= kcp, so I think we're just circling roughly the same final state. That's really encouraging validation for the idea, honestly.

[andrewrynhard]

Would be happy to chat more about this, FWIW.

[smarterclayton]

Yeah I don't think there is anything special about the types that are baked in still, other than you might actually want efficient high scale rbac / namespaces (a few other objects). Agree from a "reusable bits perspective" you want "can serve APIs" to be a bit that has some layers on top of it like "can subdivide and feel like a regular cluster". It should be possible to let the layering be natural

import (
  "k8s.io/apiserver/pkg/basic"
  "k8s.io/apiserver/pkg/apisets/core"
)

func main() {
  // simple
  api := basic.New()
  api.AllowCRDs()
  // less simple
  // i want to get namespaces!
  api.AddGroups(core.New())
  // specialized
  api.RegisterClusterScopedAdmission(&myImplementationOfConceptsUnderneathClusters{})
  // everyone starts the same
  api.Serve()
}

or whatever (massively oversimplified) to get different use cases and allow library-like composability.

[smarterclayton]

@detiber and I chatted briefly but he's out on PTO for a bit so it might get delayed. I was going to create a stub investigation in docs/investigations for "minimal kube-apiserver as reusable embeddable library" and then we can do some prep with folks over next few weeks. Sound good?

[andrewrynhard]

@detiber and I chatted briefly but he's out on PTO for a bit so it might get delayed. I was going to create a stub investigation in docs/investigations for "minimal kube-apiserver as reusable embeddable library" and then we can do some prep with folks over next few weeks. Sound good?

SGTM!

[smarterclayton]

Created a stub in #74 which we can iterate on over time.

[smarterclayton]

Ok, the stub is merged. When Jason is back we'll try to tee up a community meeting / round table where we go over use cases. In the meantime, I need to accumulate some of the historical efforts at this within Kube for reference.

[vorburger]

I've finally been able to find a little bit of time to explore this a bit further today (only actually ran kcp now).

main...vorburger:kcp-core-minimal is a first quick hack simply removing ~50 lines + imports re. if installClusterController {. What's dumb (of me), of course, is that so far I've just copy/pasted everything else.. still ~60 lines for the etcd, certs et al. set-up. Where may we move that to, for avoiding that copy/paste? And then evolve that into something like a POC of the API that Clayton outlined above... Could that, for now, move e.g. to kcp/pkg/kcp? I'm not quite sure yet how to best go about the next step.

PS: I probably can't make #96 tomorrow, but I'll try to start listening in for coming weeks, time permitting.