Supporting multiple API prefixes

[uluyol]

Introduction

Continued from #7111 and #2306 (also relevant: #3806, #6363).

There is a strong desire for kubernetes to support multiple API prefixes. This would allow us to split to the API into a more logical structure and introduce support for an experimental API prefix. Additionally, we would like to separately version different parts of the API and provide sensible scoping rules for API objects.

OpenShift serves its own objects separately from kubernetes objects using the /osapi endpoint. However, it registers its objects under the kubernetes api.Scheme (instead of a separate runtime.Scheme) and is forced to follow the kubernetes version numbers. Moreover, OpenShift's "kinds" must be unique to avoid clashing with those in kubernetes. We would like a more general approach to avoid these limitations.

Proposed Hierarchy

To achieve this, the api could have the following hierarchy: provider/apigroup/version/kind with the following properties:

• Any version/kind should be convertible to the corresponding version/kind within the same API group
• API groups within the same provider should be able to refer to objects within other groups. This is needed for things like namespaces and lists, as well as experimental API groups.

In the existing API server, a runtime.Scheme somewhat approximates a provider/apigroup, while an apiserver.APIGroupVersion corresponds to a version.

Challenges

• Much of the code assumes that only one prefix, runtime.Scheme, and related structures exist. This is especially problematic for things like namespaces.
• Referring to types in other runtime.Schemes requires that they (along with conversion and defaulting functions) be registered with the current runtime.Scheme and version.
• The master does core API-specific work (link)
• Registering new prefixes/versions/objects requires dealing with too many things (see Our API code is hard to understand #2306 and Proposal: Long term evolution of runtime.Scheme / conversion.Scheme #7111)

[uluyol]

@lavalamp @bgrant0607 @smarterclayton @derekwaynecarr

[bgrant0607]

The plugin issues are also somewhat relevant, so I'll ref them here: #991, #3201

[bgrant0607]

Pointing out a minor detail: If we add more fields to the API schema, we'll want to populate them automatically so that the user doesn't have to, in the server (#3000) and/or client.

We probably could punt on provider for now, since there would only be different values for systems that didn't use Kubernetes conventions/tooling, such as Docker, cloud providers, etc.

[bgrant0607]

Some hypothetical API groups, to test proposals against:

• end-user node-related objects: pod, pod template, secret, persistent volume claim
• end-user networking-related objects: service, route (future)
• end-user deployment/workflow objects: replication controller, daemon (future), deployment (future), job (future)
• app admin objects: namespace, limit range, resource quota, service account
• system objects: event, endpoints, component status, resource stats (future)
• infrastructure / cluster admin objects: node, persistent volume
• experimental apis

[smarterclayton]

Might want to split services off - if there are other control objects and policies around them they could potentially be big enough to be distinct (ie Openshift routes, external IPs or load balancers, affinity policies, etc). That said, they're fairly self contained.

On Jun 18, 2015, at 9:50 PM, Brian Grant notifications@github.com wrote:

Some hypothetical API groups, to test proposals against:

end-user core objects: pod, pod template, secret, persistent volume claim, service
higher-level end-user objects: replication controller, daemon (future), deployment (future), job (future)
app admin objects: namespace, limit range, resource quota, service account
system objects: event, endpoints, component status, resource stats (future)
infrastructure / cluster admin objects: node, persistent volume
experimental apis
—
Reply to this email directly or view it on GitHub.

[bgrant0607]

Re. services: Good point.

Also sort-of relevant: #1451, #1744, #2306

Worth looking at:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/api/register.go

https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/runtime/scheme.go
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/runtime/scheme_test.go
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/runtime/embedded_test.go

https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/conversion/scheme.go
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/conversion/scheme_test.go

https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/kubectl/cmd/cmd_test.go

https://github.com/openshift/origin/blob/master/pkg/image/api/v1/register.go

https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/master/master.go#L723
https://github.com/openshift/origin/blob/master/pkg/cmd/server/origin/master.go#L583

What do we need per API group?

At least api.Scheme (which is a runtime.Scheme) and the corresponding conversion.Scheme.

m.storage (master.go).

[bgrant0607]

This would benefit from #9971

[bgrant0607]

From in-person discussion: All api groups will need to respect namespaces, for objects that should be namespaced.