Crd versioning with nop Conversion #63830

mbohlool · 2018-05-14T23:13:58Z

Implements Custom Resource Definition versioning according to design doc.

Note: I recreated this PR instead of #63518. Huge number of comments there broke github.

Follow-ups: #64136

@sttts @nikhita @deads2k @liggitt @lavalamp

Add CRD Versioning with NOP converter

sttts · 2018-05-15T05:36:34Z

staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/types.go

@@ -25,6 +25,9 @@ type CustomResourceDefinitionSpec struct {
 	// Group is the group this resource belongs in
 	Group string
 	// Version is the version this resource belongs in
+	// Should be always first item in Versions field if provided.
+	// Optional, but at least one of Version or Versions must be set.
+	// Deprecated: Please use `Versions`.


commented before: why do we keep this field in the internal types? Let's use conversion to recreate it it from the Versions slice

We discussed offline: this is needed because validation is done on internal types. Without Version here we would have to validate in conversion that it matches the first item in Versions.

sttts · 2018-05-15T05:39:56Z

Second commit has a misleading commit msg.

sttts · 2018-05-15T05:44:39Z

staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/conversion/nop_converter.go

+
+	unstructOut.SetUnstructuredContent(unstructIn.UnstructuredContent())
+
+	_, err := c.ConvertToVersion(unstructOut, unstructOut.GroupVersionKind().GroupVersion())


it looks wrong: we cannot expect that the out object has a group version set, can we?

We just set the content of the out from in object. if In object has gv set, the out would have it too, right?

Isn't this a noop then if input and output GV matches?

Isn't this a noop then if input and output GV matches?

yes, that's a bug. any info we expect to get from unstructOut must be retrieved before we stomp it with unstructOut.SetUnstructuredContent(unstructIn.UnstructuredContent())

if we can't determine the target group version, we have to return an error

at least in some cases, the context arg is used to pass version info, though I'm not sure how normative that is:

kubernetes/staging/src/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go

Lines 131 to 133 in d2952c0

if err := c.convertor.Convert(obj, into, c.decodeVersion); err != nil {

return nil, gvk, err

}

That line is the only one calling ObjectConverter.Convert (outside of parameter codecs). So we know pretty well what the context is there.

Isn't this a noop then if input and output GV matches?

I think the assumption (at least from UnstructuredConverter implementation) is they are the same gvk. I am calling ConvertToVersion to take care of the list case where GVKs are the same but list items may not.

sttts · 2018-05-15T05:47:43Z

staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/conversion/nop_converter.go

+func (c *nopConverter) ConvertToVersion(in runtime.Object, target runtime.GroupVersioner) (runtime.Object, error) {
+	var err error
+	// Run the converter on the list items instead of list itself
+	if meta.IsListType(in) {


this returns true as soon as there is a items part of a CR. Use a cast to UnstructuredList here instead as @lavalamp proposed before.

Oh, I should use the cast to unstructuredList here too.

sttts · 2018-05-15T05:49:09Z

staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/conversion/nop_converter.go

+}
+
+func (c *nopConverter) convertToVersion(in runtime.Object, target runtime.GroupVersioner) error {
+	if kind := in.GetObjectKind().GroupVersionKind(); !kind.Empty() {


why is it correct to do nothing in case of an empty kind?

Compare mbohlool#5.

At https://github.com/sttts/kubernetes/blob/2c1a689952ec34e3f9ecb7bcd1772c3fa35c9597/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/patch.go#L80 the patch endpoint code call the NewFunc of the CR store which is an empty Unstructure in this case. This ends up here in the converter. To not fall over, we have this if clause.

ref: #63880

#63880 has merged. The empty kind exception here can go.

sttts · 2018-05-15T05:49:37Z

staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go

@@ -30,6 +30,7 @@ import (
 	"github.com/go-openapi/validate"
 	"github.com/golang/glog"

+	"k8s.io/apiextensions-apiserver/pkg/apiserver/conversion"


import order is wrong

sttts · 2018-05-15T05:55:09Z

staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go

@@ -640,3 +683,18 @@ func (in crdStorageMap) clone() crdStorageMap {
 	}
 	return out
 }
+
+type CrdConversionRESTOptionGetter struct {


this can be private. Also it deserves a go doc.

liggitt · 2018-05-15T14:02:43Z

pkg/master/controller/crdregistration/crdregistration_controller.go

-			cast := obj.(*apiextensions.CustomResourceDefinition)
-			c.enqueueCRD(cast)
+		UpdateFunc: func(oldObj, newObj interface{}) {
+			c.enqueueCRD(oldObj.(*apiextensions.CustomResourceDefinition))


probably worth a comment why we're enqueuing both (to make sure we process removed versions correctly)

liggitt · 2018-05-15T14:04:04Z

pkg/master/controller/crdregistration/crdregistration_controller.go

+					Group:                groupVersion.Group,
+					Version:              groupVersion.Version,
+					GroupPriorityMinimum: 1000,                                        // CRDs should have relatively low priority
+					VersionPriority:      int32(100 + len(crd.Spec.Versions) - index), // CRDs should have relatively low priority


we want this to result in 100 for CRDs with exactly one version to avoid dueling old API servers. This produces 101.

also, this comment was outstanding from the previous PR:

what happens if two CRDs define the same group versions, but in opposite order?

For example:

CRD A defines group=foo, versions=v1,v2

CRD B defines group=foo, versions=v2,v1

What should the versionpriority for foo/v1 and foo/v2 be? How do we ensure the version priority for the API service is deterministic no matter which CRD happens to come first when searching?

One possibility is to keep track of the version priority calculated for a particular CRD but keep searching through all the CRDs in case any others define it with a higher priority. Highest priority is used. That keeps it deterministic, and if all CRDs for a group define the same versions in the same order, that becomes the order in discovery

I replied there for another suggestion. What if we add Priority field to CustomResourceDefinitionVersion type? If the user does not provide Priority then we can fill it for them (if that is an accepted pattern in our APIs). By having Priority field, there is a way for our users to keep order in any way they want.

What if we add Priority field to CustomResourceDefinitionVersion type?

whether it is computed or explicitly set, there is still the possibility of the two versions having flipped priorities in two CRDs. we need to make the outcome of that situation deterministic.

Let's step back a bit here:

Why can different CRD resources have different version orders?

I want to argue that we should not support this. The same way that names must not overlap between different CRD objects, neither should versions have a mismatch (= being in different order; of course it is fine that different CRDs have different version subsets, but the order should match).

Any algorithm to create a stable order is wrong for the CRD author in certain cases: the order will potentially not match his intent.

With names we postponed the verification of the not-overlapping invariant into the NamingController which sets the NamesAccepted condition. Clearly, it can only use a temporary snapshots of the CRDs it knows (fromt the lister), so we don't have consistency there as we don't have transactions.

I want to argue further that version order is far less critical than names. Hence:

Why don't we add a VersionsAccepted condition from a VersionsController and enforce matching version orders?

I'm fine with the controller as long as it's HA safe...

It is as safe as the existing naming controller (i.e. there is a race in HA). The assumption is that CRDs are statically created by developers. Even if in an HA environment it can go wrong (= a conflict is not detected and the new CRD versions are served) in 1 out of 100 cases, it will be noticed in 99 cases, and the developer will fix the order.

On the other hand, an algorithm will silently compute some order (which does not match the intend) and the developer of the API group (assuming it is one owner of the group) will not notice. I am not sure I want to trade HA consistency with having a behaviour that the user expects and understands.

Stefan's makes a good point about a stable order algorithm leading to action at a distance. If you modify your CRD with another version and create a conflict, the stable ordering algorithm may make what you're looking at succeed while breaking something else.

By having a proper condition for it, the action happens for your change. You don't see what you expect, you check status and you see why.

It'll be fairly hard to take back the choice and fix behavior later (see TPR to CRD). I think we should learn from the naming problems and take the proper status an reconciliation approach.

liggitt · 2018-05-15T14:08:03Z

pkg/master/controller/crdregistration/crdregistration_controller_test.go

@@ -56,7 +64,7 @@ func TestHandleVersionUpdate(t *testing.T) {
 						Group:                "group.com",
 						Version:              "v1",
 						GroupPriorityMinimum: 1000,
-						VersionPriority:      100,
+						VersionPriority:      101,


this is demonstrating a change that is going to fight old API servers

liggitt · 2018-05-15T14:45:28Z

staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/conversion/nop_converter.go

+			// TODO: should this be a typed error?
+			return fmt.Errorf("%v is unstructured and is not suitable for converting to %q", kind, target)
+		}
+		if !apiextensions.HasCRDVersion(c.crd, gvk.Version) {


a general pattern I expected to see was to do "expensive" work when setting up the convertor/handler in response to CRD changes, rather than in the actual Convert/Handle methods.

for example:

// NewCRDConverter returns a new CRD converter based on the conversion settings in crd object. func NewCRDConverter(crd *apiextensions.CustomResourceDefinition) runtime.ObjectConvertor { validVersions := map[schema.GroupVersion]bool{} for _, version := range crd.Spec.Versions { validVersions[schema.GroupVersion{Group:crd.Spec.Group, Version: version.Name}] = true } return &nopConverter{ clusterScoped: crd.Spec.Scope == apiextensions.ClusterScoped, validVersions: validVersions, } }

and in the actual noop conversion:

inGV := in.GroupVersionKind().GroupVersion() if _, ok := c.validVersions[inGV]; !ok { ... return unknown version error ... } outGV := out.GroupVersionKind().GroupVersion() if _, ok := c.validVersions[outGV]; !ok { ... return unknown version error ... } ... set content and stomp version ... return nil

it looks like the handler setup was close to that already (set up maps from versions to storages, scopes, etc), which is good.

I think the setup is already like that. This function call apiextensions.HasCRDVersion(c.crd, gvk.Version) is not expensive and also replaced another check existed before. We need to do that because the handler is for all CRDs with all versions. When we pass this, in the getCrdInfo function, we have the map and that map is pre-populated with all valid versions.

structuring the convertor so that all in/out transformations are determined at construction time makes for O(1) lookups in Convert(), and as convertors become more complex (declarative transformation, webhook, etc), we'll only want to do that complex setup (building the transforming functions, the webhook client, etc) once at construction as well.

Oh, I see. I assumed this line is in handler but it is in nop_converter. My bad. It make sense now.

we do not check outGV anywhere in nop_converter

we should. the convertor's job is to convert from inGV to outGV... making sure outGV is in the list of valid versions is part of that.

liggitt · 2018-05-15T14:46:12Z

staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/conversion/nop_converter.go

+		if !apiextensions.HasCRDVersion(c.crd, gvk.Version) {
+			return fmt.Errorf("request to convert CRD to an invalid version: %s", gvk.String())
+		}
+		in.GetObjectKind().SetGroupVersionKind(gvk)


why are we changing the GVK of the incoming object?

ObjectConverter.ConvertToVersion comment says:

// This method does not guarantee that the in object is not mutated.

leaving the door open to mutate in object and return it. Do you think it is better to duplicate it? We do not have ObjectCreator interface in converter but we can pass it here if we need to.

This is the no-op conversion. Its only task is to set the GVK of an Unstructured. Both Convert and ConvetToVersion allow mutation of the incoming object.

oh, I misread the diff and thought this was part of the Convert(in, out runtime.Object) method and was confused why we were messing with in, not out. this is fine

liggitt · 2018-05-15T14:51:06Z

staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_discovery_controller.go

@@ -96,7 +96,21 @@ func (c *DiscoveryController) sync(version schema.GroupVersion) error {
 			Version:      crd.Spec.Version,
 		})

-		if crd.Spec.Version != version.Version {
+		foundThisVersion := false
+		for _, v := range crd.Spec.Versions {


if we're iterating over and adding all served versions, we can drop the append on line 94 of spec.Version, right?

liggitt · 2018-05-15T14:53:38Z

staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_discovery_controller.go

+			}
+		}
+
+		if !foundThisVersion {
 			continue
 		}
 		foundVersion = true


with multiple versions supported, this statement is no longer true:

Versions: apiVersionsForDiscovery,
// the preferred versions for a group is arbitrary since there cannot be duplicate resources
PreferredVersion: apiVersionsForDiscovery[0],

the ordering of apiVersionsForDiscovery needs to reflect the ordering of the versions in the CRDs for the group (and the open question about conflicting order needs resolving).

liggitt · 2018-05-15T14:57:01Z

staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_discovery_controller.go

-	c.queue.Add(schema.GroupVersion{Group: obj.Spec.Group, Version: obj.Spec.Version})
+	for _, v := range obj.Spec.Versions {
+		c.queue.Add(schema.GroupVersion{Group: obj.Spec.Group, Version: v.Name})
+	}
 }

 func (c *DiscoveryController) addCustomResourceDefinition(obj interface{}) {


same change needed to enqueue both old and new in updateCustomResourceDefinition?

This give the converter a chance to convert list items to the requested version.

liggitt · 2018-05-22T20:59:36Z

/lgtm

k8s-ci-robot · 2018-05-22T20:59:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, mbohlool, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~api/OWNERS~~ [liggitt]
~~pkg/master/OWNERS~~ [sttts]
~~staging/src/k8s.io/api/Godeps/OWNERS~~ [liggitt,sttts]
~~staging/src/k8s.io/apiextensions-apiserver/OWNERS~~ [sttts]
~~staging/src/k8s.io/apimachinery/pkg/OWNERS~~ [liggitt,sttts]
~~staging/src/k8s.io/metrics/Godeps/OWNERS~~ [sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mbohlool · 2018-05-22T22:18:45Z

/retest

lavalamp · 2018-05-22T23:18:18Z

/kind feature /sig api-machinery

…

On Tue, May 22, 2018 at 4:16 PM Kubernetes Submit Queue < ***@***.***> wrote: [MILESTONENOTIFIER] Milestone Pull Request Labels *Incomplete* @liggitt <https://github.com/liggitt> @mbohlool <https://github.com/mbohlool> *Action required*: This pull request requires label changes. If the required changes are not made within 3 days, the pull request will be moved out of the v1.11 milestone. *kind*: Must specify exactly one of kind/bug, kind/cleanup or kind/feature . *sig owner*: Must specify at least one label prefixed with sig/. Help - Additional instructions <https://git.k8s.io/sig-release/ephemera/issues.md> - Commands for setting labels <https://go.k8s.io/bot-commands> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#63830 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAnglrs2sPVS4SsMh4-RcpeC4evQPNwCks5t1JxWgaJpZM4T-p7I> .

lavalamp · 2018-05-22T23:23:13Z

/status approved-for-milestone

…

On Tue, May 22, 2018 at 4:20 PM Kubernetes Submit Queue < ***@***.***> wrote: [MILESTONENOTIFIER] Milestone Pull Request *Needs Approval* @liggitt <https://github.com/liggitt> @mbohlool <https://github.com/mbohlool> @kubernetes/sig-api-machinery-misc <https://github.com/orgs/kubernetes/teams/sig-api-machinery-misc> *Action required*: This pull request must have the status/approved-for-milestone label applied by a SIG maintainer. If the label is not applied within 7 days, the pull request will be moved out of the v1.11 milestone. Pull Request Labels - sig/api-machinery: Pull Request will be escalated to these SIGs if needed. - priority/important-soon: Escalate to the pull request owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. - kind/feature: New functionality. Help - Additional instructions <https://git.k8s.io/sig-release/ephemera/issues.md> - Commands for setting labels <https://go.k8s.io/bot-commands> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#63830 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAnglilPxQuOVc0S5g-cNFSLgagSubvAks5t1J1JgaJpZM4T-p7I> .

k8s-github-robot · 2018-05-22T23:23:51Z

[MILESTONENOTIFIER] Milestone Pull Request: Up-to-date for process

@liggitt @mbohlool

Pull Request Labels

sig/api-machinery: Pull Request will be escalated to these SIGs if needed.
priority/important-soon: Escalate to the pull request owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
kind/feature: New functionality.

Help

k8s-github-robot · 2018-05-23T05:14:45Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-05-23T06:11:55Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot requested review from dchen1107, deads2k, nikhiljindal, sttts, wojtek-t and zmerlynn May 14, 2018 23:14

mbohlool mentioned this pull request May 14, 2018

CRD Versioning with no-op Conversion #63518

Closed

lavalamp mentioned this pull request May 15, 2018

WIP - apiextensions: low-level converter wiring – towards versioning #63802

Closed

kubernetes deleted a comment from k8s-ci-robot May 15, 2018

mbohlool force-pushed the crd_versioning_nop branch 2 times, most recently from aee96d2 to 928e86c Compare May 15, 2018 04:52

kubernetes deleted a comment from k8s-ci-robot May 15, 2018

sttts reviewed May 15, 2018

View reviewed changes

liggitt reviewed May 15, 2018

View reviewed changes

mbohlool added 4 commits May 22, 2018 13:54

CRD versioning validation and defaulting

8a39e53

Do not bypass same version unstructed conversion if it is a list

531041c

This give the converter a chance to convert list items to the requested version.

CRD versioning with no-op converter

0f6d98a

Generated files

c25514a

mbohlool force-pushed the crd_versioning_nop branch from 2b7d8db to c25514a Compare May 22, 2018 20:54

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 22, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 22, 2018

mbohlool added this to the v1.11 milestone May 22, 2018

mbohlool added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label May 22, 2018

k8s-github-robot added the milestone/incomplete-labels label May 22, 2018

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. labels May 22, 2018

k8s-github-robot added milestone/needs-approval and removed milestone/incomplete-labels labels May 22, 2018

k8s-ci-robot added the status/approved-for-milestone label May 22, 2018

k8s-github-robot removed the milestone/needs-approval label May 22, 2018

k8s-github-robot merged commit 45c94a1 into kubernetes:master May 23, 2018

This was referenced Jun 4, 2018

Multiple CRD APIVersions and migration #57316

Closed

Umbrella issue for CRDs moving to GA #58682

Closed

mbohlool mentioned this pull request Jun 7, 2018

Custom Resource Definition Versioning kubernetes/enhancements#544

Closed

ayj mentioned this pull request Jun 27, 2018

Move networking APIs to beta1 istio/istio#6613

Closed

liggitt mentioned this pull request Aug 24, 2018

CRD webhook conversion #67006

Merged


		unstructOut.SetUnstructuredContent(unstructIn.UnstructuredContent())

		_, err := c.ConvertToVersion(unstructOut, unstructOut.GroupVersionKind().GroupVersion())

	if err := c.convertor.Convert(obj, into, c.decodeVersion); err != nil {
	return nil, gvk, err
	}

Crd versioning with nop Conversion #63830

Crd versioning with nop Conversion #63830

Conversation

mbohlool commented May 14, 2018 • edited by sttts

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts commented May 15, 2018

sttts May 15, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt May 15, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbohlool May 15, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbohlool May 15, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt May 15, 2018 • edited

Choose a reason for hiding this comment

liggitt commented May 22, 2018

k8s-ci-robot commented May 22, 2018

mbohlool commented May 22, 2018

lavalamp commented May 22, 2018 via email

lavalamp commented May 22, 2018 via email

k8s-github-robot commented May 22, 2018

k8s-github-robot commented May 23, 2018

k8s-github-robot commented May 23, 2018

mbohlool commented May 14, 2018 •

edited by sttts

sttts May 15, 2018 •

edited

liggitt May 15, 2018 •

edited

mbohlool May 15, 2018 •

edited

mbohlool May 15, 2018 •

edited

liggitt May 15, 2018 •

edited