Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
5 contributors

Users who have contributed to this file

@sttts @tallclair @liggitt @k8s-ci-robot @chuckha
790 lines (663 sloc) 31.7 KB
title authors owning-sig participating-sigs reviewers approvers editor creation-date last-updated status see-also
Vanilla CRD OpenAPI Subset: Structural Schemas
@sttts
@mbohlool
sig-api-machinery
sig-api-machinery
@liggitt
@apelisse
@mbohlool
@DirectXMan12
@lavalamp
@deads2k
TBD
2019-04-25
2019-05-09
implementable

Vanilla CRD OpenAPI subset: structural schemas

OpenAPI has the goal of describing every API possible. CRDs have the goal of presenting a consistent API to users of Kubernetes. This KEP proposes a restriction of permissible CRD OpenAPI validation schemas to match the later and to enable us add advanced CRD features like server-side pruning, defaulting, apply and potentially Protobuf support in the future.

Table of Contents

Summary

The CRD validation schemas today support nearly the whole OpenAPI v3 schema language. Here, we propose to impose a restriction called structural schema to that language, which will

  • significantly reduce difficulty in moving forward with advanced features like pruning, defaulting, server-side apply, read-only fields and protobuf in the near and medium term future
  • while keeping needed expressivity to express Kubernetes-like APIs including powerful value validations.

Schema generators (like crd-gen and openapi-gen) which recurse over API Golang types produce structural schemas naturally. Developers can enrich those structural schemas with nearly arbitrary value validation logic, usually using in-code +kubebuilder-like tags.

The restriction is an API change of the CustomResourceDefinition kind in the apiextensions.k8s.io/v1beta1 API group, but in the sense that

  • CRDs created via the v1beta1 endpoints are not restricted such that existing CRDs keep working as before,
  • but CRDs created or updated via the v1beta1 endpoints have to conform to the new restrictions if they want to make use of new features.

In the v1 CustomResourceDefinition API we will enforce structural schemas on creation and do ratcheting validation on update (allow updates with non-structural schemas if the original schema was non-structural already).

To communicate to the developer that a CRD does not follow the structural schema restriction and will therefore not benefit from future developments, we add a NonStructuralSchema condition to the CRD.

Motivation

  • OpenAPI has the goal of describing every API possible. We have the goal of presenting a consistent API to users of Kubernetes. OpenAPI therefore makes a fantastic output format able to describe anything, but a poor input format, because there's not "one correct way" to do a given thing. It is easy to accidentally (or intentionally) describe things in OpenAPI v3 that are not similar to other Kubernetes APIs and far outside of the Kuberntes API conversions.
  • OpenAPI v3 is a very open and wide spec that can define crazy things using logical operators (not(anyOf(oneof ...))), leading to complicated semantics, making advanced CRD features difficult or even impossible to implement.

Goals

  • Restrict CRD validation OpenAPI v3 schemas to give one canonical way to express objects we intend to support as Kubernetes-like types.
  • Simplify the semantics of CRD validation OpenAPI v3 schemas to make implementation of advanced feature like pruning, defaulting, server-side apply and read-only validation considerably less difficult.
  • Support embedding of existing API types into a CRD via RawExtensions or via Golang type reuse (e.g. of PodSpecs).
  • Allow a rich OpenAPI v3 validation schema for CustomResource validation, but allow us to programmatically derive the restricted spec for all other purposes.
  • Prospective goal: have a canonical mapping from a restricted schema to Golang types and Protobuf v3.

Non-Goals

  • a human readable spec and convenience to write. We focus on machine readability and clear and simple semantics.
  • model all kind of crazy JSON objects. We focus on Kubernetes-like objects with an OpenAPI schema derived via code generation (crd-gen) from Golang types.

Context

We assume the following context of OpenAPI schemas in the Kubernetes architecture and ecosystem:

Consumers of the restricted OpenAPI Schema

  • server side validation (consumes structural schema including value validation)
  • OpenAPI publishing
  • server side apply
  • pruning
  • defaulting
  • possibly in the future:
    • server side read-only update validation
    • Protobuf publishing

Producers of the restricted OpenAPI Schema

  • kube-openapi's openapi-gen mechanically producing go-openapi definitions for native types compiled into kube-apiserver
  • crd-gen/crd-schema-gen in https://github.com/kubernetes-sigs/controller-tools mechanically producing CRD validation schemas from Golang types with tags
  • a developer adding custom OpenAPI schemas for openapi-gen or crd-gen/crd-schema-gen.
  • a developer manually writing complete CRD validation schemas.

Information flow of OpenAPI schemas in Kubernetes

Criteria

To allow only a subset of features of OpenAPI v3, we need to have guidelines for what features we want to restrict and what features we need to permit. We would like to select a single canonical way in OpenAPI to represent these Kubernetes-like types:

  • non-polymorphics types: primitive types, objects, strings maps, arrays.

    Examples:

    • string maps like labels and annotations, but also map[string]int64
    • IPv4 vs IPv6 strings: same fields with different formats
    • embedded PodSpec (a non-trivial object)
  • discriminated union types (as described in KEP: Union Types).

    Examples:

    • VolumeSource, DeploymentStrategy, …
  • a very restrictive, fixed set of polymorphic types.

    Examples:

    • IntOrString
    • RawExtensions, i.e. unconstrained, embedded toplevel types
    • arbitrary JSON

Note: we explicitly accept that polymorphic types (like IntOrString) are an anti-pattern for Kubernetes-like types and we do not want to encourage its use beyond IntOrString.

As a soft criterion to validate the direction, restricted CRD OpenAPI schemas should represent a model that can also be represented with other parts of the system:

Golang types <=> OpenAPI <=> Protobuf.

Each of these is to be restricted to lead to a canonical mapping.

Proposal

We propose to define a subset of OpenAPI v3 called structural schema which

  • allows to algorithmically derive specified types and object field names
  • without the need of semantical understanding of all OpenAPI v3 constructs.

Here is an example of what we want to avoid:

properties:
  foo:
    type: string
  bar:
    minimum: 42
anyOf:
- properties:
    foo:
      pattern: abc
    bar:
      type: string
- properties:
    bar:
      type: integer
      minimum: 1
      maximum: 0

In this example, .foo is specified as string even without looking at the anyOf construct, but .bar's type is unclear without deeper semantic knowledge of anyOf and of minimum and maximum.

We want to be able to mechanically extract specified object field names and types using only the information specified in properties, items, additionalProperties and type. Other constructs like logical junctors should be usable to express complex validations, but should not be essential for the specification of the type structure of JSON values.

We reach this goal in defining structural schema by following this guiding principle:

Structural Completeness: all types and the possible object fields within a JSON object must be apparent from a structural schema without understanding the logical junctors anyOf, allOf, oneOf and not.

Formal Definition

We define a structural schema by choosing another data structure representation of JSONSchemaProps, while adding vendor extension fields for:

  • x-kubernetes-preserve-unknown-fields: true – recursive version of additionalProperties: true, influencing pruning to not prune unknown fields (compare pruning KEP for more details).
  • x-kubernetes-embedded-resource: true – marks a RawExtension field to embed a complete runtime.Object like object with TypeMeta and ObjectMeta.
  • x-kubernetes-unions: [...] as defined in KEP: Union Types
  • x-kubernetes-int-or-string: true – marks a IntOrString polymorphic type.

Observe that any validated JSONSchemaProps can be represented as StructuralSchema. Validation allows us to drop some unsupported fields (which are part of JSONSchemaProps but which we forbid in CRDs):

type StructuralSchema struct {
	Items                *StructuralSchema
	Properties           map[string]StructuralSchema

	Generic
	Extensions
	
	*ValueValidation
}

type Extensions struct {
	// x-kubernetes-unions lists unions, i.e. sets of fields which are 
	// mutual exclusive.
	XUnions []Union
	
	// x-kubernetes-preserve-unknown-fields stops the API server
	// decoding step from pruning fields which are not specified
	// in the validation schema. This affects fields recursively,
	// but switches back to normal pruning behaviour if nested
	// properties or additionalProperties are specified in the schema.
	// This can either be true or null. False is forbidden.
        XPreserveUnknownFields *bool
        
	// x-kubernetes-embedded-resource defines that the value is an
	// embedded Kubernetes runtime.Object, with TypeMeta and
	// ObjectMeta. The type must be object. It is allowed to further
	// restrict the embedded object. kind, apiVersion and metadata
	// are validated automatically. x-kubernetes-preserve-unknown-fields
	// is allowed to be true, but does not have to be if the object
	// is fully specified (up to kind, apiVersion, metadata).
        XEmbeddedResource bool
        
	// x-kubernetes-int-or-string specifies that this value is
	// either an integer or a string. If this is true, an empty
	// type is allowed and type as child of anyOf is permitted
	// if following one of the following patterns:
	//
	// 1) anyOf:
	//    - type: integer
	//    - type: string
	// 2) allOf:
	//    - anyOf:
	//      - type: integer
	//      - type: string
	//    - ... zero or more
        XIntOrString bool
}

type Union struct {
	// discriminator is the name of the discriminator field.
	Discriminator string
	// fields lists the union field names with their discriminator value.
	// This must be undefined if discriminator is empty.
	Fields map[string]string
}

type Generic struct {
	// type specifies the type of a value.
	// It can be object, array, number, integer, boolean, string.
	// It is optional only if x-kubernetes-preserve-unknown-fields 
	// or x-kubernetes-int-or-string is true.
	Type        string
	AdditionalProperties *StructuralOrBool
	
	Description string
	Title       string
        Nullable    bool
	Default     *JSON
        ReadOnly    bool
}

type ValueValidation struct {
	Format               string
	Maximum              *float64
        ExclusiveMaximum     bool
        Minimum              *float64
        ExclusiveMinimum     bool
        MaxLength            *int64
        MinLength            *int64
        Pattern              string
        MaxItems             *int64
        MinItems             *int64
        UniqueItems          bool
        MultipleOf           *float64
        Enum                 []JSON
        MaxProperties        *int64
        MinProperties        *int64
        Required             []string
        AllOf                []NestedValueValidation
        OneOf                []NestedValueValidation
        AnyOf                []NestedValueValidation
        Not                  *NestedValueValidation
}

type NestedValueValidation struct {
	ValueValidation

	Items      *NestedValueValidation
        Properties map[string]NestedValueValidation
        
        // Anything set in the following will make the scheme 
        // non-structural, with the exception of these two patterns if 
        // x-kubernetes-int-or-string is true:
        //
        // 1) anyOf:
        //    - type: integer
        //    - type: string
        // 2) allOf:
        //    - anyOf:
        //      - type: integer
        //      - type: string
        //    - <NestedValueValidation>
        ForbiddenGenerics Generic
        ForbiddenExtensions Extensions
}

Note: we do not have to change the API types to this schema right away. Take it as hypothetical data structure for the sake of definition, potentially used though during validation.

We add the following additional validations for CRDs based on the representation as StructuralSchema:

  1. structurality: both ForbiddenGenerics and ForbiddenExtensions only have zero values, with the two exceptions of (4).

  2. completeness: the schema is structurally complete

  3. RawExtension: for every schema with x-kubernetes-embedded-resource: true, type: object is set, and either properties are specified or x-kubernetes-preserve-unknown-fields: true is set.

  4. IntOrString: for x-kubernetes-int-or-string: true either type is empty under anyOf and allOf or the schema structure is either

    x-kubernetes-int-or-string: true
    anyOf:
    - type: integer
    - type: string
    ...

    or

    x-kubernetes-int-or-string: true
    allOf:
    - anyOf:
      - type: integer
      - type: string
    - ...
    ...

Structurally complete means that any specified JSON path in a schema has a specified type outside of any logical junctors (andOf,anyOf,oneOf,not; compare the guiding principle above and its example). In other words, we validate that

  • every object field (properties),
  • every array value (items) and
  • every string map value (additionalProperties)

has a non-empty type outside of logical junctors, with the possible exceptions of

  • x-kubernetes-int-or-string: true is set in the empty-type schema
  • x-kubernetes-preserve-unknown-fields: true is set in the empty-type schema.

If such a type is empty, we report the following structural schema validation error (via the NonStructuralSchema condition messages, and via real API rejection according to the API Compatibility Plan), e.g.:

.properties[foo].items.properties[bar].type must be non-empty

The additional validations have the following consequences:

  1. additionalProperties, description, title, nullable, default, readOnly and type (with the two x-kubernetes-int-or-string: true exceptions) and the new x-kubernetes-* extensions cannot be used inside of logical junctors allOf, anyOf, oneOf, not
  2. When setting StructuralSchema.ValueValidation recursively to nil in a schema, the schema becomes weaker.

The second consequence means that we can correctly validate types and the existence of fields without understanding value validations (i.e. without rejecting an object as invalid which is actually valid). This is crucial for knowingly incomplete schema algorithms as used in pruning, defaulting, apply and elsewhere.

Metadata

The metadata field at the object root is implicitly specified and validated. In order to enforce that every CustomResource is a good citizen in the API Machinery of the API server, i.e. that features like owner references, finalizers, server-side apply etc. work as designed, we do not want that CRDs restrict metadata fields other than name and generateName.

Hence, we restrict

  1. that properties[metadata] at the root of the schema does not specify anything else than type:object and the properties name and generateName, i.e. at most:
    properties:
      metadata:
        type: object
        properties:
          name: <any-schema>
          generateName: <any-schema>
          # nothing else here
        # nothing else here
      ...
    ...
  2. that metadata at the object root is not specified inside of ValueValidation, e.g.
    anyOf:
      properties:
        metadata: <any-schema>
    ...
    is forbidden, but
    anyOf:
      properties:
        embedded:
          x-kubernetes-embedded-resource: true
          properties:
            metadata: <any-schema>
    ...
    is allowed.

The restrictions do not apply to embedded resources (specified via x-kubernetes-embedded-resource) as these are not instances yet which interact with API machinery beyond pure validation.

Note, that it is discouraged to completely specify metadata for embedded resources, but only those restrictions which go beyond the implicit metadata validation induced by x-kubernetes-embedded-resource.

Unfolding the Extensions

During publishing we unfold the new x-kubernetes-* extensions:

  • x-kubernetes-int-or-string: true unfolds to
    x-kubernetes-int-or-string: true
    anyOf:
    - type: integer
    - type: string
    if ValueValidation.AnyOf is unset, otherwise to:
    x-kubernetes-int-or-string: true
    allOf:
    - anyOf:
      - type: integer
      - type: string
    - <ValueValidation.AnyOf value>
  • x-kubernetes-embedded-resource: true unfolds to
    x-kubernetes-embedded-resource: true
    properties:
      apiVersion:
        type: string
        pattern: <some-apiVersion-regex>
      kind:
        type: string
      metadata:
        type: object
        <some-further-ObjectMeta-schema>
    required:
    - kind
    - apiVersion
    if the three properties are not already specified, or an analog allOf variant logically and'ing the existing and the unfolded schema.
  • x-kubernetes-unions: [{"discriminator":"type","fields":{"foo":"Foo", "bar":"Bar"}}] unfolds to
    x-kubernetes-unions:
    - discriminator: type
      fields:
        foo: Foo
        bar: Bar
    oneOf:
      - required: ["foo"]
      - required: ["bar"]
    properties:
      type:
        type: string
        enum:
        - Foo
        - Bar
    if no oneOf is specified, the analog and'ed variant with allOf otherwise.

These unfoldings are subject to change and explicitly not part of the API, but merely illustration and a rough sketch here. Code generators and other tools must not depend on the unfolding to happen in one way or another. It is guaranteed though that the x-kubernetes-* fields are preserved during publishing and hence can be used by tooling as an abstract hint for semantics which are complex to express in OpenAPI.

The guiding principle is that we don't want OpenAPI-consuming tooling, e.g. client generators, to implement OpenAPI schema pattern heuristics to recover the original types like unions, RawExtensions or IntOrString. We support them with our abstract, officially supported vendor extensions for those cases.

Litmus Test Examples

The structural schema with these extra validations is expressive enough for the litmus test of types given in the criteria section.

Note: the kubebuilder tags are partially just sketches because the tags in crd-gen are subject to change and are heavily extended right now.

  1. discriminated union:

    type DiscriminatedVolumeSource struct {
      // +discriminator
      Type string
      // +union
      EmptyDir *EmptyDir
      // +union
      HostPath *HostPath
      // +required
      SharedField string
    }
    
    type HostPath struct {
      // +kubebuilder:pattern:"/.*"
      // +required
      Path string
    }

    is specified by

    type: object
    x-kubernetes-unions:
    - discriminator: type
      fields:
        emptyDir: EmptyDir
        hostPath: HostPath
    properties:
      type:
        type: string
        enum: ["EmptyDir","HostPath"]
      emptyDir:
        type: object
      hostPath:
        type: object
        properties:
          path:
            type: string
            pattern: "/.*"
          required: ["path"]
      sharedField:
        type: string
    required:
    - sharedField

    and published as OpenAPI v3 with the following additional schema fields (in OpenAPI v2 this cannot be expressed):

    oneOf:
    - required: ["emptyDir"]
    - required: ["hostPath"]

    After dropping the value validation, we get:

    type: object
    x-kubernetes-unions:
    - discriminator: type
      fields:
        emptyDir: EmptyDir
        hostPath: HostPath
    properties:
      type:
        type: string
      emptyDir:
        type: object
      hostPath:
        type: object
        properties:
          path:
            type: string
      sharedField:
        type: string
  2. non-polymorphic type, polymorphic format for IPv4 and IPv6:

    // +kubebuilder:anyOf:pattern="^(?:(?:25[0...$",pattern="^(([0-9a-fA-F]...$"
    type IPv4Orv6 string

    is specified by:

    type: string
    anyOf:
    - pattern: "^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}...$"
    - pattern: "(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|...$"

    and published the same way as OpenAPI v3 and as type: string in OpenAPI v2 (neither anyOf nor pattern exist in OpenAPI v2).

    After dropping the value validation, we get:

    type: string
  3. RawExtensions, i.e. unconstrained, embedded toplevel types

    type Foo struct {
      // +nullable
      // +kubebuilder:properties=apiVersion:pattern=v1
      X runtime.RawExtension
    }

    specified by:

    type: object
    nullable: true
    x-kubernetes-embedded-resource: true
    x-kubernetes-preserve-unknown-fields: true
    properties:
      apiVersion:
        pattern: v1

    and published as OpenAPI v3 with the following additional schema fields:

    properties:
      apiVersion:
        allOf:
        - type: string
          pattern: <some-apiVersion-regex>
        - pattern: v1
      kind:
        type: string
      metadata:
        type: object
        <some-further-ObjectMeta-schema>

    In OpenAPI v2 pattern does not exist and is dropped.

    After dropping the value validation, we get:

    type: object
    nullable: true
    x-kubernetes-embedded-resource: true
    x-kubernetes-preserve-unknown-fields: true
    properties:
      apiVersion:
        type: string
      kind:
        type: string
      metadata:
        type: object
        <some-further-non-value-validation-ObjectMeta-schema>
  4. embedded JSON, further specified by the user using tags:

    type Foo struct {
      // +nullable
      // +kubebuilder:type=object
      Json JSON
    }

    specified by:

    type: object
    properties:
      json:
        type: object
        x-kubernetes-preserve-unknown-fields: true
        nullable: true

    and published the same way as OpenAPI v3. For OpenAPI v2 we have to drop type: object and nullable: true as the latter does not exist in v2 and type: object is too restrictive (does not allow null as value).

    Dropping value validation does not change anything.

  5. string maps:

    // +kubebuilder:additionalProperties:minimum=42
    map[string]int

    specified by:

    type: object
    additionalProperties:
      type: integer
      minimum: 42

    and published as OpenAPI v3 and v2 the same way.

    After dropping the value validation, we get:

    type: object
    additionalProperties:
      type: integer
  6. IntOrString:

    type Foo struct {
      // +kubebuilder:pattern=abc
      // +kubebuilder:anyOf=minimum=42,maximum=50;minimum=52,maximum=60
      X *IntOrString
    }
    
    type IntOrString struct {
      Integer *int
      String *string
    } + marshallers

    specified by:

    x-kubernetes-int-or-string: true
    pattern: abc
    anyOf:
    - minimum: 42
      maximum: 50
    - minimum: 52
      maximum: 60

    or alternatively by:

    x-kubernetes-int-or-string: true
    allOf:
    - anyOf: // optionally provided and accepted
      - type: integer
      - type: string
    - pattern: abc
      anyOf:
      - minimum: 42
        maximum: 50
      - minimum: 52
        maximum: 60

    The latter is what is also published as OpenAPI v3, but as x-kubernetes-int-or-string: true in OpenAPI v2 (neither anyOf, nor pattern exists in v2).

    After dropping the value validation, we get:

    x-kubernetes-int-or-string: true

Note: the OpenAPI v2 output given above is the same we get from the OpenAPI v2 filtering in Kubernetes 1.14: that filtering in the CRD OpenAPI v2 publishing pipeline supports the unrestricted range of the CRD OpenAPI v3 language and turns it into a weaker variant understandable by kubectl 1.13 and 1.14. Hence, our changes above have no influence on the OpenAPI v2 publishing other than passing through of our new x-kubernetes-* vendor extensions.

Also note that structural schemas enforce types and properties outside of logical junctors. OpenAPI v2 publishing will preserve these and hence, structural schemas lead to a more complete OpenAPI v2 spec, client-side validation and kubectl explain output.

Design Details

Scope of necessary Changes

  • To implement this KEP we won't have to modify CustomResource validation in any way. We can keep using the go-openapi based validation algorithm.
  • To implement this KEP we won't have to modify kubectl in any way. We can though add support for the new x-kubernetes-* fields, though we will probably wait for OpenAPI v3 publishing and support in kubectl before doing that.
  • We don't have to change the CRD API types as described above in the proposal (only add the vendor extensions), but we can implement the structural schema check via a number of additional CRD validations on-top of the existing CRD API types.

API Compatibility Plan

  • When creating v1 CRDs, only structural schemas (with value-validation) are allowed.

  • When updating v1 CRDs, non-structural schemas are only allowed if the existing persisted object contained a non-structural schema already.

  • Any new feature and any feature not promoted to beta yet, will be available for structural schemas only:

    • webhook conversion:
      • rejected at CRD validation time on create and on update which sets the webhook; user gets validation error
      • pre-existing webhooks on update will stay allowed
    • publishing
      • not published at publish time; the NonStructuralSchema condition is added to CRD status with a warning
    • pruning (which will be with explicit opt-in in v1beta1)
      • rejected at CRD validation time if pruning is enabled in the CRD; user gets validation error
    • defaulting
      • rejected at CRD validation time if defaults are set in the CRD; user gets validation error
    • x-kubernetes-unions field:
      • rejected at CRD validation time if this field is set; user gets validation error

    This should create natural motivation for the community to phase out old, non-structural schemas without breaking beta compatibility guarantees.

We add a condition NonStructuralSchema to the CRD objects to reflect that information to the user about a non-structural schema. This condition is only created in case its value is True. It is removed if it turns to False due to schema updates.

Implementation Plan

For Kubernetes 1.15, must haves:

  • Implement structural-schema API validation for 1.15, but don't reject non-structural objects, only set the NonStructuralSchema condition.
  • Do not publish a CRD schema if NonStructuralSchema condition is true (pre-requisite of promoting publishing to beta).

For Kubernetes 1.15, stretch goals:

  • Do not accept webhook conversion to be set if the schemas are not structural.

    Note: we intend to bring webhook conversion to beta in 1.15. Hence, we won't block webhook conversion on this item, but indent to make it conditional on structural schemas if we are confident enough about the structural schema validation.

Totally uncommitted for 1.15, but possible as alpha:

  • pruning
  • defaulting

Test Plan

Next to unit tests, we will add integration tests with real CustomResourceDefinitions. Those will:

  • create and update CRDs with
    • structural
    • non-structural and
    • no schema, with
    • one version and
    • with multiple versions and different schema and check that the NonStructuralSchema condition is set correctly.
  • add integration and/or e2e test-cases to CRD OpenAPI publishing which make sure that publishing is not done for structural schemas. Add tests using the new x-kubernetes-* vendor extension fields.
  • add integration and/or e2e test-cases to CRD webhook conversion which make sure that webhooks are rejected on create and on update if a non-structural schema is set.

Graduation Criteria

The structural schema feature will graduate with the features which have it as a pre-requisite:

  • webhook conversion to beta will depend on a schema being structural. The feature itself does not depend on structural schemas. So promoting webhook conversion to beta with structural schema validation in place has low risk.
  • defaulting/pruning heavily depend on structural schemas. They will be the main test bed of the structural schema code.

Upgrade / Downgrade Strategy

On the API surface we add

  • a NonStructuralSchema condition. This is binary compatible during upgrade and downgrade, i.e. no special migration steps are necessary for the cluster admin.
  • a x-kubernetes-* vendor extensions to the CRD type. These will be dropped on write on downgrade, which is to be expected and accepted for new fields in beta resources.

Version Skew Strategy

  • kubectl 1.14 has to cope with published schemas created from unfolding of the x-kubernetes-* fields. The result of unfolding is a valid schema of today's CRD schema language and therefore supported by the OpenAPI v2 publishing algorithm. Hence, unfolding does not add anything new.
  • the new condition is ignored by kubectl 1.14.

Implementation History

You can’t perform that action at this time.