customresources-subresources.md

Subresources for CustomResources

Authors: @nikhita, @sttts

Table of Contents

1. Abstract
2. Goals
3. Non-Goals
4. Proposed Extension of CustomResourceDefinition
   1. API Types
   2. Feature Gate
5. Semantics
   1. Validation Behavior
      1. Status
      2. Scale
   2. Status Behavior
   3. Scale Behavior
      1. Status Replicas Behavior
      2. Selector Behavior
6. Implementation Plan
7. Alternatives
   1. Scope

Abstract

CustomResourceDefinitions (CRDs) were introduced in 1.7. The objects defined by CRDs are called CustomResources (CRs). Currently, we do not provide subresources for CRs.

However, it is one of the most requested features and this proposal seeks to add /status and /scale subresources for CustomResources.

Goals

1. Support status/spec split for CustomResources:
   1. Status changes are ignored on the main resource endpoint.
   2. Support a /status subresource HTTP path for status changes.
   3. metadata.Generation is increased only on spec changes.
2. Support a /scale subresource for CustomResources.
3. Maintain backward compatibility by allowing CRDs to opt-in to enable subresources.
4. If a CustomResource is already structured using spec/status, allow it to easily transition to use the /status and /scale endpoint.
5. Work seamlessly with JSON Schema validation.

Non-Goals

1. Allow defining arbitrary subresources i.e. subresources except /status and /scale.

Proposed Extension of CustomResourceDefinition

API Types

The addition of the following external types in apiextensions.k8s.io/v1beta1 is proposed:

type CustomResourceDefinitionSpec struct {
    ...
    // SubResources describes the subresources for CustomResources
    // This field is alpha-level and should only be sent to servers that enable
    // subresources via the CurstomResourceSubResources feature gate.
    // +optional
    SubResources *CustomResourceSubResources `json:"subResources,omitempty"`
}

// CustomResourceSubResources defines the status and scale subresources for CustomResources.
type CustomResourceSubResources struct {
    // Status denotes the status subresource for CustomResources
    Status *CustomResourceSubResourceStatus `json:"status,omitempty"`
    // Scale denotes the scale subresource for CustomResources
    Scale  *CustomResourceSubResourceScale  `json:"scale,omitempty"`
}

// CustomResourceSubResourceStatus defines how to serve the HTTP path <CR Name>/status.
type CustomResourceSubResourceStatus struct {
}

// CustomResourceSubResourceScale defines how to serve the HTTP path <CR name>/scale.
type CustomResourceSubResourceScale struct {
    // required, e.g. “.spec.replicas”. Must be under `.spec`.
    // Only JSON paths without the array notation are allowed.
    SpecReplicasPath string `json:"specReplicasPath"`
    // optional, e.g. “.status.replicas”. Must be under `.status`.
    // Only JSON paths without the array notation are allowed.
    StatusReplicasPath string `json:"statusReplicasPath,omitempty"`
    // optional, e.g. “.spec.labelSelector”. Must be under `.spec`.
    // Only JSON paths without the array notation are allowed.
    LabelSelectorPath string `json:"labelSelectorPath,omitempty"`
    // ScaleGroupVersion denotes the GroupVersion of the Scale
    // object sent as the payload for /scale. It allows transition
    // to future versions easily.
    // Today only autoscaling/v1 is allowed.
    ScaleGroupVersion schema.GroupVersion `json:"groupVersion"`
}

Feature Gate

The SubResources field in CustomResourceDefinitionSpec will be gated under the CustomResourceSubResources alpha feature gate. If the gate is not open, the value of the new field within CustomResourceDefinitionSpec is dropped on creation and updates of CRDs.

Scale type

The Scale object is the payload sent over the wire for /scale. The polymorphic Scale type i.e. autoscaling/v1.Scale is used for the Scale object.

Since the GroupVersion of the Scale object is specified in CustomResourceSubResourceScale, transition to future versions (eg autoscaling/v2.Scale) can be done easily.

Note: If autoscaling/v1.Scale is deprecated, then it would be deprecated here as well.

Semantics

Validation Behavior

Status

The status endpoint of a CustomResource receives a full CR object. Changes outside of the .status subpath are ignored. For validation, the JSON Schema present in the CRD is validated only against the .status subpath.

To validate only against the schema for the .status subpath, oneOf and anyOf constructs are not allowed within the root of the schema, but only under a properties sub-schema (with this restriction, we can project a schema to a sub-path). The following is forbidden in the CRD spec:

validation:
    openAPIV3Schema:
        oneOf:
            ...

Note: The restriction for oneOf and anyOf allows us to write a projection function ProjectJSONSchema(schema *JSONSchemaProps, path []string) (*JSONSchemaProps, error) that can be used to apply a given schema for the whole object to only the sub-path .status or .spec.

Scale

Moreover, if the scale subresource is enabled:

On update, we copy the values from the Scale object into the specified paths in the CustomResource, if the path is set (StatusReplicasPath and LabelSelectorPath are optional). If StatusReplicasPath or LabelSelectorPath is not set, we validate that the value in Scale is also not specified and return an error otherwise.

On get and on update (after copying the values into the CustomResource as described above), we verify that:

• The value at the specified JSON Path SpecReplicasPath (e.g. .spec.replicas) is a non-negative integer value and is not empty.

• The value at the optional JSON Path StatusReplicasPath (e.g. .status.replicas) is an integer value if it exists (i.e. this can be empty).

• The value at the optional JSON Path LabelSelectorPath (e.g. .spec.labelSelector) is a valid label selector if it exists (i.e. this can be empty).

Note: The values at the JSON Paths specified by SpecReplicasPath, LabelSelectorPath and StatusReplicasPath are also validated with the same rules when the whole object or, in case the /status subresource is enabled, the .status sub-object is updated.

Status Behavior

If the /status subresource is enabled, the following behaviors change:

• The main resource endpoint will ignore all changes in the status subpath. (note: it will not reject requests which try to change the status, following the existing semantics of other resources).

• The .metadata.generation field is updated if and only if the value at the .spec subpath changes. Additionally, if the spec does not change, .metadata.generation is not updated.

• The /status subresource receives a full resource object, but only considers the value at the .status subpath for the update. The value at the .metadata subpath is not considered for update as decided in kubernetes/kubernetes#45539.

Both the status and the spec (and everything else if there is anything) of the object share the same key in the storage layer, i.e. the value at .metadata.resourceVersion is increased for any kind of change. There is no split of status and spec in the storage layer.

The /status endpoint supports both get and update verbs.

Scale Behavior

The number of CustomResources can be easily scaled up or down depending on the replicas field present in the .spec subpath.

Only ScaleSpec.Replicas can be written. All other values are read-only and changes will be ignored. i.e. upon updating the scale subresource, two fields are modified:

1. The replicas field is copied back from the Scale object to the main resource as specified by SpecReplicasPath in the CRD, e.g. .spec.replicas = scale.Spec.Replicas.

2. The resource version is copied back from the Scale object to the main resource before writing to the storage: .metadata.resourceVersion = scale.ResourceVersion. In other words, the scale and the CustomResource share the resource version used for optimistic concurrency. Updates with outdated resource versions are rejected with a conflict error, read requests will return the resource version of the CustomResource.

The /scale endpoint supports both get and update verbs.

Status Replicas Behavior

As only the scale.Spec.Replicas field is to be written to by the CR user, the user-provided controller (not any generic CRD controller) counts its children and then updates the controlled object by writing to the /status subresource, i.e. the scale.Status.Replicas field is read-only.

Selector Behavior

CustomResourceSubResourceScale.LabelSelectorPath is the label selector over CustomResources that should match the replicas count. The value in the Scale object is one-to-one the value from the CustomResource if the label selector is non-empty. Intentionally we do not default it to another value from the CustomResource (e.g. .spec.template.metadata.labels) as this turned out to cause trouble (e.g. in kubectl apply) and it is generally seen as a wrong approach with existing resources.

Implementation Plan

The /scale and /status subresources are mostly distinct. It is proposed to do the implementation in two phases (the order does not matter much):

1. /status subresource
2. /scale subresource

Alternatives

Scope

In this proposal we opted for an opinionated concept of subresources i.e. we restrict the subresource spec to the two very specific subresources: /status and /scale. We do not aim for a more generic subresource concept. In Kubernetes there are a number of other subresources like /log, /exec, /bind. But their semantics is much more special than /status and /scale. Hence, we decided to leave those other subresources to the domain of User provided API Server (UAS) instead of inventing a more complex subresource concept for CustomResourceDefinitions.

Note: The types do not make the addition of other subresources impossible in the future.

We also restrict the JSON path for the status and the spec within the CustomResource. We could make them definable by the user and the proposed types actually allow us to open this up in the future. For the time being we decided to be opinionated as all status and spec subobjects in existing types live under .status and .spec. Keeping this pattern imposes consistency on user provided CustomResources as well.