diff --git a/pkg/apis/postgres-operator.crunchydata.com/README.md b/pkg/apis/postgres-operator.crunchydata.com/README.md new file mode 100644 index 000000000..ef314de19 --- /dev/null +++ b/pkg/apis/postgres-operator.crunchydata.com/README.md @@ -0,0 +1,121 @@ + + +# Custom Resource Definitions + +These directories contain Go types that serve as [DTO]s for communicating with the [Kubernetes API]. +We use [controller-gen] to produce [CRD]s based on these Go types with [schemas](validation.md) that match. + +This [directory](.) contains our API Group, `postgres-operator.crunchydata.com`, and each subdirectory is a version: + +- v1beta1 is compatible with Kubernetes 1.30, OpenShift 4.14, and later +- v1 uses newer CRD features and requires Kubernetes 1.30, OpenShift 4.17, and later + +``` +pkg/apis/postgres-operator.crunchydata.com +├── v1 +└── v1beta1 +``` + +[controller-gen]: https://book.kubebuilder.io/reference/controller-gen +[CRD]: https://docs.k8s.io/tasks/extend-kubernetes/custom-resources/custom-resource-definitions +[DTO]: https://martinfowler.com/eaaCatalog/dataTransferObject.html +[Kubernetes API]: https://docs.k8s.io/concepts/overview/kubernetes-api + + +# CRD Versions + +Kubernetes organizes API resources into Groups. Each resource is represented by a Kind that can have multiple Versions. The shape of a CRD reflects this: + +```yaml +kind: CustomResourceDefinition +metadata: + name: "ideas.example.com" # {spec.plural}.{spec.group} +spec: + group: "example.com" # one group (G) + names: + kind: Idea # one kind (K) + plural: ideas # one resource (R) + singular: idea # one resource (R) + versions: # many versions (V) + - name: v1beta1 + schema: … + - name: v1 + schema: … +``` + + + +Every Kubernetes API request includes the Group, Resource, Version, and Kind of its payload and expected response. +The version affects how Kubernetes handles the request, but it does *not* affect how Kubernetes stores the result. +Every Kubernetes [object] is stored according to its Group, Resource, Namespace, and Name. + +> [!NOTE] +> - The API request URL contains the Group + Version + Resource (GVR). +> - The API request body includes the Group + Version (GV) as [`apiVersion`] and Kind (K) as `kind`. +> - [RBAC] matches on the Group + Resource (GR) of an API request. +> - The etcd key of each object contains the Group + Resource (GR), Namespace and Name. + +This allows a variety of clients to concurrently use whichever API versions they understand. +Kubernetes converts what is stored to or from the version in the API request. +This means, however, that *every* version of a resource **must** be equivalent *every other* version. + +Each CRD indicates which versions Kubernetes should accept from clients with `served=true`. +Kubernetes stores custom resource objects in the *single* version indicated with `storage=true`. + +> [!IMPORTANT] +> We use the `None` conversion strategy and [validation ratcheting](validation.md#validation-ratcheting)... + +[`apiVersion`]: https://docs.k8s.io/reference/using-api#api-groups +[object]: https://docs.k8s.io/concepts/overview/working-with-objects +[RBAC]: https://docs.k8s.io/reference/kubernetes-api/authorization-resources/role-v1 + + diff --git a/pkg/apis/postgres-operator.crunchydata.com/validation.md b/pkg/apis/postgres-operator.crunchydata.com/validation.md index 49a243d4c..92b9fa11b 100644 --- a/pkg/apis/postgres-operator.crunchydata.com/validation.md +++ b/pkg/apis/postgres-operator.crunchydata.com/validation.md @@ -4,10 +4,9 @@ # SPDX-License-Identifier: Apache-2.0 --> -# Custom Resource Definitions +# Custom Resource Definition Schemas -These directories contain Go types that serve as [DTO]s for communicating with the Kubernetes API. -We use the [controller-gen] tool to produce [CRD]s with schemas that match the Go types. +These directories contain Go types that [controller-gen] uses to generate matching [CRD] schemas. The CRD schema tells Kubernetes what fields and values are allowed in our API objects and how to handle changes to values. > [!TIP] @@ -15,7 +14,7 @@ The CRD schema tells Kubernetes what fields and values are allowed in our API ob CRD schemas are modified OpenAPI 3.0 [validation] schemas. Much of the schema defines what fields, types, and values are *allowed*. -`controller-gen` considers the [Go type] of a field and its [validation markers] for this. +`controller-gen` considers the field's [Go type] and [validation markers] for this. Kubernetes uses its own algorithm to consider and accept changes to API objects: [Server-Side Apply], SSA. CRD schemas contain non-standard attributes that affect SSA. @@ -25,9 +24,6 @@ CRD schemas contain non-standard attributes that affect SSA. [controller-gen]: https://book.kubebuilder.io/reference/controller-gen [CRD]: https://docs.k8s.io/tasks/extend-kubernetes/custom-resources/custom-resource-definitions -[DTO]: https://martinfowler.com/eaaCatalog/dataTransferObject.html -[Go type]: https://go.dev/ref/spec#Types -[Kubernetes API]: https://docs.k8s.io/concepts/overview/kubernetes-api [processing markers]: https://book.kubebuilder.io/reference/markers/crd-processing [Server-Side Apply]: https://docs.k8s.io/reference/using-api/server-side-apply [validation]: https://docs.k8s.io/tasks/extend-kubernetes/custom-resources/custom-resource-definitions#validation @@ -92,7 +88,7 @@ The `additionalProperties` property indicates that the keys are unknown; these f # CEL Rules > [!IMPORTANT] -> When possible, use [OpenAPI properties](#FIXME) rather than CEL rules. +> When possible, use [OpenAPI properties](#openapi-properties) rather than CEL rules. > The former do not affect the CRD [validation budget](#FIXME). ## Optional field syntax @@ -109,3 +105,88 @@ likewise be considered optional. The optional field syntax is only available in K8s 1.29+. [optional field marker]: https://pkg.go.dev/github.com/google/cel-go/cel#hdr-Syntax_Changes-OptionalTypes. + +## CEL Availability + +Kubernetes' capabilities with CEL are continuously expanding. +Different versions of Kubernetes have different CEL functions, syntax, and features. + +```asciidoc +:controller-tools: https://github.com/kubernetes-sigs/controller-tools/releases + +[cols=",,", options="header"] +|=== +| Kubernetes | OpenShift | `controller-gen` + +| 1.25 Beta, `CustomResourceValidationExpressions` gate +| OCP 4.12 +| link:{controller-tools}/v0.9.0[v0.9.0] has `rule` and `message` fields on the `XValidation` marker + +| 1.27 adds `messageExpression` +| OCP 4.14 +| link:{controller-tools}/v0.15.0[v0.15.0] adds `messageExpression` field to the `XValidation` marker + +| 1.28 adds `reason` and `fieldPath` +| OCP 4.15 +| link:{controller-tools}/v0.16.0[v0.16.0] adds `reason` and `fieldPath` to the `XValidation` marker + +| 1.29 GA | OCP 4.16 | + +| 1.30 enables link:#validation-ratcheting[validation ratcheting]; link:https://pr.k8s.io/123475[fixes fieldPath]… +| OCP 4.17 +| link:{controller-tools}/v0.17.3[v0.17.3] adds `optionalOldSelf` to the `XValidation` marker + +| 1.34 link:https://pr.k8s.io/132837[fixes IntOrString cost] +| ? +| link:{controller-tools}/v0.18.0[v0.18.0] allows validation on IntOrString + +| 1.35 link:https://pr.k8s.io/132798[shows values when validation fails] +| ? +| n/a + +|=== +``` + + + +Some details are missing from the Go package documentation: https://pr.k8s.io/130660 + +| CEL [libraries](https://code.k8s.io/staging/src/k8s.io/apiserver/pkg/cel/library), extensions, etc. | Kubernetes | OpenShift | +| --- | --- | --- | +| kubernetes.authz | 1.28 | +| kubernetes.authzSelectors | 1.32 | +| kubernetes.format | 1.32 | [4.18](https://github.com/openshift/kubernetes/pull/2140) | +| kubernetes.lists | 1.24 | 4.12 | +| kubernetes.net.cidr | 1.31 | [4.16](https://github.com/openshift/kubernetes/pull/1828) | +| kubernetes.net.ip | 1.31 | [4.16](https://github.com/openshift/kubernetes/pull/1828) | +| kubernetes.quantity | 1.29 | 4.16 | +| kubernetes.regex | 1.24 | 4.12 | +| kubernetes.urls | 1.24 | 4.12 | +| [cross-type numeric comparison](https://pkg.go.dev/github.com/google/cel-go/cel#CrossTypeNumericComparisons) | 1.29 | 4.16 | +| [optional types](https://pkg.go.dev/github.com/google/cel-go/cel#OptionalTypes) | 1.29 | 4.16 | +| [strings](https://pkg.go.dev/github.com/google/cel-go/ext#Strings) v0 | 1.24 | 4.12 | +| [strings](https://pkg.go.dev/github.com/google/cel-go/ext#Strings) v2 | 1.30 | 4.17 | +| [sets](https://pkg.go.dev/github.com/google/cel-go/ext#Sets) | 1.30 | 4.17 | +| [two-variable comprehension](https://pkg.go.dev/github.com/google/cel-go/ext#TwoVarComprehensions) | 1.33 | + + +# Validation Ratcheting + +> **Feature Gate:** `CRDValidationRatcheting` +> +> Enabled in Kubernetes 1.30 and GA in 1.33 (OpenShift 4.17 and ~4.20) + +[Validation ratcheting] allows update operations to succeed when unchanged fields are invalid. +This allows CRDs to add or "tighten" validation without breaking existing CR objects. + +Some schema changes are not ratcheted: + +- OpenAPI `allOf`, `oneOf`, `anyOf`, `not`; values in fields with these must be valid +- OpenAPI `required`; required fields are always required +- Removing `additionalProperties`; undefined fields are always dropped +- Adding or removing fields (names) in `properties`; undefined fields are dropped, and values in new fields must be valid +- Changes to `x-kubernetes-list-type` or `x-kubernetes-list-map-keys`; values in these fields must be valid +- Rules containing `oldSelf`; these are [transition rules] and should do their own ratcheting + +[transition rules]: https://docs.k8s.io/tasks/extend-kubernetes/custom-resources/custom-resource-definitions#transition-rules +[Validation ratcheting]: https://docs.k8s.io/tasks/extend-kubernetes/custom-resources/custom-resource-definitions#validation-ratcheting