Provide actual schema for Kubernetes CustomResourceDefinition of TridentBackendConfig

[ergonben]

Thanks for working on NetApp Trident.

Currently, the CustomResourceDefinition for kind TridentBackendConfig has no actual schema:

schema:
  openAPIV3Schema:
    type: object # That is, anything goes.
    x-kubernetes-preserve-unknown-fields: true

As I understand, this is to accommodate for the variety of backend storage platforms.

However, it should be possible to express this variety in OpenAPI, using constructs such as anyOf, array, enum, etc.

The benefits would be a precise documentation of the config (addressing #861) and automatic validation, promoting the Kubernetes integration.

Describe the solution you'd like

A CustomResourceDefinition with a schema along these lines (obviously not ready but you get the idea):

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: tridentbackendconfigs.trident.netapp.io
spec:
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          type: object
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              type: object
              properties:
                version:
                  type: integer
                storageDriverName:
                  type: string
                backendName:
                  type: string
                managementLIF:
                  type: string
                dataLIF:
                  type: string
                svm:
                  type: string
                autoExportPolicy:
                  type: boolean
                autoExportCIDRs:
                  type: array
                  items:
                    type: string
                credentials:
                  type: object
                  properties:
                    name:
                      type: string

Describe alternatives you've considered

As an alternative, the schema could be partially refined. Meaning some fields may still be intentionally left flexible using type: object if an OpenAPI specification is not possible or too complex.

Additional context

On a tangent, having a schema would support the use of the official Terraform Kubernetes provider. Without a schema, the provider tries to replace TridentBackendConfigs on every Terraform apply operation, even if nothing has changed. Such a replacement does does not terminate (unless there are no PersistentVolumes on the TridentBackend), forcing the user to abort the apply operation, which leaves the TridentBackendConfig in state "Deleting" – clearly undesired. For more details about this symptom, see hashicorp/terraform-provider-kubernetes#1382.

More about CustomResourceDefinition.

[rdemarinis]

Hi,

I suggest the same approach for other CRDs such as TridentMirrorRelationship and TridentActionMirrorUpdate

[clintonk]

Hi, @ergonben. You are correct that the schema-less nature of TridentBackendConfig is to accommodate several backend types whose definitions are all very different. They also change over time, making any schema a moving target. Generating a schema to encompass everything seems prohibitively complicated, unless perhaps there is some automation capable of generating that. Are you aware of tooling with that level of sophistication? And then we have to worry about CRD versioning during upgrades & downgrades, even if we stick with v1 on the CRD. One approach might be to modify the CRDs via the tridentctl install --generate-custom-yaml mechanism to add an autogenerated schema.

[ergonben]

@clintonk: Thanks your explanations.

I'm not aware of such tooling but then I also haven't worked much with CRDs.

By the way, if it helps anyone, this is the Terraform resource we use for our limited use case:

# Trident's CustomResourceDefinition (CRD) for TridentBackendConfigs lacks a
# schema. Without a schema, the Kubernetes provider tries to replace
# TridentBackendConfigs on every Terraform apply operation, even if nothing has
# changed. Such a replacement does does not terminate (unless there are no
# PersistentVolumes on the TridentBackend), forcing the user to abort the apply
# operation, which leaves the TridentBackendConfig in state "Deleting", which is
# clearly undesired.
#
# Thus, we overwrite the official Trident CRD with our own, until Trident
# provides a schema (https://github.com/NetApp/trident/issues/964).
#
# Other than the schema, the following CRD is copied from the Trident source
# (https://github.com/NetApp/trident/blob/v24.10.0/cli/k8s_client/yaml_factory.go#L2066, update-worthy).
resource "kubernetes_manifest" "crd_trident_backend_config" {
  manifest = {
    apiVersion = "apiextensions.k8s.io/v1"
    kind       = "CustomResourceDefinition"
    metadata = {
      name = "tridentbackendconfigs.trident.netapp.io"
    }
    spec = {
      group = "trident.netapp.io"
      versions = [
        {
          name    = "v1"
          served  = true
          storage = true
          schema = {
            openAPIV3Schema = {
              type = "object"
              properties = {
                apiVersion = { type = "string" }
                kind       = { type = "string" }
                metadata   = { type = "object" }
                spec = {
                  type = "object"
                  properties = {
                    # For Trident's ONTAP NAS config options, see
                    # https://docs.netapp.com/us-en/trident/trident-use/ontap-nas-examples.html.
                    # We only maintain the subset of options for our use case.
                    version           = { type = "integer" }
                    storageDriverName = { type = "string" }
                    backendName       = { type = "string" }
                    managementLIF     = { type = "string" }
                    dataLIF           = { type = "string" }
                    svm               = { type = "string" }
                    autoExportPolicy  = { type = "boolean" }
                    autoExportCIDRs = {
                      type  = "array"
                      items = { type = "string" }
                    }
                    credentials = {
                      type       = "object"
                      properties = { name = { type = "string" } }
                    }
                  }
                }
              }
            }
          }
          subresources = {
            status = {}
          }
          additionalPrinterColumns = [
            # `priority = 0` dropped as otherwise Terraform does not converge.
            {
              name        = "Backend Name"
              type        = "string"
              description = "The backend name"
              jsonPath    = ".status.backendInfo.backendName"
            },
            {
              name        = "Backend UUID"
              type        = "string"
              description = "The backend UUID"
              jsonPath    = ".status.backendInfo.backendUUID"
            },
            {
              name        = "Phase"
              type        = "string"
              description = "The backend config phase"
              jsonPath    = ".status.phase"
            },
            {
              name        = "Status"
              type        = "string"
              description = "The result of the last operation"
              jsonPath    = ".status.lastOperationStatus"
            },
            {
              name        = "Storage Driver"
              type        = "string"
              description = "The storage driver type"
              priority    = 1
              jsonPath    = ".spec.storageDriverName"
            },
            {
              name        = "Deletion Policy"
              type        = "string"
              description = "The deletion policy"
              priority    = 1
              jsonPath    = ".status.deletionPolicy"
            },
          ]
        },
      ]
      scope = "Namespaced"
      names = {
        plural   = "tridentbackendconfigs"
        singular = "tridentbackendconfig"
        kind     = "TridentBackendConfig"
        shortNames = [
          "tbc",
          "tbconfig",
          "tbackendconfig",
        ]
        categories = [
          "trident",
          "trident-internal",
          "trident-external",
        ]
      }
    }
  }
}

[killerclaffey]

+1 to fix this, my k8s team was appalled by this, My lead K8s engineer sent me the following

"Having a fully built-out OpenAPI v3 schema for your Custom Resource Definition (CRD) in Kubernetes, rather than a permissive spec: {}, offers significant advantages for validation and overall API robustness. Primarily, it enables the Kubernetes API server to perform server-side validation of your custom resources. This means any object submitted to the cluster that doesn't conform to the defined schema (e.g., missing required fields, incorrect data types, invalid values) will be rejected immediately, preventing the creation of malformed or non-functional resources. This robust validation improves the reliability and stability of your custom controllers, as they can assume incoming objects adhere to a well-defined contract. Furthermore, a detailed schema provides a much better developer experience, offering clear error messages, enabling IDE auto-completion, and facilitating the generation of client libraries and documentation, ultimately making your custom API easier to use and maintain."