Support for automatic import of multi-cluster platforms like Cluster API, Karmada, Clusternet etc.

[Iceber]

What would you like to be added?

clusterpedia was originally designed to provide resource querying for multiple clusters on top of a multi-cluster management platform.

The most common multicluster platforms today, such as cluster api, karmada, clusternet, etc., clusterpedia needs to be able to automatically discover these clusters and make them more useful.

We need a conversion policy to automatically create/update/delete the PediaClusters based on changes to cluster api and karmada‘s Cluster resource.

Of course we don't just support these open source products, we also need to be able to support user-developed multi-cluster platforms very well

Why is this needed?

Enables users to better connect their existing platforms to clusterpedia.

Design

updated at 2022-7-11

The core of the implementation of automatic discovery of clusters managed by a multi-cloud platform is the conversion of the CR representing the managed cluster to PediaCluster

We need to use two new custom resources:

• ClusterImportPolicy
• PediaClusterLifecycle

ClusterImportPolicy need to define the following things:

• CR types that need to be converted to PediaCluster
• Other resources referenced during the conversion
• Create(and update) templates for PediaCluster
• Conditions that trigger the creation(and deletion) of a PediaCluster

Based on the above requirements, we can show a ClusterImportPolicy example:

apiVersion: policy.clusterpedia.io/v1alpha1
kind: ClusterImportPolicy
metadata:
  name: karmada
spec:
  source:
    group: "cluster.karmada.io"
    resource: clusters
  references:
    - group: ""
      resource: secrets
      namespaceTemplate: "{{ .source.spec.secretRef.namespace }}"
      nameTemplate: "{{ .source.spec.secretRef.name }}"
      key: secret
  nameTemplate: 'karmada-{{ .source.metadata.name }}'
  template: |
    spec:
      apiserver: "{{ .source.spec.apiEndpoint }}"
      tokenData: "{{ .references.secret.data.token }}"
      caData: "{{ .references.secret.data.caBundle }}"
      syncResources:
        - group: ""
          resources:
            - "pods"
  creationCondition: |
    {{ if ne .source.spec.apiEndpoint "" }}
      {{ range .source.status.conditions }}
        {{ if eq .type "Ready" }}
          {{ if eq .status "True" }} true {{ end }}
        {{ end }}
      {{ end }}
    {{ end }}

We make extensive use of templates, which allows for more flexible syntax.

The .spec.nameTemplate is used to set the name of the converted PediaCluster

If .metadata.name is also set in .spec.template, then use .spec.nameTemplate to override it

.spec.references defines the resources involved, and the later items can refer to the previous ones.

.spec.creationCondition defines when a PediaCluster can be created. This field is required and will only be created if the value is true (case insensitive) after template processing.

.spec.deletionCondition defines when to delete PediaCluster, the default is to delete PediaCluster after source object is deleted.

.spec.template defines the resource templates for creating and updating PediaCluster

.spec.updationTemplate defines the template that will be used to update the PediaCluster using patch, so that user can choose to update only the information about the cluster autha and authz. This field is optional, if it is not set, the resource will be updated using .spec.template.

When a source object is created does not mean that the PediaCluster needs to be created.
For multi-cloud platforms, it may not be simple to successfully manage a cluster, so you can configure the timing of creating a PediaCluster via .spec.creationCondition.

A PediaCluster object will be deleted simultaneously when the source object is deleted, or the user can use .spec.deletedCondition to decide when to delete it earlier.

The ClusterImportPolicy defines how the resource type represented by .spec.source is converted to a PediaCluster object, and when a source object is created, a PediaClusterLifecycle object is created simultaneously, PediaClusterLifecycle maintains the relationship between a source object and a PediaCluster object, and decides when and how to create a PediaCluster according to the policy defined in ClusterImportPolicy.

apiVersion: policy.clusterpedia.io/v1alpha1
kind: PediaClusterLifecycle
metadata:
  name: karmada-member1
  ownerReferences:
  - apiVersion: policy.clusterpedia.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ClusterImportPolicy
    name: karmada
    uid: 8d98cfd8-6570-410f-910e-2cf3c9c79189
spec:
  creationCondition: |
    {{ if ne .source.spec.apiEndpoint "" }}
      {{ range .source.status.conditions }}
        {{ if eq .type "Ready" }}
          {{ if eq .status "True" }} true {{ end }}
        {{ end }}
      {{ end }}
    {{ end }}
  references:
  - group: ""
    key: secret
    nameTemplate: '{{ .source.spec.secretRef.name }}'
    namespaceTemplate: '{{ .source.spec.secretRef.namespace }}'
    resource: secrets
    version: v1
  source:
    group: cluster.karmada.io
    name: member1
    namespace: ""
    resource: clusters
    version: v1alpha1
  template: |
    spec:
      apiserver: "{{ .source.spec.apiEndpoint }}"
      tokenData: "{{ .references.secret.data.token }}"
      caData: "{{ .references.secret.data.caBundle }}"
      syncResources:
        - group: ""
          resources:
            - "pods"
status:
  conditions:
  - lastTransitionTime: "2022-07-07T09:49:14Z"
    message: ""
    reason: PediaClusterCreated
    status: "True"
    type: Created
  - lastTransitionTime: "2022-07-07T10:04:47Z"
    message: ""
    reason: PediaClusterUpdated
    status: "True"
    type: Updating
  references:
  - group: ""
    name: member1
    namespace: karmada-cluster
    resource: secrets
    version: v1

The .spec.source will specify the specific resource by namespace and name in the PediaClusterLifecycle.
The other fields under spec have the some content and role as ClusterImportPolicy

Note that there is no field like .spec.importPolicyRefName to reuse ClusterImportPolicy, the purpose is to simplify the relationship between ClusterImportPolicy, PediaClusterLifecycle and PediaCluster.

When the ClusterImportPolicy object is updated, it updates the PediaClusterLifecycle, which thentriggers the PediaClusterLifecycle to update the PediaCluster, which will be consistent with the PediaClusterLifecycle and not with the ClusterImportPolicy

[chaunceyjiang]

We need a conversion policy to automatically create/update/delete clusters based on changes to cluster api and karmada‘s Cluster resource.

Sorry ,i didn't understand its meaning. Can you explain it more clearly?

create/update/delete clusters

I guess you're talking about the cluster CRD of clusterpedia?

[Iceber]

@chaunceyjiang Yes, the programme is still being refined, so the descriptions will be vague.

Example, clusterpedia will automatically trigger the creation/update/deletion of PediaCluster based on the creation of CRDs representing clusters in the Cluster API, updates to the kube config related configuration, and the deletion of these CRDs

[chaunceyjiang]

Hi, I have a little question. Is this just a connection to the cluster API, karmada itself, or will it also collect status the member clusters managed by them?

[Iceber]

@chaunceyjiang Synchronize resources within these clusters that are managed by a multi-cloud platform.
The detailed design has been updated.

[Iceber]

Since ClusterImportPolicy and PediaClusterLifecycle are likely to be modified more frequently, we put them under a new group - policy.clusterpedia.io, and possibly under cluster.clusterpedia.io in the future in 1.0

[Iceber]

The authentication information obtained from the source cluster may have very high permissions, while the only permissions required by clusterpedia are actually read-only.

We can add a spec.adminKubeConfig template field, and then we create a low privilege sa for clusterpedia based on this adminKubeConfig

[Iceber]

#259 has completed the basic features, there are still some minor optimizations to be done.

Template fields support support over 70 template functions

Unimplemented features:

• More detailed field validation
• UpdationTemplate
• DeletionCondition
• Friendly kubectl get pediaclusterlifecycle and kubectl get clusterimportlifecycle printing
• adminKubeConfig
• Simplify log information

[chaunceyjiang]

I like this feature, but I have a question. Because of some permissions and security problems, I want some clusters managed by karmada not to be automatically imported into clusterpedia. How can I set ClusterImportPolicy?

[Iceber]

I like this feature, but I have a question. Because of some permissions and security problems, I want some clusters managed by karmada not to be automatically imported into clusterpedia. How can I set ClusterImportPolicy?

You can avoid the creation of these clusters by creationCondition, and because of the flexibility of the template, you can tell by name or labels, annotations or other fields.