Tilt provider support for CAPI

[jds9090]

Most people who use CAPI provision clusters using clusterctl(https://cluster-api.sigs.k8s.io/clusterctl/overview.html). When we want to use kamaji as control plane with CAPI, we can use cluster-api-control-plane-provider-kamaji.(https://github.com/clastix/cluster-api-control-plane-provider-kamaji). But it still needs kamaji to be installed(clastix/cluster-api-control-plane-provider-kamaji#49).

Recently, from CAPI v1.5.0(https://github.com/kubernetes-sigs/cluster-api/releases/tag/v1.5.0), we can use the addon option of clusterctl and I suggest we deploy both kamaji and cluster-api-control-plane-provider-kamaji with the option.

Here is my suggestion using the addon option of clusterctl.

ubuntu@kamaji-tilt:~/.cluster-api$ clusterctl init --control-plane kamaji --infrastructure openstack --addon kamaji-core
Fetching providers
Skipping installing cert-manager as it is already installed
Installing Provider="cluster-api" Version="v1.5.0" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.5.0" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kamaji" Version="v0.3.0" TargetNamespace="kamaji-system"
Installing Provider="infrastructure-openstack" Version="v0.7.3" TargetNamespace="capo-system"
Installing Provider="addon-kamaji-core" Version="v0.0.1" TargetNamespace="kamaji-core-system"

Your management cluster has been initialized successfully!

You can now create your first workload cluster by running the following:

  clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -

ubuntu@kamaji-tilt:~/.cluster-api$ k get po -A
NAMESPACE                       NAME                                                         READY   STATUS    RESTARTS   AGE
capi-kubeadm-bootstrap-system   capi-kubeadm-bootstrap-controller-manager-7f7545c44f-w8kjs   1/1     Running   0          24s
capi-system                     capi-controller-manager-6c8c59f4-8x29l                       1/1     Running   0          25s
capo-system                     capo-controller-manager-6cb6565fd9-5jrlp                     1/1     Running   0          23s
cert-manager                    cert-manager-5b8f9b9d96-jb4qh                                1/1     Running   0          4m22s
cert-manager                    cert-manager-cainjector-54f68bfb64-z872n                     1/1     Running   0          4m22s
cert-manager                    cert-manager-webhook-f6c8487d6-d9b75                         1/1     Running   0          4m21s
kamaji-core-system              capi-kamaji-core-controller-manager-b58d8997b-59bhk          1/1     Running   0          22s
kamaji-system                   capi-kamaji-controller-manager-7bbd5c8c95-cwsdq              1/1     Running   0          24s
kube-system                     coredns-565d847f94-96xnv                                     1/1     Running   0          10m
kube-system                     coredns-565d847f94-cg5f4                                     1/1     Running   0          10m
kube-system                     etcd-kind-control-plane                                      1/1     Running   0          10m
kube-system                     kindnet-pj62f                                                1/1     Running   0          10m
kube-system                     kube-apiserver-kind-control-plane                            1/1     Running   0          10m
kube-system                     kube-controller-manager-kind-control-plane                   1/1     Running   0          10m
kube-system                     kube-proxy-9hhkx                                             1/1     Running   0          10m
kube-system                     kube-scheduler-kind-control-plane                            1/1     Running   0          10m
local-path-storage              local-path-provisioner-684f458cdd-6x4bn                      1/1     Running   0          10m

You can also create a pre-configured kind cluster (if you have not already done so) and launch your development environment, run

cd cluster-api
make tilt-up

[jds9090]

~/.cluster-api/clusterctl.yaml file example as follows:

providers:
  - name: "kamaji"
    url: "https://github.com/clastix/cluster-api-control-plane-provider-kamaji/releases/v0.3.0/control-plane-components.yaml"
    type: "ControlPlaneProvider"
  - name: "kamaji-core"
    url: "https://github.com/clastix/kamaji/releases/v0.3.x/install.yaml"
    type: "AddonProvider"

[prometherion]

Thanks for the really helpful feature request, @jds9090: TIL about CAPI addons, thanks for sharing!

Definitely a green light to set up a TILT-based dev environment scaffolding, it seems a popular project and I definitely should give it a try.

In regard to the CAPI addons, I think there's a pre-requisite issue, #292, since manifests would be required to get exposed using a GitHub release, and we're working on there.

However, just installing Kamaji is not enough since along with that a Datastore must be created, and configured properly.

The preferred way to install Kamaji is via Helm which offers hooks to automate some tasks using Kubernetes jobs, like the creation of an etcd cluster, properly configured with multi-tenancy. By default, when installing Kamaji, a default one is created, although I would suggest a more fine-grained control using the kamaji-etcd chart which offers a more robust lifecycle management (such as backup).

I'd like to start a discussion on how to address this, just sharing my thoughts.

1. Let's implement the TILT dev environment
2. Address the release of Kamaji manifests using plain YAML files exposed on the GitHub release page
   • This means we're removing the etcd manifests from those files
3. Consider if the same approach, such as CAPI Addons exposed as a single YAML manifest in the GH release, could be adopted for kamaji-etcd, although this should be tracked in its repository

Tagging for visibility @maxgio92 @ptx96 @bsctl and @dierbei who did the Go releaser implementation.

[jds9090]

Thanks for the really helpful feature request, @jds9090: TIL about CAPI addons, thanks for sharing!

Definitely a green light to set up a TILT-based dev environment scaffolding, it seems a popular project and I definitely should give it a try.

In regard to the CAPI addons, I think there's a pre-requisite issue, #292, since manifests would be required to get exposed using a GitHub release, and we're working on there.

However, just installing Kamaji is not enough since along with that a Datastore must be created, and configured properly.

The preferred way to install Kamaji is via Helm which offers hooks to automate some tasks using Kubernetes jobs, like the creation of an etcd cluster, properly configured with multi-tenancy. By default, when installing Kamaji, a default one is created, although I would suggest a more fine-grained control using the kamaji-etcd chart which offers a more robust lifecycle management (such as backup).

I'd like to start a discussion on how to address this, just sharing my thoughts.

1. Let's implement the TILT dev environment

2. Address the release of Kamaji manifests using plain YAML files exposed on the GitHub release page

   • This means we're removing the etcd manifests from those files

3. Consider if the same approach, such as CAPI Addons exposed as a single YAML manifest in the GH release, could be adopted for kamaji-etcd, although this should be tracked in its repository

Tagging for visibility @maxgio92 @ptx96 @bsctl and @dierbei who did the Go releaser implementation.

I agree with you. I also had to install datastore with this command for each tenant:

helm install kamaji-etcd clastix/kamaji-etcd --set datastore.enabled=true

In my case, I wanted tenant clusters to be deployed with their own datastore for security. Please, consider also this case because some people are more interested in this case for some specific policy of companies .

I hope you will come up with more fine-grained control.

[jds9090]

@prometherion In the view of CAPI, we can use this addon to install datastore(https://github.com/kubernetes-sigs/cluster-api-addon-provider-helm/blob/main/docs/quick-start.md) per tenant. If tenant uses the default, it may say "already installed".
Let me test it.

[jds9090]

Unfortunately, the addon-provider is not what I want to find.. this is a replacement of ClusterResourceSet(https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220712-cluster-api-addon-orchestration.md#motivation)