Kubernetes Controller Development

Since January 2023 one of my tasks is developing Kubernetes Controllers and CRDs with Go.

This is not an introduction to the topic.

Basics

First, you need to learnthe basics Go: go.dev/learn

You need to learn the basics of Kubernetes: kubernetes.io

You need to learn Kubernetes Client Go. Unfortunately only same examples exist.

Learn the API Conventions

Things I learned

Here are some things I learned.

status.conditions are great.

Cluster-API has a more precise way of handling conditions. The proposal explains it: Condition Proposal

cluster-api has some nice helpers: conditions

I wrote a small tool to check all conditions of all resources in a cluster: check-conditions

CRDs

Custom Resource Definitions are a way to extend the API schema of Kubernetes. It is like adding new tables to relational database.

You should know one big drawback of CRDs: CRDs are not namespaces. This means you can only install one version of your CRD in the cluster.

You can't install version v1 in namespace "foo" and version v2 in namespace "bar".

If you just need a way to configure your application, then a ConfigMap might be enough. This has the benefit that you can run two versions of your applications in parallel.

Related: Should I use a ConfigMap or a custom resource?

If you want to create a CRD, then read the newbie friendly docs of Kubebuilder

CRDs: `make run`

If you write a CRD and your own controller, then it is likely that you will use Tilt to run your controller in a development cluster (in most cases a kind cluster on your device).

But remember: a controller is just a Kubernetes API client. You can run it anywhere you like. If you created your project with Kubebuilder, you can use make run to run the controller without any container. This makes it easy for debugging with delve.

Unfortunately, if you use web-hooks, then this will not work, except you set up a way so that the API server can reach your webhook.

CRDs: status will get lost upon backup+restore

Guideline: A controller does not read its own status.

After a backup+restore the status will be empty. The controller need to check the actual state and then create a matching status.

CRDs: no foreign keys

In a relational database you can create a foreign key, and you can be shure that the foreign key will never be broken.

I admit, that I miss this feature in the Kubernetes API.

CRDs: Webhooks: Order of restore

Imagine you have a CRD which represents a datacenter.

And you have a second CRD which represents a server.

We created a validating web hook which ensures that for a server spec.datacenter contains a string which has a corresponding datacenter.metadata.name.

The web hook works fine.

Except during a restore.

Related: https://velero.io/docs/main/troubleshooting/#known-issue-with-restoring-resources-when-admission-webhooks-are-enabled

Disabling the web hook during restore is a manual task which I would like to avoid. This is not solved yet.

Client.Get(key, r.Client.Get(ctx, key, &myresource) finds nothing?

If r.Client.Get(key, r.Client.Get(ctx, key, &myresource) does not find a resource, but you can see it with kubectl, then two things could be wrong:

RBAC
Client Cache

To check RBAC, you can use that (if your Go code can't find a secret):

kubectl describe secret \
   --as=system:serviceaccount:your-namespace-of-the-controller:your-serviceaccount \
    -n namespace-of-secret \
    your-secret

You can find the name of the service-account by looking at the pod.

Client Caching: check ctrl.NewManager() (usualy in main.go). Maybe there is some filter which makes the cache hide a lot of objects which would be visible otherwise.

Use controller-runtime Client (even for non-controllers)

Yes, you may use the controller-runtime Client, even if you do not develop a controller.

I prefer this Client to the client-Go Client.

Use controllerutil.CreateOrUpdate()

The controllerutil.CreateOrUpdate() function is very handy, if you reconcile child resources.

Example: You have CRD MyParent which creates "child" resources of type "MyChild". During the reconcile of MyParent you can use above function to ensure that the child resource is in the desired state. This detects changes, so it is not only "create child, if id does not existing yet".

Tools

Tools I use for developing with Golang and Kubernetes

vscode
kubespy Tools for observing Kubernetes resources in real time
kubectl stern Multi pod and container log tailing for Kubernetes
check-conditions my tool the check all conditions of all resources in a cluster

Related

Thomas WOL: Working out Loud

Feedback

I love feedback and I love to hear from you. Just create an issue and tell me what's on your mind.

CRDs: ownerRef will be lost upon backup+restore

vmware-tanzu/velero#4707

License

guettli/kubernetes-controller-development

ErrorLooks like something went wrong!

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Error
Looks like something went wrong!

Packages