The controller-runtime library provides various abstractions to watch and reconcile resources in a Kubernetes cluster via CRUD (Create, Update, Delete, as well as Get and List in this case) operations. Operators use at least one controller to perform a coherent set of tasks within a cluster, usually through a combination of CRUD operations. The Operator SDK uses controller-runtime's Client interface, which provides the interface for these operations.
controller-runtime defines several interfaces used for cluster interaction:
client.Client: implementers perform CRUD operations on a Kubernetes cluster.manager.Manager: manages shared dependencies, such as Caches and Clients.reconcile.Reconciler: compares provided state with actual cluster state and updates the cluster on finding state differences using a Client.
Clients are the focus of this document. A separate document will discuss Managers.
The SDK relies on a manager.Manager to create a client.Client interface that performs Create, Update, Delete, Get, and List operations within a reconcile.Reconciler's Reconcile function. The SDK will generate code to create a Manager, which holds a Cache and a Client to be used in CRUD operations and communicate with the API server. By default a Controller's Reconciler will be populated with the Manager's Client which is a split-client.
pkg/controller/<kind>/<kind>_controller.go:
func newReconciler(mgr manager.Manager) reconcile.Reconciler {
return &ReconcileKind{client: mgr.GetClient(), scheme: mgr.GetScheme()}
}
type ReconcileKind struct {
// Populated above from a manager.Manager.
client client.Client
scheme *runtime.Scheme
}A split client reads (Get and List) from the Cache and writes (Create, Update, Delete) to the API server. Reading from the Cache significantly reduces request load on the API server; as long as the Cache is updated by the API server, read operations are eventually consistent.
An operator developer may wish to create their own Client that serves read requests(Get List) from the API server instead of the cache, for example. controller-runtime provides a constructor for Clients:
// New returns a new Client using the provided config and Options.
func New(config *rest.Config, options client.Options) (client.Client, error)client.Options allow the caller to specify how the new Client should communicate with the API server.
// Options are creation options for a Client
type Options struct {
// Scheme, if provided, will be used to map go structs to GroupVersionKinds
Scheme *runtime.Scheme
// Mapper, if provided, will be used to map GroupVersionKinds to Resources
Mapper meta.RESTMapper
}Example:
import (
"sigs.k8s.io/controller-runtime/pkg/client/config"
"sigs.k8s.io/controller-runtime/pkg/client"
)
cfg, err := config.GetConfig()
...
c, err := client.New(cfg, client.Options{})
...Note: defaults are set by client.New when Options are empty. The default scheme will have the core Kubernetes resource types registered. The caller must set a scheme that has custom operator types registered for the new Client to recognize these types.
Creating a new Client is not usually necessary nor advised, as the default Client is sufficient for most use cases.
A Reconciler implements the reconcile.Reconciler interface, which exposes the Reconcile method. Reconcilers are added to a corresponding Controller for a Kind; Reconcile is called in response to cluster or external Events, with a reconcile.Request object argument, to read and write cluster state by the Controller, and returns a reconcile.Result. SDK Reconcilers have access to a Client in order to make Kubernetes API calls.
Note: For those familiar with the SDK's old project semantics, Handle received resource events and reconciled state for multiple resource types, whereas Reconcile receives resource events and reconciles state for a single resource type.
// ReconcileKind reconciles a Kind object
type ReconcileKind struct {
// client, initialized using mgr.Client() above, is a split client
// that reads objects from the cache and writes to the apiserver
client client.Client
// scheme defines methods for serializing and deserializing API objects,
// a type registry for converting group, version, and kind information
// to and from Go schemas, and mappings between Go schemas of different
// versions. A scheme is the foundation for a versioned API and versioned
// configuration over time.
scheme *runtime.Scheme
}
// Reconcile watches for Events and reconciles cluster state with desired
// state defined in the method body.
// The Controller will requeue the Request to be processed again if an error
// is non-nil or Result.Requeue is true, otherwise upon completion it will
// remove the work from the queue.
func (r *ReconcileKind) Reconcile(request reconcile.Request) (reconcile.Result, error)Reconcile is where Controller business logic lives, i.e. where Client API calls are made via ReconcileKind.client. A client.Client implementer performs the following operations:
// Get retrieves an API object for a given object key from the Kubernetes cluster
// and stores it in obj.
func (c Client) Get(ctx context.Context, key ObjectKey, obj runtime.Object) errorNote: An ObjectKey is simply a client package alias for types.NamespacedName.
Example:
import (
"context"
"github.com/example-org/app-operator/pkg/apis/cache/v1alpha1"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
)
func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
...
app := &v1alpha1.App{}
ctx := context.TODO()
err := r.client.Get(ctx, request.NamespacedName, app)
...
}// List retrieves a list of objects for a given namespace and list options
// and stores the list in obj.
func (c Client) List(ctx context.Context, opts *ListOptions, obj runtime.Object) errorA client.ListOptions sets filters and options for a List call:
type ListOptions struct {
// LabelSelector filters results by label. Use SetLabelSelector to
// set from raw string form.
LabelSelector labels.Selector
// FieldSelector filters results by a particular field. In order
// to use this with cache-based implementations, restrict usage to
// a single field-value pair that's been added to the indexers.
FieldSelector fields.Selector
// Namespace represents the namespace to list for, or empty for
// non-namespaced objects, or to list across all namespaces.
Namespace string
// Raw represents raw ListOptions, as passed to the API server. Note
// that these may not be respected by all implementations of interface,
// and the LabelSelector and FieldSelector fields are ignored.
Raw *metav1.ListOptions
}Example:
import (
"context"
"fmt"
"k8s.io/api/core/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
)
func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
...
// Return all pods in the request namespace with a label of `app=<name>`
opts := &client.ListOptions{}
opts.SetLabelSelector(fmt.Sprintf("app=%s", request.NamespacedName.Name))
opts.InNamespace(request.NamespacedName.Namespace)
podList := &v1.PodList{}
ctx := context.TODO()
err := r.client.List(ctx, opts, podList)
...
}// Create saves the object obj in the Kubernetes cluster.
// Returns an error
func (c Client) Create(ctx context.Context, obj runtime.Object) errorExample:
import (
"context"
"k8s.io/api/apps/v1"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
)
func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
...
app := &v1.Deployment{ // Any cluster object you want to create.
...
}
ctx := context.TODO()
err := r.client.Create(ctx, app)
...
}// Update updates the given obj in the Kubernetes cluster. obj must be a
// struct pointer so that obj can be updated with the content returned
// by the API server. Update does *not* update the resource's status
// subresource
func (c Client) Update(ctx context.Context, obj runtime.Object) errorExample:
import (
"context"
"k8s.io/api/apps/v1"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
)
func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
...
dep := &v1.Deployment{}
err := r.client.Get(context.TODO(), request.NamespacedName, dep)
...
ctx := context.TODO()
dep.Spec.Selector.MatchLabels["is_running"] = "true"
err := r.client.Update(ctx, dep)
...
}When updating the status subresource from the client,
the StatusWriter must be used which can be gotten with Status()
// Status() returns a StatusWriter object that can be used to update the
// object's status subresource
func (c Client) Status() (client.StatusWriter, error)Example:
import (
"context"
cachev1alpha1 "github.com/example-inc/memcached-operator/pkg/apis/cache/v1alpha1"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
)
func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
...
mem := &cachev1alpha1.Memcached{}
err := r.client.Get(context.TODO(), request.NamespacedName, mem)
...
ctx := context.TODO()
mem.Status.Nodes = []string{"pod1", "pod2"}
err := r.client.Status().Update(ctx, mem)
...
}// Delete deletes the given obj from Kubernetes cluster.
func (c Client) Delete(ctx context.Context, obj runtime.Object, opts ...DeleteOptionFunc) errorA client.DeleteOptionFunc sets fields of client.DeleteOptions to configure a Delete call:
// DeleteOptionFunc is a function that mutates a DeleteOptions struct.
type DeleteOptionFunc func(*DeleteOptions)
type DeleteOptions struct {
// GracePeriodSeconds is the duration in seconds before the object should be
// deleted. Value must be non-negative integer. The value zero indicates
// delete immediately. If this value is nil, the default grace period for the
// specified type will be used.
GracePeriodSeconds *int64
// Preconditions must be fulfilled before a deletion is carried out. If not
// possible, a 409 Conflict status will be returned.
Preconditions *metav1.Preconditions
// PropagationPolicy determined whether and how garbage collection will be
// performed. Either this field or OrphanDependents may be set, but not both.
// The default policy is decided by the existing finalizer set in the
// metadata.finalizers and the resource-specific default policy.
// Acceptable values are: 'Orphan' - orphan the dependents; 'Background' -
// allow the garbage collector to delete the dependents in the background;
// 'Foreground' - a cascading policy that deletes all dependents in the
// foreground.
PropagationPolicy *metav1.DeletionPropagation
// Raw represents raw DeleteOptions, as passed to the API server.
Raw *metav1.DeleteOptions
}Example:
import (
"context"
"k8s.io/api/core/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
)
func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
...
pod := &v1.Pod{}
err := r.client.Get(context.TODO(), request.NamespacedName, pod)
...
ctx := context.TODO()
if pod.Status.Phase == v1.PodUnknown {
// Delete the pod after 5 seconds.
err := r.client.Delete(ctx, pod, client.GracePeriodSeconds(5))
...
}
...
}import (
"context"
"reflect"
appv1alpha1 "github.com/example-org/app-operator/pkg/apis/app/v1alpha1"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
)
type ReconcileApp struct {
client client.Client
scheme *runtime.Scheme
}
func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
// Fetch the App instance.
app := &appv1alpha1.App{}
err := r.client.Get(context.TODO(), request.NamespacedName, app)
if err != nil {
if errors.IsNotFound(err) {
return reconcile.Result{}, nil
}
return reconcile.Result{}, err
}
// Check if the deployment already exists, if not create a new deployment.
found := &appsv1.Deployment{}
err = r.client.Get(context.TODO(), types.NamespacedName{Name: app.Name, Namespace: app.Namespace}, found)
if err != nil {
if errors.IsNotFound(err) {
// Define and create a new deployment.
dep := r.deploymentForApp(app)
if err = r.client.Create(context.TODO(), dep); err != nil {
return reconcile.Result{}, err
}
return reconcile.Result{Requeue: true}, nil
} else {
return reconcile.Result{}, err
}
}
// Ensure the deployment size is the same as the spec.
size := app.Spec.Size
if *found.Spec.Replicas != size {
found.Spec.Replicas = &size
if err = r.client.Update(context.TODO(), found); err != nil {
return reconcile.Result{}, err
}
return reconcile.Result{Requeue: true}, nil
}
// Update the App status with the pod names.
// List the pods for this app's deployment.
podList := &corev1.PodList{}
labelSelector := labels.SelectorFromSet(labelsForApp(app.Name))
listOps := &client.ListOptions{Namespace: app.Namespace, LabelSelector: labelSelector}
if err = r.client.List(context.TODO(), listOps, podList); err != nil {
return reconcile.Result{}, err
}
// Update status.Nodes if needed.
podNames := getPodNames(podList.Items)
if !reflect.DeepEqual(podNames, app.Status.Nodes) {
app.Status.Nodes = podNames
if err := r.client.Status().Update(context.TODO(), app); err != nil {
return reconcile.Result{}, err
}
}
return reconcile.Result{}, nil
}
// deploymentForApp returns a app Deployment object.
func (r *ReconcileKind) deploymentForApp(m *appv1alpha1.App) *appsv1.Deployment {
lbls := labelsForApp(m.Name)
replicas := m.Spec.Size
dep := &appsv1.Deployment{
TypeMeta: metav1.TypeMeta{
APIVersion: "apps/v1",
Kind: "Deployment",
},
ObjectMeta: metav1.ObjectMeta{
Name: m.Name,
Namespace: m.Namespace,
},
Spec: appsv1.DeploymentSpec{
Replicas: &replicas,
Selector: &metav1.LabelSelector{
MatchLabels: lbls,
},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: lbls,
},
Spec: corev1.PodSpec{
Containers: []corev1.Container{{
Image: "app:alpine",
Name: "app",
Command: []string{"app", "-a=64", "-b"},
Ports: []corev1.ContainerPort{{
ContainerPort: 10000,
Name: "app",
}},
}},
},
},
},
}
// Set App instance as the owner and controller.
// NOTE: calling SetControllerReference, and setting owner references in
// general, is important as it allows deleted objects to be garbage collected.
controllerutil.SetControllerReference(m, dep, r.scheme)
return dep
}
// labelsForApp creates a simple set of labels for App.
func labelsForApp(name string) map[string]string {
return map[string]string{"app_name": "app", "app_cr": name}
}