Skip to content
An end-to-end scenario showing how to use the Operator Framework.
Branch: master
Clone or download
AlexNPavel Merge pull request #27 from AlexNPavel/ci
.travis.yml,hack/ci: add markdown link checker
Latest commit 7bf6aa1 Nov 16, 2018
Type Name Latest commit message Commit time
Failed to load latest commit information.
hack/ci .travis.yml,hack/ci: add markdown link checker Nov 5, 2018
memcachedoperator.0.0.1.csv.yaml Update apiVersion to Sep 19, 2018

Getting Started

The Operator Framework (intro blog post) is an open source toolkit to manage Kubernetes native applications, called operators, in an effective, automated, and scalable way. Operators take advantage of Kubernetes's extensibility to deliver the automation advantages of cloud services like provisioning, scaling, and backup/restore while being able to run anywhere that Kubernetes can run.

This guide shows how to build a simple memcached operator and how to manage its lifecycle from install to update to a new version. For that, we will use two center pieces of the framework:

  • Operator SDK: Allows your developers to build an operator based on your expertise without requiring knowledge of Kubernetes API complexities.
  • Lifecycle Manager: Helps you to install, update, and generally manage the lifecycle of all of the operators (and their associated services) running across your clusters.

Build an operator using the Operator SDK

The Operator SDK makes it easier to build Kubernetes native applications, a process that can require deep, application-specific operational knowledge. The SDK not only lowers that barrier, but it also helps reduce the amount of boilerplate code needed for many common management capabilities, such as metering or monitoring.

This section walks through an example of building a simple memcached operator using tools and libraries provided by the Operator SDK. This walkthrough is not exhaustive; for an in-depth explanation of these steps, see the SDK's user guide.

Requirements: Please make sure that the Operator SDK is installed on the development machine. Additionally, the Operator Lifecycle Manager must be installed in the cluster (1.8 or above to support the apps/v1beta2 API group) before running this guide.

Create a new project

Use the CLI to create a new memcached-operator project:

$ mkdir -p $GOPATH/src/
$ cd $GOPATH/src/
$ operator-sdk new memcached-operator
Create cmd/manager/main.go
Run dep ensure ...
Run dep ensure done
Run git init ...
Run git init done
$ cd memcached-operator

This creates the memcached-operator project.

Learn more about the project directory structure from the SDK project layout documentation.


The main program for the operator cmd/manager/main.go initializes and runs the Manager.

The Manager will automatically register the scheme for all custom resources defined under pkg/apis/... and run all controllers under pkg/controller/....

The Manager can restrict the namespace that all controllers will watch for resources:

mgr, err := manager.New(cfg, manager.Options{Namespace: namespace})

By default this will be the namespace that the operator is running in. To watch all namespaces leave the namespace option empty:

mgr, err := manager.New(cfg, manager.Options{Namespace: ""})

Add a new Custom Resource Definition

Add a new Custom Resource Definition (CRD) API called Memcached, with APIVersion and Kind Memcached.

$ operator-sdk add api --kind=Memcached

This will scaffold the Memcached resource API under pkg/apis/cache/v1alpha1/....

Define the Memcached spec and status

Modify the spec and status of the Memcached Custom Resource (CR) at pkg/apis/cache/v1alpha1/memcached_types.go:

type MemcachedSpec struct {
    // Size is the size of the memcached deployment
    Size int32 `json:"size"`
type MemcachedStatus struct {
    // Nodes are the names of the memcached pods
    Nodes []string `json:"nodes"`

After modifying the *_types.go file always run the following command to update the generated code for that resource type:

$ operator-sdk generate k8s

Add a new Controller

Add a new Controller to the project that will watch and reconcile the Memcached resource:

$ operator-sdk add controller --kind=Memcached

This will scaffold a new Controller implementation under pkg/controller/memcached/....

For this example replace the generated Controller file pkg/controller/memcached/memcached_controller.go with the example memcached_controller.go implementation.

The example Controller executes the following reconciliation logic for each Memcached CR:

  • Create a memcached Deployment if it doesn't exist
  • Ensure that the Deployment size is the same as specified by the Memcached CR spec
  • Update the Memcached CR status with the names of the memcached pods

The next two subsections explain how the Controller watches resources and how the reconcile loop is triggered. Skip to the Build section to see how to build and run the operator.

Resources watched by the Controller

Inspect the Controller implementation at pkg/controller/memcached/memcached_controller.go to see how the Controller watches resources.

The first watch is for the Memcached type as the primary resource. For each Add/Update/Delete event the reconcile loop will be sent a reconcile Request (a namespace/name key) for that Memcached object:

err := c.Watch(
    &source.Kind{Type: &cachev1alpha1.Memcached{}},

The next watch is for Deployments but the event handler will map each event to a reconcile Request for the owner of the Deployment. Which in this case is the Memcached object for which the Deployment was created. This allows the controller to watch Deployments as a secondary resource.

err := c.Watch(
    &source.Kind{Type: &appsv1.Deployment{}},
        IsController: true,
        OwnerType:    &cachev1alpha1.Memcached{}},

Reconcile loop

Every Controller has a Reconciler object with a Reconcile() method that implements the reconcile loop. The reconcile loop is passed the Request argument which is a Namespace/Name key used to lookup the primary resource object, Memcached, from the cache:

func (r *ReconcileMemcached) Reconcile(request reconcile.Request) (reconcile.Result, error) {
    // Lookup the Memcached instance for this reconcile request
    memcached := &cachev1alpha1.Memcached{}
    err := r.client.Get(context.TODO(), request.NamespacedName, memcached)

For a guide on Reconcilers, Clients, and interacting with resource Events, see the Client API doc.

Build and run the operator

Before running the operator, the CRD must be registered with the Kubernetes apiserver:

$ kubectl create -f deploy/crds/cache_v1alpha1_memcached_crd.yaml

Once this is done, there are two ways to run the operator:

  • As a Deployment inside a Kubernetes cluster
  • As Go program outside a cluster

1. Run as a Deployment inside the cluster

Build the memcached-operator image and push it to a registry:

$ operator-sdk build
$ sed -i 's|REPLACE_IMAGE||g' deploy/operator.yaml
# On OSX use:
$ sed -i "" 's|REPLACE_IMAGE||g' deploy/operator.yaml
$ docker push

The Deployment manifest is generated at deploy/operator.yaml. Be sure to update the deployment image as shown above since the default is just a placeholder.

Setup RBAC and deploy the memcached-operator:

$ kubectl create -f deploy/service_account.yaml
$ kubectl create -f deploy/role.yaml
$ kubectl create -f deploy/role_binding.yaml
$ kubectl create -f deploy/operator.yaml

Verify that the memcached-operator is up and running:

$ kubectl get deployment
memcached-operator       1         1         1            1           1m

2. Run locally outside the cluster

This method is preferred during development cycle to deploy and test faster.

Set the name of the operator in an environment variable:

export OPERATOR_NAME=memcached-operator

Run the operator locally with the default kubernetes config file present at $HOME/.kube/config:

$ operator-sdk up local --namespace=default
2018/09/30 23:10:11 Go Version: go1.10.2
2018/09/30 23:10:11 Go OS/Arch: darwin/amd64
2018/09/30 23:10:11 operator-sdk Version: 0.0.6+git
2018/09/30 23:10:12 Registering Components.
2018/09/30 23:10:12 Starting the Cmd.

You can use a specific kubeconfig via the flag --kubeconfig=<path/to/kubeconfig>.

Create a Memcached CR

Create the example Memcached CR that was generated at deploy/crds/cache_v1alpha1_memcached_cr.yaml:

$ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml
apiVersion: ""
kind: "Memcached"
  name: "example-memcached"
  size: 3

$ kubectl apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

Ensure that the memcached-operator creates the deployment for the CR:

$ kubectl get deployment
memcached-operator       1         1         1            1           2m
example-memcached        3         3         3            3           1m

Check the pods and CR status to confirm the status is updated with the memcached pod names:

$ kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
example-memcached-6fd7c98d8-7dqdr     1/1       Running   0          1m
example-memcached-6fd7c98d8-g5k7v     1/1       Running   0          1m
example-memcached-6fd7c98d8-m7vn7     1/1       Running   0          1m
memcached-operator-7cc7cfdf86-vvjqk   1/1       Running   0          2m
$ kubectl get memcached/example-memcached -o yaml
kind: Memcached
  clusterName: ""
  creationTimestamp: 2018-03-31T22:51:08Z
  generation: 0
  name: example-memcached
  namespace: default
  resourceVersion: "245453"
  selfLink: /apis/
  uid: 0026cc97-3536-11e8-bd83-0800274106a1
  size: 3
  - example-memcached-6fd7c98d8-7dqdr
  - example-memcached-6fd7c98d8-g5k7v
  - example-memcached-6fd7c98d8-m7vn7

Update the size

Change the spec.size field in the memcached CR from 3 to 4 and apply the change:

$ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml
apiVersion: ""
kind: "Memcached"
  name: "example-memcached"
  size: 4

$ kubectl apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

Confirm that the operator changes the deployment size:

$ kubectl get deployment
example-memcached    4         4         4            4           5m

Reference implementation

The above walkthrough follows the actual implementation process used to produce the memcached-operator in the SDK samples repo.

Manage the operator using the Operator Lifecycle Manager

The previous section has covered manually running an operator. In the next sections, we will explore using the operator Lifecycle Manager which is what enables a more robust deployment model for operators being run in production environments.

The Operator Lifecycle Manager helps you to install, update, and generally manage the lifecycle of all of the operators (and their associated services) on a Kubernetes cluster. It runs as an Kubernetes extension and lets you use kubectl for all the lifecycle management functions without any additional tools.

Generate an operator manifest

The first step to leveraging the Operator Lifecycle Manager is to create a manifest. An operator manifest describes how to display, create and manage the application, in this case memcached, as a whole. It is required for the Operator Lifecycle Manager to function.

For the purpose of this guide, we will continue with this predefined manifest file for the next steps. If you’d like, you can alter the image field within this manifest to reflect the image you built in previous steps, but it is unnecessary. In the future, the Operator SDK CLI will generate an operator manifest for you, a feature that is planned for the next release of the Operator SDK.

Deploy the Operator

Deploying an operator is as simple as applying the operator’s manifest to the desired namespace in the cluster.

$ curl -Lo memcachedoperator.0.0.1.csv.yaml
$ kubectl apply -f memcachedoperator.0.0.1.csv.yaml
$ kubectl get ClusterServiceVersion memcachedoperator.v0.0.1 -o json | jq '.status'

After applying this manifest, nothing has happened yet, because the cluster does not meet the requirements specified in our manifest. Create the CustomResourceDefinition and RBAC rules for the Memcached type managed by the operator:

$ kubectl apply -f deploy/rbac.yaml
$ kubectl apply -f deploy/crd.yaml

Because the Operator Lifecycle Manager creates operators in a particular namespace when a manifest has been applied, administrators can leverage the native Kubernetes RBAC permission model to restrict which users are allowed to install operators.

Create an application instance

The memcached operator is now running in the memcached namespace. Users interact with operators via instances of CustomResources; in this case, the resource has the Kind Memcached. Native Kubernetes RBAC also applies to CustomResources, providing administrators control over who can interact with each operator.

Creating instances of memcached in this namespace will now trigger the memcached operator to instantiate pods running the memcached server that are managed by the operator. The more CustomResources you create, the more unique instances of memcached will be managed by the memcached operator running in this namespace.

$ cat <<EOF | kubectl apply -f -
apiVersion: ""
kind: "Memcached"
  name: "memcached-for-wordpress"
  size: 1
$ cat <<EOF | kubectl apply -f -
apiVersion: ""
kind: "Memcached"
  name: "memcached-for-drupal"
  size: 1
$ kubectl get Memcached
NAME                      AGE
memcached-for-drupal      22s
memcached-for-wordpress   27s
$ kubectl get pods
NAME                                       READY     STATUS    RESTARTS   AGE
memcached-app-operator-66b5777b79-pnsfj    1/1       Running   0          14m
memcached-for-drupal-5476487c46-qbd66      1/1       Running   0          3s
memcached-for-wordpress-65b75fd8c9-7b9x7   1/1       Running   0          8s

Update an application

Manually applying an update to the operator is as simple as creating a new operator manifest with a replaces field that references the old operator manifest. The Operator Lifecycle Manager will ensure that all resources being managed by the old operator have their ownership moved to the new operator without fear of any programs stopping execution. It is up to the operators themselves to execute any data migrations required to upgrade resources to run under a new version of the operator.

The following command demonstrates applying a new operator manifest using a new version of the operator and shows that the pods remain executing:

$ curl -Lo memcachedoperator.0.0.2.csv.yaml
$ kubectl apply -f memcachedoperator.0.0.2.csv.yaml
$ kubectl get pods
NAME                                       READY     STATUS    RESTARTS   AGE
memcached-app-operator-66b5777b79-pnsfj    1/1       Running   0          3s
memcached-for-drupal-5476487c46-qbd66      1/1       Running   0          14m
memcached-for-wordpress-65b75fd8c9-7b9x7   1/1       Running   0          14m


Hopefully, this guide was an effective demonstration of the value of the Operator Framework for building and managing operators. There is much more that we left out in the interest of brevity. The Operator Framework and its components are open source, so please feel encouraged to jump into each individually and learn what else you can do. If you want to discuss your experience, have questions, or want to get involved, join the Operator Framework mailing list.

You can’t perform that action at this time.