This article is the first in a series of articles and tutorials that show you how to build and deploy a Kubernetes Operator. In this article, learn what Kubernetes Operators are, what they do, and key features including the Operator SDK, Operator capability levels and Operator Hub. You will learn all of the basic knowledge needed to develop an operator implemented using Golang.
If you are already familiar with operators, you can either skim this article or skip ahead to the next Operator tutorial: Develop and deploy a Memcached Operator on OpenShift which shows how to develop and deploy your first operator to an OpenShift cluster. Alternately, read one of the other articles in the learning path.
This article assumes you want to learn the basic concepts about Kubernetes Operators and steps needed to develop a Golang-based operator to manage Kubernetes resources.
- You have a basic understanding of Kubernetes concepts and how to install a workload
- You have little to no knowledge about Kubernetes Operators concepts
- You have little to no experience developing operators
This article should take roughly 15-30 minutes to complete.
Operators are extensions to Kubernetes that use custom resources to manage Kubernetes applications and their components. Operators are used to automate software configuration and maintenance activities that are typically performed by human operators. That's why they are called operators!
While Kubernetes is great at managing stateless applications, when you need more complex configuration details for a stateful application, such as a database, operators are very useful. An operator can also automate other more complex lifecycle management tasks such as version upgrades, failure recovery, and scaling.
Operators extend the Kubernetes control plane with specialized functionality to manage a workload on behalf of a Kubernetes admin. An operator includes these components:
- A custom resource definition (CRD) that defines a schema of settings available for configuring the workload
- A custom resource (CR) is the Kubernetes API extension created by a CRD
- A custom resource instance sets specific values for the settings defined by the CRD to describe the configuration of a workload
- A controller customized for the workload that configures the current state of the workload to match the desired state represented by the values in the CR
Operators have the following features:
- The user provides configuration settings within a CR, and then the operator translates the configuration into low-level actions, using the operator's custom controller logic to implement the translation
- An operator introduces new object types through its custom resource definition. These objects can be handled by the Kubernetes API just like native Kubernetes objects, including interaction via Kubernetes client tools and inclusion in role-based access control policies (RBAC)
The article What is a Kubernetes operator? by Red Hat explains more details about operators.
In Kubernetes, controllers in the control plane run in a control loop that repeatedly compares the desired state of the cluster to its current state. If the states don't match, then the controller takes action to adjust the current state to more closely match the desired state.
Similarly, the controller in an operator watches a specific CR type and takes application-specific actions to make the workload's current state match the desired state expressed in the CR.
This diagram illustrates how the control plane runs the controllers in a loop, where some controllers are built into Kubernetes and some are part of operators:
The controllers in the control plane are optimized for stateless workloads and one set of controllers works for all stateless workloads because they're all very similar. The controller in an operator is customized for one particular stateful workload. Each stateful workload has its own operator with its own controller that knows how to manage this workload.
Kubernetes needs operators to automate tasks that are normally performed manually by IT operations personnel. A stateful workload is more difficult to manage than a stateless workload. The state in a workload changes how a workload: needs to be installed, upgrades to a new version, recovers from failures, needs to be monitored, and scales out and back in again. The operator will take care of everything needed to make sure the service keeps running successfully. An operator makes its service more self-managing, enabling an application team to spend less effort managing the service so that it can spend more effort using the service.
To use an operator, you must first deploy the operator as a workload in your cluster. You can deploy it like any other workload, via the Kubernetes CLI or using a Helm chart. Additionally, you can make use of Operator Hub or deploy your operator through the Operator Lifecycle Manager.
Operators are usually implemented in one of three main technologies: Go, Ansible, or Helm.
- Go: Code written in Go is very powerful and can do almost anything Kubernetes can do. Kubernetes itself is implemented in Go, so operators implemented in Go are a good fit.
- Ansible: Ansible is a good choice for infrastructure teams that have already written Ansible modules. Ansible is declarative and human-readable, and can express almost as much functionality as Go.
- Helm: Simpler to implement, but functionality is limited to Helm features.
For the rest of the learning path, we will use Go to implement a Kubernetes operator, as 71% of operators from OperatorHub.io are written in Go. Also, Kubernetes itself is written in Go, so this makes writing the operator which interacts with the Kubernetes API much easier.
You can read more about the pros and cons of each approach in Build Your Kubernetes Operator With the Right Tool. The Operator SDK supports all three technologies.
The Operator SDK is a set of open source tools to build, test, and package operators. The SDK CLI enables you to scaffold a project and also provides commands to generate code. The SDK uses make, a build automation tool that runs commands configured in a Makefile to generate executable code and libraries from source code.
The SDK includes pre-built make commands that we will use to develop our operator, such as:
make manifests-- generates YAML manifests based onkubebuildermarkersmake install-- compiles source code into executablesmake generate-- updates the generated code based on an operator's API schemamake docker-build-- builds the operator's Docker container imagemake docker-push-- pushes the Docker imagemake deploy-- deploys all of the operator's resources to the clustermake undeploy-- deletes all of the operator's deployed resources from the cluster
These commands in the SDK greatly simplify implementing an operator.
The SDK also enables you to install the Operator Lifecycle Manager (OLM) using the operator-sdk olm install command. The OLM is a set of cluster resources that manage the lifecycle of an operator. Once installed, you can get the status of the OLM using operator-sdk olm status, which verifies whether the SDK can successfully communicate with the OLM components in the cluster.
In addition to operator, custom resource, and custom resource definition, you need to understand what we mean by the Operator SDK operand and managed resource:
- Operand - the managed workload provided by the Operator as a service
- Managed resources - the Kubernetes objects or off-cluster services the Operator uses to constitute an Operand (also known as secondary resources)
Some operators are more sophisticated at managing their operand's lifecycle than others. The Operator Capability Levels model defines five levels of sophistication, as illustrated here:
This model aims to provide guidance for the features that users can expect from a particular operator. As the picture shows, you can only use Ansible and Go to achieve all five capability levels. Helm can only be used to achieve the first two levels.
Capability levels build on top of one another. That means that if an operator has level 3 capabilities, then it should also have all of the capabilities required from level 1 and level 2.
Let's examine the capabilities of a level 1 operator in more detail.
In level 1, your Operator can provision an application as described by a custom resource. The CR specifies all of the configuration details. Avoid making the user create and manage configuration files outside of Kubernetes — that's what the CR is for.
When a custom resource is created, that triggers the Operator, which responds by creating and installing the Operand. If the custom resource is deleted, then the Operator removes the Operand.
To install an Operand, the controller creates the managed resources for that Operand and installs them, which causes the cluster to install the Operand. These managed resources are typical Kubernetes workload kinds such as a Deployment and a Service, as well as other kinds like a ConfigMap, a Secret, and a PersistentVolumeClaim.
You can specify the configuration of these managed resources in the custom resource's specification (that is, the spec section of its YAML); I cover this in greater detail in the next article.
A simple controller may simply copy values out of the CR's specification and into the appropriate fields of the managed resources, maybe transforming the values as needed. Once the cluster installs the Operand, the controller gathers the status of those managed resources and updates the custom resource's status (that is, the status section of its YAML); I cover this in greater detail in my next article. Status is how the CR remembers the managed resources that were created for its Operand.
Now, let's say that you want to increase the capacity of the Operand. How would you do this using the Operator?
You can do this by updating the custom resource's specification — perhaps it has a size setting, and you increase it. When you update the custom resource's specification, you are specifying a different configuration for the Operand. The controller notices the changes in the custom resource and responds by changing the configuration of the managed resources — perhaps to increase the number of pods or create new persistent volume claims.
Once the controller applies these changes to the managed resources, the cluster responds by applying them the Operand, thereby scaling the Operand.
We will only cover the first level of the capability levels in this article, but you can read more about the other four levels at Operator Capability Levels in the Operator SDK documentation.
OperatorHub.io is a web-based application that enables finding, installing, and publishing operators. It is a one stop shop for Kubernetes operators, as shown below:
Check out How to contribute an Operator for details of how to package, test, preview, and submit your operator for addition to the Hub.
In this article, you learned how operators can extend the base Kubernetes functionality using custom controllers and custom resources. You also learned that the Operator SDK offers code scaffolding tools to enable you to write your Operator more easily, and offers guidelines for the capability levels of an operator. Lastly, I introduced you to OperatorHub.io where you can browse existing operators and submit your own.
The next article, Anatomy of an Operator, demystified, takes a closer look at the Kubernetes architecture that enables Operators to work.
If you would rather go straight to developing an Operator, go to the Develop and deploy a Memcached Operator tutorial.