Creating Custom Operators

The memhog-operator is an example on how to create custom operators for Kubernetes.

The purpose of the memhog-operator is to watch for Pods in a namespace and monitor its memory usage. If the memory consumption of the Pod crosses a threshold, it will be vertically autoscaled by the operator.

Specifically, the operator will deploy a new copy of the Pod with a higher set of resource requests and limit, and then terminate the original Pod. The details of the higher resources are held within an AppMonitor, a CustomResourceDefinition (CRD).

memhog: An example Pod that this operator would monitor.

Note: The memhog-operator is strictly for educational & demo purposes. It is not intended to be used for any other use-cases.

Operator Structure

The memhog-operator is a combination of a CustomResourceDefinition known as the AppMonitor, and a custom Controller to enforce state.

The AppMonitor encapsulates the autoscaling details for a Pod. The controller watches a Namespace for an AppMonitor, and for Pods that wish to be monitored (via Annotation). It then applies the operational thresholds and requirements declared in the AppMonitor onto the Pod.

Requirements

• Kubernetes v1.7.0+
• Prometheus on k8s with cAdvisor exposing container_memory_usage_bytes
• glide v0.11.1

Process

• To monitor the Pod's resource memory consumption, the operator requires that the Pod have an annotation in its spec.template.metadata to associate itself with the memhog-operator.

  e.g. The memhog example is annotated as such:

  apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    name: memhog
  spec:
    replicas: 1
    template:
      metadata:
        labels:
          name: memhog
        annotations:
          app-monitor.kubedemo.com/monitor: "true"
      spec:
        containers:
        - name: memhog
          image: quay.io/metral/memhog:v0.0.1
          imagePullPolicy: Always
          resources:
            limits:
              memory: 384Mi
            requests:
              memory: 256Mi
          ...
  

• Run the annotated Pod, noting the resource requests & limits set that are also required by the Operator.

• In the Pod's namespace there must also be an instantiated object of the custom AppMonitor CRD that the Operator depends on. For example, this AppMonitor states that any Pod being monitored by the memhog-operator will have its resources doubled when 75% or more of its memory has been used e.g.:

  apiVersion: kubedemo.com/v1
  kind: AppMonitor
  metadata:
    name: johnny-cache
  spec:
    memThresholdPercent: 75   # Percentage of (memory used) / (memory limit)
    memMultiplier: 2          # Multiplier factor used to increase memory resource requests & limits
  

• The memhog-operator will watch the Pod's memory usage by querying Prometheus, applying the AppMonitor to the Pod as memory usage is retrieved. If the AppMonitor threshold is crossed, the Operator will redeploy the Pod with higher resource requests & limits based on the multiplier.

• If the Pod is redeployed, it will have updated and increased resource requests & limits e.g.:

  ...
  resources:
    limits:
      memory: 768Mi
    requests:
      memory: 512Mi
  ...

Building & Running

// Build
$ glide install -s -u -v
$ make

Run Locally

Note: Prometheus is assumed to be running locally on http://localhost:9090

$ $GOPATH/bin/memhog-operator -v2 --prometheus-addr=http://localhost:9090 --kubeconfig=$HOME/.kube/config

Run on Kubernetes

Note: Prometheus is assumed to be running by default on http://prometheus.tectonic-system:9090 (e.g. Tectonic Cluster) by the Operator Deployment

// Create cluster role & cluster role binding to work with CRD's.
$ kubectl create -f k8s/roles/role.yaml

$ kubectl create -f k8s/deploy/memhog-operator-deploy.yaml

References

• ThirdPartyResources (TPR) -> CustomResourceDefinitions (CRD)

• CRD Example

• The TPR is dead. Long live the CRD: Custom Resources in Kubernetes v1.7

• Working with Controllers

• Writing a Custom Controller - Aaron Levy (CoreOS)

  • KubeCon EU 2017 Video

• DaemonSet Controller Code

• Deprecated

  • TPRs are Deprecated
  • Extend the Kubernetes API with TPRs