Canary deployments with Ambassador and Azure Pipelines

Rationale

Azure pipelines have a deployment mechanism for doing Canaries. This mechanims can use an existing service mesh in your cluster but, if you are not using Istio/Linkerd/etc., it does it with regular Pods: it creates a new Deployment with the same labels but with the image being tested. Your existing Service will send traffic to the old pods as well as to the new pods (as they both match the Service selector), and this will be done proportionally to the number of replicas of each Deployment.

The main problem with this solution is the granularity of the traffic split. A canary for 10% traffic would require 9 pods in the old deployment and 1 with the new one, something that is sometimes neither possible nor desirable.

Ambassador can split traffic by assigning different weight to Mappings, as described here. This provides much finer grain control over the traffic without the need to change the number of pods.

Proposed solution

Instead of using the Azure pipelines facilities for splitting traffic with new Pods, we have created a new canarize.py script for generating canary deployments and Mappings for sending traffic to these deployments.

The Azure Pipeline will deploy the canary by iterating over a list of weights. For each weight, and given

• the manifests for your Service/Deployment
• an image built and pushed with the current source code
• the current weight for the canary

it will create and apply:

• new *-canary versions of those Service/Deployment, using the image from the current build.
• a Mapping that will send weight% of the requests to the canary Service.

References

• Canary deployment strategy for Kubernetes deployments
• Deployment strategies
• Canary releases with Ambassador

Overview of this repository

• ./app:

  app.py - Simple Flask based web server instrumented using Prometheus instrumentation library for Python applications. A custom counter is set up for the number of 'good' and 'bad' responses given out based on the value of success_rate variable.

  Dockerfile - Used for building the image with each change made to app.py. With each change made to app.py, build pipeline (CI) is triggered and the image gets built and pushed to the container registry.

• ./manifests:

  deployment.yml - Contains specification of the sampleapp Deployment workload corresponding to the image published earlier. This manifest file is used not just for the stable version of Deployment object, but for deriving the -canary variant of the workloads as well.

  service.yml - Creates a sampleapp Service for routing requests to the pods spun up by the Deployment.

  mapping.yml - Ambassador Mapping for routing all the requests to / to the sampleapp Service.

• ./misc:

  service-monitor.yml - Used for setup of a ServiceMonitor object to set up Prometheus metric scraping.

  fortio-deploy.yml - Used for setup of fortio deployment that is subsequently used as a load-testing tool to send a stream of requests to the sampleapp service deployed earlier. With sampleapp service's selector being applicable for all the three pods resulting from the Deployment objects that get created during the course of this how-to guide - sampleapp, sampleapp-baseline and sampleapp-canary, the stream of requests sent to sampleapp get routed to pods under all these three deployments.

• ./misc:

  canarize.py - Script for generating a Canary for a Service/Deployment.

Preparing your cluster

• Create a Kubernetes cluster in Azure by following the official docs. We will assume your have created it in a Resource group called Ambassador-Azure-Pipeline, and the cluster name will be Ambassador-Azure-Pipeline.
• Get the credentials for your Kuberentes cluster (using the resource group and the cluster name):

  $ az aks get-credentials --resource-group Ambassador-Azure-Pipeline --name Ambassador-Azure-Pipeline

• Install Ambassador in this cluster by following the instructions. For example:

  edgectl install

• Install the prometheus operator. For example, with Helm 3:

  helm repo add bitnami https://charts.bitnami.com/bitnami
  helm install prometheus bitnami/prometheus-operator

  Prometheus can be accessed with kubectl port-forward --namespace default svc/prometheus-prometheus-oper-prometheus 9090:9090

Preparing the repo

Azure DevOps project

• Navigate to "All services" > "Azure DevOps", or go to https://dev.azure.com directly. Register a new project.

Connections

• In your Azure DevOps project, navigate to "Project settings" > "Service connections", or go to https://dev.azure.com/<user>/<project>/_settings/adminservices directly. Then:
  • Create a new Docker Registry connection, select Azure Container Registry and select one of the existing registries. Assign a name like azurepipelinescanaryk8s
  • Add another Kubernetes service connection for connecting to your existing kubernetes cluster. Name it k8sEnvironment.
  • Configure a namespace for your environment by navigating to "Pipelines" > "Environments" > "k8sEnvironment". Click the "Add Resource" button and chose "Add Kubernetes namespace". Select the desired namespace (default for this sample), then "Validate and create".

Customizing the pipeline definition

• Review the variables in the azure-pipelines.yml, specially the containerRegistry and environment. Their values should match the service connections created previously.
• In manifests/deployment.yml, replace the image with your container registry's URL.

Creating the Azure DevOps Pipeline

• In your Azure DevOps project, navigate to "Pipelines" > "Create Pipeline".
• Connect & Select your code location. Azure Repos Git is a great option if you do not wish to open your GitHub account. It will however require you to push your repo to a new location.
• Review & Run!