Permalink
Switch branches/tags
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
884 lines (656 sloc) 28 KB
title description type order
Documentation
Advanced, in-depth configuration of Keel
guide
4

While it is extremely easy to set up Keel and just use provided default configuration, sometimes you need additional configuration to better solve your use-case.

Policies

Use policies to define when you want your application to be updated. Providers can have different mechanisms of getting configuration for your application, but policies are consistent across all of them. Following semver best practices allows you to safely automate application updates.

Available policies:

  • all: update whenever there is a version bump or a new prerelease created (ie: 1.0.0 -> 1.0.1-rc1)
  • major: update major & minor & patch versions
  • minor: update only minor & patch versions (ignores major)
  • patch: update only patch versions (ignores minor and major versions)
  • force: force update even if tag is not semver, ie: latest, optional label: keel.sh/match-tag=true which will enforce that only the same tag will trigger force update.
  • glob: use wildcards to match versions, example:
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata: 
  name: wd
  namespace: default
  labels: 
      name: "wd"
  annotations:
      keel.sh/policy: "glob:build-*"  # <- build-1, build-2, build-foo will match this. 
  • regexp: use regular expressions to match versions, example:
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata: 
  name: wd
  namespace: default
  labels: 
      name: "wd"
  annotations:
      keel.sh/policy: "regexp:^([a-zA-Z]+)$"

Pre-release tags

According to semver (http://semver.org/) spec, version tags can have additional metadata such as 1.0.0-dev, 1.0.0-prod, etc. Keel deals with semver tags by only allowing updates with the same version metadata.

Example:

  • tag: 0.5.0-dev policy: minor will only be updated by 0.6.0-dev and not 0.5.0-prod
  • tag 0.5.0 policy: minor will not be updated by 0.6.0-rc1

This way you can easily separate different environments and update them independently.

Additional settings

Keel tries to mostly rely on your resource configuration files, such as deployment, daemonset, statefulset labels & annotations. Here is an example with all available options:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: wd-ds
  namespace: default
  labels: 
      name: "wd"
      keel.sh/policy: minor     # update policy (available: patch, minor, major, all, force)
      keel.sh/trigger: poll     # enable active repository checking (webhooks and GCR would still work)
      keel.sh/approvals: "1"    # required approvals to update
      keel.sh/match-tag: "true" # only makes a difference when used with 'force' policy, will only update if tag matches :dev->:dev, :prod->:prod 
  annotations:
      keel.sh/pollSchedule: "@every 1m"
      keel.sh/notify: chan1,chan2  # chat channels to sent notification to
spec:
  selector:
    matchLabels:
      name: wd-ds
  template:
    metadata:
      labels:
        name: wd-ds
    spec:      
      containers:
      - name: wd-ds
        image: karolisr/webhook-demo:master
        imagePullPolicy: Always            
        name: wd
        command: ["/bin/webhook-demo"]
        ports:
          - containerPort: 8090       
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8090
          initialDelaySeconds: 30
          timeoutSeconds: 10

Providers

Providers are direct integrations into schedulers or other tools (ie: Helm). Providers are handling events created by triggers. Each provider can handle events in different ways, for example Kubernetes provider identifies impacted deployments and starts rolling update while Helm provider communicates with Tiller, identifies releases by Chart and then starts update.

Available providers:

  • Kubernetes (supports Deployments, DaemonSets, StatefulSets)
  • Helm

While the goal is often the same, different providers can choose different update strategies.

Kubernetes

Kubernetes provider was the first, simplest provider added to Keel. Policies and trigger configuration for each application deployment is done through labels.

Policies are specified through special label:

keel.sh/policy=all

A policy to update only minor releases:

keel.sh/policy=minor

Kubernetes example

Here is an example application deployment.yaml where we instruct Keel to update container image whenever there is a new version:

apiVersion: extensions/v1beta1
kind: Deployment
metadata: 
  name: wd
  namespace: default
  labels: 
      name: "wd"
      keel.sh/policy: all
spec:
  replicas: 1
  template:
    metadata:
      name: wd
      labels:
        app: wd        

    spec:
      containers:                    
        - image: karolisr/webhook-demo:0.0.2
          imagePullPolicy: Always            
          name: wd
          command: ["/bin/webhook-demo"]
          ports:
            - containerPort: 8090       
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8090
            initialDelaySeconds: 30
            timeoutSeconds: 10
          securityContext:
            privileged: true      

If Keel gets an event that karolisr/webhook-demo:0.0.3 is available - it will update deployment and therefore start a rolling update.

StatefulSet example

StatefulSets can also be updated by Kubernetes once image has changed:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: wd
  namespace: default
  labels: 
      name: "wd"
      keel.sh/policy: major    
spec:
  replicas: 1
  serviceName: "wd"
  selector:
    matchLabels:
      app: wd
  template:
    metadata:
      name: wd
      labels:
        app: wd        
    spec:      
      containers:                    
        - image: karolisr/webhook-demo:0.0.8
          imagePullPolicy: Always            
          name: wd
          command: ["/bin/webhook-demo"]
          ports:
            - containerPort: 8090       
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8090
            initialDelaySeconds: 30
            timeoutSeconds: 10              

Deployment polling example

While the deployment above works perfect for both webhook and Google Cloud Pubsub triggers sometimes you can't control these events and the only available solution is to check registry yourself. This is where polling trigger comes to the rescue.

Note: when image with non-semver style tag is supplied (ie: latest) Keel will monitor SHA digest. If tag is semver - it will track and notify providers when new versions are available.

Add labels:

keel.sh/policy=force # add this to enable updates of non-semver tags
keel.sh/trigger=poll

To specify custom polling schedule, check [cron expression format]({{ page.url }}#cron-expression-format)

Note that even if polling trigger is set - webhooks or pubsub events can still trigger updates

Example deployment file for polling:

apiVersion: extensions/v1beta1
kind: Deployment
metadata: 
  name: wd
  namespace: default
  labels: 
      name: "wd"
      keel.sh/policy: force
      keel.sh/trigger: poll      
  annotations:
      keel.sh/pollSchedule: "@every 10m"
spec:
  replicas: 1
  template:
    metadata:
      name: wd
      labels:
        app: wd        

    spec:
      containers:
        - image: karolisr/webhook-demo:latest # this would start repository digest checks
          imagePullPolicy: Always            
          name: wd
          command: ["/bin/webhook-demo"]
          ports:
            - containerPort: 8090       
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8090
            initialDelaySeconds: 30
            timeoutSeconds: 10
          securityContext:
            privileged: true      

DaemonSet polling example

Since DaemonSets have labels and annotations, their configuration is not different from Deployment or StatefulSet configuration:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: wd-ds
  namespace: default
  labels: 
      name: "wd"
      keel.sh/policy: minor
      keel.sh/trigger: poll            
  annotations:
      keel.sh/pollSchedule: "@every 1m"          
spec:
  selector:
    matchLabels:
      name: wd-ds
  template:
    metadata:
      labels:
        name: wd-ds
    spec:      
      containers:
      - name: wd-ds
        image: karolisr/webhook-demo:0.0.8
        imagePullPolicy: Always            
        name: wd
        command: ["/bin/webhook-demo"]
        ports:
          - containerPort: 8090       
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8090
          initialDelaySeconds: 30
          timeoutSeconds: 10
        securityContext:
          privileged: true         

Helm

Helm helps you manage Kubernetes applications — Helm Charts helps you define, install, and upgrade even the most complex Kubernetes application. More information can be found on project's website https://helm.sh/.

Keel works directly with Tiller (a daemon that is used by Helm CLI) to manage release upgrades when new images are available.

Helm example

Keel is configured through your chart's values.yaml file.

Here is an example application values.yaml file where we instruct Keel to track and update specific values whenever there is a new version:

replicaCount: 1
image:
  repository: karolisr/webhook-demo
  tag: "0.0.8"
  pullPolicy: IfNotPresent
service:
  name: webhookdemo
  type: ClusterIP
  externalPort: 8090
  internalPort: 8090

keel:
  # keel policy (all/major/minor/patch/force)
  policy: all
  # images to track and update
  images:
    - repository: image.repository # [1]
      tag: image.tag  # [2]

If Keel gets an event that karolisr/webhook-demo:0.0.9 is available - it will upgrade release values so Helm can start updating your application.

  • [1] resolves during runtime image.repository -> karolisr/webhook-demo
  • [2] resolves during runtime image.tag -> 0.0.8

Helm same tag force updates

If you are not using versioning and pushing to the same tag, you should modify your template to change it on each update.

Current consensus on a best way to "force" update Helm releases is by modifying your pod spec template by adding:

date/deploy-date: {{ now | quote }}

annotation. This way Helm's Tiller will always detect a change in your template and Kubernetes will start a rolling update on the resource.

Helm configuration polling example

This example demonstrates Keel configuration for polling.

Note that even if polling trigger is set - webhooks or pubsub events can still trigger updates

replicaCount: 1
image:
  repository: karolisr/webhook-demo
  tag: "0.0.8"
  pullPolicy: IfNotPresent
service:
  name: webhookdemo
  type: ClusterIP
  externalPort: 8090
  internalPort: 8090

keel:
  # keel policy (all/major/minor/patch/force)
  policy: all
  # trigger type, defaults to events such as pubsub, webhooks
  trigger: poll
  # polling schedule
  pollSchedule: "@every 2m"
  # images to track and update
  images:
    - repository: image.repository # [1]
      tag: image.tag  # [2]

Triggers

Triggers are entry points into the Keel. Their task is to collect information regarding updated images and send events to providers.

Available triggers:

  • Webhooks
    • Native Webhooks
    • DockerHub Webhooks
    • Quay Webhooks
    • Harbor webhooks
    • Gitlab webhooks
    • Receiving webhooks without public endpoint
  • Google Cloud GCR registry
  • Polling

Webhooks

Webhooks are "user-defined HTTP callbacks". They are usually triggered by some event, such as pushing image to a registry. Native webhooks (simplified version) are accepted at /v1/webhooks/native endpoint with a payload that has name and tag fields:

{
  "name": "gcr.io/v2-namespace/hello-world", 
  "tag": "1.1.1"
}

Keel by default runs HTTP server on port 9300. Create a service and either expose it to the internet or use https://webhookrelay.com to receive webhooks.

DockerHub Webhooks

DockerHub uses webhooks to inform 3rd party systems about repository related events such as pushed image.

https://docs.docker.com/docker-hub/webhooks - go to your repository on https://hub.docker.com/r/your-namespace/your-repository/~/settings/webhooks/ and point webhooks to /v1/webhooks/dockerhub endpoint. Any number of repositories can send events to this endpoint.

Quay Webhooks

Documentation on how to setup Quay webhooks for Repository Push events is available here: https://docs.quay.io/guides/notifications.html. These webhooks should be delivered to /v1/webhooks/quay endpoint. Any number of repositories can send events to this endpoint.

Azure Webhooks

Documentation on how to setup Azure webhooks is available here: https://docs.microsoft.com/en-us/azure/container-registry/container-registry-webhook. Azure webhooks should be delivered to /v1/webhooks/azure endpoint.

Harbor webhooks

Keel supports https://github.com/goharbor/harbor webhooks. Harbor webhooks should be delivered to /v1/webhooks/registry endpoint. Harbor webhooks are based on Docker registry notifications.

Gitlab webhooks

Keel supports Gitlab registry notifications also known as webhooks (https://docs.gitlab.com/ee/administration/container_registry.html#configure-container-registry-notifications). Gitlab webhooks should be delivered to /v1/webhooks/registry endpoint. Gitlab webhooks are based on Docker registry notifications.

Receiving webhooks without public endpoint

If you don't want to expose your Keel service - recommended solution is https://webhookrelay.com/ which can deliver webhooks to your internal Keel service through a sidecar container.

Example sidecar container configuration for your deployments.yaml:

        - image: webhookrelay/webhookrelayd:latest
          name: webhookrelayd          
          env:                         
            - name: KEY
              valueFrom:
                secretKeyRef:
                  name: webhookrelay-credentials
                  key: key                
            - name: SECRET
              valueFrom:
                secretKeyRef:
                  name: webhookrelay-credentials
                  key: secret
            - name: BUCKET
              value: dockerhub      

Google Cloud GCR registry

If you are using Google Container Engine with Container Registry - search no more, pubsub trigger is for you.

You will need to create a Google Service Account to use PubSub functionality.

Since Keel requires access for the pubsub in GCE Kubernetes to work - your cluster node pools need to have permissions. If you are creating a new cluster - just enable pubsub from the start. If you have an existing cluster - currently the only way is to create a new node-pool through the gcloud CLI (more info in the docs):

Create a node pool with enabled pubsub scope

gcloud container node-pools create new-pool --cluster CLUSTER_NAME --scopes https://www.googleapis.com/auth/pubsub

Create a service account

Detailed tutorial on creating and configuring service account to access Google services is available here: https://cloud.google.com/kubernetes-engine/docs/tutorials/authenticating-to-cloud-platform.

High level steps:

  1. Create a service account through Google cloud console with a role: roles/pubsub.editor (Keel needs to create topics as well for GCR registries).
  2. Furnish a new private key and choose key type as JSON.
  3. Import credentials as a secret:
kubectl create -n <KEEL NAMESPACE> secret generic pubsub-key --from-file=key.json=<PATH-TO-KEY-FILE>.json
  1. Configure the application with the Secret

Update Keel's environment variables

Make sure that in the deployment.yaml you have set environment variables PUBSUB=1, PROJECT_ID=your-project-id and GOOGLE_APPLICATION_CREDENTIALS to your secrets yaml.

Polling

Since only the owners of docker registries can control webhooks - it's often convenient to use polling. Be aware that registries can be rate limited so it's a good practice to set up reasonable polling intervals. While configuration for each provider can be slightly different - each provider has to accept several polling parameters:

  • Explicitly enable polling trigger
  • Supply polling schedule (defaults to 1 minute intervals)

Cron expression format

Custom polling schedule can be specified as cron format or through predefined schedules (recommended solution).

Available schedules:

Entry Description Equivalent To
@yearly (or @annually) Run once a year, midnight, Jan. 1st 0 0 0 1 1 *
@monthly Run once a month, midnight, first of month 0 0 0 1 * *
@weekly Run once a week, midnight on Sunday 0 0 0 * * 0
@daily (or @midnight) Run once a day, midnight 0 0 0 * * *
@hourly Run once an hour, beginning of hour 0 0 * * * *

Polling with AWS ECR

If you are using polling trigger with Amazon ECR registry, Keel deployment requires several environment variables:

AWS_ACCESS_KEY_ID=AKIA.........
AWS_SECRET_ACCESS_KEY=3v..............
AWS_REGION=us-east-2 # <- where your registry is

Documentation on setting up credentials can be found here: https://docs.aws.amazon.com/amazonswf/latest/awsrbflowguide/set-up-creds.html.

Intervals

You may also schedule a job to execute at fixed intervals. This is supported by formatting the cron spec like this:

@every <duration>

where duration is a string accepted by time.ParseDuration.

For example, @every 1h30m10s would indicate a schedule that activates every 1 hour, 30 minutes, 10 seconds.

Tip: If you want to disable polling support for your Keel installation - set environment variable POLL=0.

Approvals

Users can specify on deployments and Helm charts how many approvals do they have to collect before a resource gets updated. Main features:

  • non-blocking - multiple deployments/helm releases can be queued for approvals, the ones without specified approvals will be auto updated.
  • extensible - current implementation focuses on Slack but additional approval collection mechanisms are trivial to implement.
  • out of the box Slack integration - the only needed Keel configuration is Slack auth token, Keel will start requesting approvals and users will be able to approve.
  • stateful - uses github.com/rusenask/k8s-kv for persistence so even after updating itself (restarting) it will retain existing info.
  • self cleaning - expired approvals will be removed after deadline is exceeded.

Enabling approvals

Approvals are enabled by default but currently there is only one way to approve/reject updates: Slack - commands like:

  • keel get approvals - get all pending/approved/rejected approvals
  • keel approve <identifier> - approve specified request.
  • keel reject <identifier> - reject specified request.

Make sure you have set export SLACK_TOKEN=<your slack token here> environment variable for Keel deployment.

If you wish to specify a special channel for approval requests, supply SLACK_APPROVALS_CHANNEL=<approvals channel name> environment variable and then invite Keel bot to that channel.

Configuring via Kubernetes deployments

The only required configuration for Kubernetes deployment to enable approvals is to add keel.sh/approvals: "1" with a number (string! as the underlying type is map[string]string) of required approvals.

apiVersion: extensions/v1beta1
kind: Deployment
metadata: 
  name: wd
  namespace: default
  labels: 
      name: "wd"
      keel.sh/policy: all
      keel.sh/trigger: poll      
      keel.sh/approvals: "1"

Configuring via Helm charts

To enable approvals for a Helm chart update Keel config section in values.yaml with a required number of approvals:

replicaCount: 1
image:
  repository: karolisr/webhook-demo
  tag: "0.0.13"
  pullPolicy: IfNotPresent 
service:
  name: webhookdemo
  type: ClusterIP
  externalPort: 8090
  internalPort: 8090

keel:
  # keel policy (all/major/minor/patch/force)
  policy: all
  # trigger type, defaults to events such as pubsub, webhooks
  trigger: poll
  # polling schedule
  pollSchedule: "@every 1m"
  # approvals required to proceed with an update
  approvals: 1
  # approvals deadline in hours
  approvalDeadline: 24 
  # images to track and update
  images:
    - repository: image.repository
      tag: image.tag

Configuring approvals with Slack

Slack configuration can be sometimes quite confusing. If something has changed, please create an issue.

Step 1: adding bot app and getting token

Go to your Slack apps page: https://[your-slack-community].slack.com/apps/A0F7YS25R-bots?page=1

Slack bots

Set name to Keel

Slack bot name

Step 2: supplying token to Keel

Use provided token as an environment variable in Keel's deployment:

SLACK_TOKEN=token

Approving through Slack example

Keel will send notifications to your Slack group about pending approvals. Approval process is as simple as replying to Keel:

  • Approve: keel approve default/whr:0.4.12
  • Reject it: keel reject default/whr:0.4.12

Example conversation:

Approvals

Approving through Hipchat example

Coming soon...

Managing approvals through HTTP endpoint

For third party integrations it can be useful to approve/reject/delete via HTTP endpoint. You can send an approval request via HTTP endpoint:

Method: POST Endpoint: /v1/approvals

{
  "identifier": "default/myimage:1.5.5", // <- identifier for the approval request
  "action": "approve", // <- approve/reject/delete, defaults to "approve"
  "voter": "john",  
}

Listing pending approvals through HTTP endpoint

You can also view pending/rejected/approved update request on /v1/approvals Keel endpoint (make sure you have service exported). Example response:

Method: GET Endpoint: /v1/approvals

[
	{
		"provider": "helm",
		"identifier": "default/wd:0.0.15",
		"event": {
			"repository": {
				"host": "",
				"name": "index.docker.io/karolisr/webhook-demo",
				"tag": "0.0.15",
				"digest": ""
			},
			"createdAt": "0001-01-01T00:00:00Z",
			"triggerName": "poll"
		},
		"message": "New image is available for release default/wd (0.0.13 -> 0.0.15).",
		"currentVersion": "0.0.13",
		"newVersion": "0.0.15",
		"votesRequired": 1,
		"deadline": "2017-09-26T09:14:54.979211563+01:00",
		"createdAt": "2017-09-26T09:14:54.980936804+01:00",
		"updatedAt": "2017-09-26T09:14:54.980936824+01:00"
	}
]

Notifications

Keel can send notifications on successful or failed deployment updates. There are several types of notifications - trusted webhooks or Slack, Hipchat messages.

Notification types:

Pre-deployment update - fired before doing update. Can be used to drain running tasks.

{
	"name":"preparing to update deployment",
	"message":"Preparing to update deployment <your deployment namespace/name> (gcr.io/webhookrelay/webhook-demo:0.0.10)",
	"createdAt":"2017-07-23T23:51:46.478440258+01:00",
	"type":"preparing deployment update",
	"level":"LevelDebug"
}

Successful deployment update - fired after successful update.

{
	"name":"deployment update",
	"message":"Successfully updated deployment <your deployment namespace/name> (gcr.io/webhookrelay/webhook-demo:0.0.10)",
	"createdAt":"2017-07-23T23:51:46.478440258+01:00",
	"type":"deployment update",
	"level":"LevelSuccess"
}

Failed deployment update - fired after failed event.

{
	"name":"deployment update",
	"message":"Deployment <your deployment namespace/name> (gcr.io/webhookrelay/webhook-demo:0.0.10) update failed, error: <error here> ", 
	"createdAt":"2017-07-23T23:51:46.478440258+01:00",
	"type":"deployment update",
	"level":"LevelError"
}

Release notes in notifications

You can optionally set a release notes link (or anything that's allowed by annotation value or Helm chart value) in your configuration.

If you are using Kubernetes, use annotation keel.sh/releaseNotes:

apiVersion: extensions/v1beta1
kind: Deployment
metadata: 
  name: wd
  namespace: default
  labels: 
      name: "wd"
      keel.sh/policy: minor
  annotations:
      keel.sh/releaseNotes: "https://github.com/keel-hq/keel/releases"

If you are using Helm, your Keel config now accepts releaseNotes:

name: app1
image:
  repository: gcr.io/v2-namespace/hello-world
  tag: 1.1.0

keel:  
  policy: force  
  trigger: poll  
  images:
    - repository: image.repository
      tag: image.tag
      releaseNotes: https://github.com/keel-hq/keel/releases			

Release notes will be sent with a successful update.

Webhook notifications

To enabled webhook notifications provide an endpoint via WEBHOOK_ENDPOINT environment variable inside Keel deployment.

Webhook payload sample:

{
	"name": "update deployment",
	"message": "Successfully updated deployment default/wd (karolisr/webhook-demo:0.0.10)",
	"createdAt": "2017-07-08T10:08:45.226565869+01:00"	
}

Slack notifications

Slack notifications

First, get a Slack token, info about that can be found in the docs. Then, provide token via SLACK_TOKEN environment variable. You should also provide SLACK_CHANNELS environment variable with a comma separated list of channels where these notifications should be delivered to.

Keel will be sending messages when deployment updates succeed or fail.

Hipchat notifications

Coming soon...

Mattermost notifications

Mattermost is an open source Slack alternative, it's written in Go and React.

If you don't have a Mattermost server, you can set one up by using Docker:

docker run --name mattermost-preview -d --publish 8065:8065 mattermost/mattermost-preview

Server should be reachable on: http://localhost:8065/

Now, enable "incoming webhooks" in your Mattermost server. Documentation can be found here:

Mattermost webhooks

Also, you should enable icon and username override so users know that webhooks are coming from Keel:

Mattermost username and icon override

Now, set MATTERMOST_ENDPOINT environment variable for Keel with your Mattermost webhook endpoint:

Mattermost configuration

That's it, Keel notifications for Mattermost enabled:

Mattermost notification

If you want to override bot's username, you can supply MATTERMOST_USERNAME=somethingelse environment variable.

Notification levels

Set notification levels via NOTIFICATION_LEVEL environment variable. Available levels: debug, info, success, warn, error, fatal. This setting defaults to info.

Overriding default channels per deployment

Some notification providers such as Slack and Hipchat allow apps to send notifications to multiple channels.

For standard Kubernetes deployments use annotation with channels separated by commas: keel.sh/notify=channelName1,channelName2.

notifications per deployment

For Helm chart please specify notificationChannels string list to Keel config section in values.yaml:

notificationChannels:
  - chan1
  - chan2

Your values.yaml should look like this:

notifications per chart