Easy CI/CD for Kubernetes clusters with python and jinja2
Clone or download
FreakyGranny Merge pull request #33 from 2gis/feature/recusively-include
I-23 add support of env/file macro in included file
Latest commit 6aadada Sep 17, 2018

README.md

k8s-handle

Easy CI/CD for Kubernetes clusters with python and jinja2

k8s-handle is a command line tool that facilitates continuous delivery for Kubernetes applications. Also k8s-handle supports environments, so you can use same deployment templates for different environments like staging and production.

Table of contents

Features

  • Easy to use command line interface
  • Configure any variables in one configuration file (config.yaml)
  • Templating for kubernetes resource files (jinja2) with includes, loops, if-else and so on.
  • Loading variables from environment
  • Includes for configuration (includes in config.yaml) for big deploys
  • Async and sync mode for deploy (wait for deployment, statefulset, daemonset ready)
  • Strict mode, stop deploy if any warning appear
  • Easy integration with CI pipeline (gitlab ci for example)
  • Ability to destroy resources (deploy and destroy from git branches, gitlab environments)

Before you begin

$ cat > ~/.kube/kubernetes.ca.crt << EOF
> <paste your cluster CA here>
>EOF
cat > ~/.kube/config << EOF
apiVersion: v1
kind: Config
preferences: {}
clusters:
- cluster:
    certificate-authority: kubernetes.ca.crt
    server: < protocol://masterurl:port >
  name: my-cluster
contexts:
- context:
    cluster: my-cluster
    namespace: my-namespace
    user: my-user
  name: my-context
current-context: my-context
users:
- name: my-user
  user:
    token: <your token>
EOF

Usage with docker

$ cd $WORKDIR
$ git clone https://github.com/2gis/k8s-handle-example.git
$ cd k8s-handle-example
$ docker run --rm -v $(pwd):/tmp/ -v "$HOME/.kube:/root/.kube" 2gis/k8s-handle k8s-handle deploy -s staging --use-kubeconfig
INFO:templating:File "/tmp/k8s-handle/configmap.yaml" successfully generated
INFO:templating:Trying to generate file from template "secret.yaml.j2" in "/tmp/k8s-handle"
INFO:templating:File "/tmp/k8s-handle/secret.yaml" successfully generated
INFO:templating:Trying to generate file from template "deployment.yaml.j2" in "/tmp/k8s-handle"
INFO:templating:File "/tmp/k8s-handle/deployment.yaml" successfully generated
INFO:k8s.resource:ConfigMap "k8s-starter-kit-nginx-conf" already exists, replace it
INFO:k8s.resource:Secret "k8s-starter-kit-secret" already exists, replace it
INFO:k8s.resource:Deployment "k8s-starter-kit" does not exist, create it

                         _(_)_                          wWWWw   _
             @@@@       (_)@(_)   vVVVv     _     @@@@  (___) _(_)_
            @@()@@ wWWWw  (_)\    (___)   _(_)_  @@()@@   Y  (_)@(_)
             @@@@  (___)     `|/    Y    (_)@(_)  @@@@   \|/   (_)
              /      Y       \|    \|/    /(_)    \|      |/      |
           \ |     \ |/       | / \ | /  \|/       |/    \|      \|/
            \|//    \|///    \|//  \|/// \|///    \|//    |//    \|//
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Using with CI/CD tools

If you using Gitlab CI, TeamCity or something else, you can use docker runner/agent, script will slightly different:

$ k8s-handle deploy -s staging

Configure checkout for https://github.com/2gis/k8s-handle-example.git and specific branch without-kubeconfig Also you need to setup next env vars:

  • K8S_NAMESPACE
  • K8S_MASTER_URI
  • K8S_CA_BASE64
  • K8S_TOKEN

use image 2gis/k8s-handle:

Notice: If you use Gitlab CI, you can configure Kubernetes integration and just use --use-kubeconfig flag.

Usage

$ python k8s-handle.py deploy -s staging --use-kubeconfig
INFO:templating:Trying to generate file from template "configmap.yaml.j2" in "/tmp/k8s-handle"
INFO:templating:File "/tmp/k8s-handle/configmap.yaml" successfully generated
INFO:templating:Trying to generate file from template "secret.yaml.j2" in "/tmp/k8s-handle"
INFO:templating:File "/tmp/k8s-handle/secret.yaml" successfully generated
INFO:templating:Trying to generate file from template "deployment.yaml.j2" in "/tmp/k8s-handle"
INFO:templating:File "/tmp/k8s-handle/deployment.yaml" successfully generated
INFO:k8s.resource:ConfigMap "k8s-starter-kit-nginx-conf" already exists, replace it
INFO:k8s.resource:Secret "k8s-starter-kit-secret" already exists, replace it
INFO:k8s.resource:Deployment "k8s-starter-kit" does not exist, create it

                         _(_)_                          wWWWw   _
             @@@@       (_)@(_)   vVVVv     _     @@@@  (___) _(_)_
            @@()@@ wWWWw  (_)\    (___)   _(_)_  @@()@@   Y  (_)@(_)
             @@@@  (___)     `|/    Y    (_)@(_)  @@@@   \|/   (_)
              /      Y       \|    \|/    /(_)    \|      |/      |
           \ |     \ |/       | / \ | /  \|/       |/    \|      \|/
            \|//    \|///    \|//  \|/// \|///    \|//    |//    \|//
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
$ kubectl get configmap 
NAME                         DATA      AGE
k8s-starter-kit-nginx-conf   1         1m
$ kubectl get secret | grep starter-kit
k8s-starter-kit-secret   Opaque                                1         1m
$ kubectl get deploy
NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
k8s-starter-kit   1         1         1            1           3m

Now set replicas_count in config.yaml to 3, and run again in sync mode

$ python k8s-handle.py deploy -s staging --use-kubeconfig --sync-mode
...
INFO:k8s.resource:Deployment "k8s-starter-kit" already exists, replace it
INFO:k8s.resource:desiredReplicas = 3, updatedReplicas = 3, availableReplicas = 1
INFO:k8s.resource:Deployment not completed on 1 attempt, next attempt in 5 sec.
INFO:k8s.resource:desiredReplicas = 3, updatedReplicas = 3, availableReplicas = 2
INFO:k8s.resource:Deployment not completed on 2 attempt, next attempt in 5 sec.
INFO:k8s.resource:desiredReplicas = 3, updatedReplicas = 3, availableReplicas = 3
INFO:k8s.resource:Deployment completed on 3 attempt
...
$ kubectl get deploy
NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
k8s-starter-kit   3         3         3            3           7m

Example

You can start by example https://github.com/2gis/k8s-handle-example. There are nginx with index.html and all needed kubernetes resources for deploy them.

$ cd $WORKDIR
$ git clone https://github.com/2gis/k8s-handle-example.git
$ cd k8s-handle-example
$ IMAGE_VERSION=latest k8s-handle-os deploy -s staging --use-kubeconfig --sync-mode
INFO:__main__:Using default namespace k8s-handle-test
INFO:templating:Trying to generate file from template "configmap.yaml.j2" in "/tmp/k8s-handle"
INFO:templating:File "/tmp/k8s-handle/configmap.yaml" successfully generated
INFO:templating:Trying to generate file from template "deployment.yaml.j2" in "/tmp/k8s-handle"
INFO:templating:File "/tmp/k8s-handle/deployment.yaml" successfully generated
INFO:templating:Trying to generate file from template "service.yaml.j2" in "/tmp/k8s-handle"
INFO:templating:File "/tmp/k8s-handle/service.yaml" successfully generated
INFO:k8s.resource:ConfigMap "example-nginx-conf" does not exist, create it
INFO:k8s.resource:Deployment "example" does not exist, create it
INFO:k8s.resource:desiredReplicas = 1, updatedReplicas = 1, availableReplicas = None
INFO:k8s.resource:Deployment not completed on 1 attempt, next attempt in 5 sec.
INFO:k8s.resource:desiredReplicas = 1, updatedReplicas = 1, availableReplicas = None
INFO:k8s.resource:Deployment not completed on 2 attempt, next attempt in 5 sec.
INFO:k8s.resource:desiredReplicas = 1, updatedReplicas = 1, availableReplicas = 1
INFO:k8s.resource:Deployment completed on 3 attempt
INFO:k8s.resource:Service "example" does not exist, create it
$ kubectl -n k8s-handle-test get svc 
NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
example   NodePort   10.100.132.168   <none>        80:31153/TCP   52s
$ curl http://<any node>:31153
<h1>Hello world!</h1>
Deployed with k8s-handle.

Docs

Configuration structure

k8s-handle work with 2 components:

  • conifg.yaml (or any other yaml file through -c argument) store all configuration for deploy
  • templates catalog, where your can store all required templates for kubernetes resource files (can be changed through TEMPLATES_DIR env var)

Environments

If your have testing, staging, production-zone-1, production-zone-2, etc, you can easily cover all environments with one set of templates for your application without duplication.

Common section

In common section you can specify any variables you want to be common:

common:
    app_name: my-shiny-app
    app_port: 8080

Both of this example variables are common and don't change between environments

Any other sections

Let's specify testing environment

testing:
    replicas: 1
    request_cpu: 100m 
    request_memory: 128M
    some_option: disabled

In testing in most cases we don't want performance from our application so we can keep 1 replica and small amount of resources for it. Also you can set some options to disabled state, in case when you don't want to affect any integrated systems during testing during testing.

staging:
    replicas: 2
    request_cpu: 200m 
    request_memory: 512M

Some teams use staging for integration and demo, so we can increase replicas and resources for our service.

production-zone-1:
    replicas: 50
    request_cpu: 1000m
    request_memory: 1G
    production: "true"
    never_give_up: "true"

In production we need to process n thousand RPS, so set replicas to 50, increase resources and set all production variables to ready for anything values.

Deploy specific environment

In your CI/CD script you can deploy any environment

$ k8s-handle deploy -s staging # Or testing or production-zone-1

In Gitlab CI for example you can create manual job for each environment

Templates

Templates in k8s-handle use jinja2 syntax and support all standard filters + some special:

  • {{ my_var | b64encode }} - encode value of my_var to base64
  • {{ my_var | b64decode }} - decode value of my_var from base64

Warning: You can use filters only for templates and can't for config.yaml

You can put *.j2 templates in 'templates' directory and specify it in config.yaml

testing:
    replicas: 1
    request_cpu: 100m 
    request_memory: 128M
    some_option: disabled
    templates:
    - template: my-deployment.yaml.j2

the same template you can use in each section you want:

staging:
    ...
    templates:
    - template: my-deployment.yaml.j2
    
production-zone-1:
  ...
  templates:
  - template: my-deployment.yaml.j2

Variables

Merging with common

All variables defined in common merged with deployed section and available as context dict in templates rendering, for example:

common:
  common_var: common_value 
testing:
  testing_variable: testing_value

After rendering this template some-file.txt.j2:

common_var = {{ common_var }}
testing_variable = {{ testing_variable }}

will be generated file some-file.txt with content:

common_var = common_value
testing_variable = testing_value

Load variables from environment

If you want to use environment variables in your templates(for docker image tag generated by build for example), you can use next construction in config.yaml:

common:
  image_version: "{{ env='TAG' }}"

Load variables from yaml file

common:
  test: "{{ file='include.yaml' }}"

include.yaml:

- 1
- 2 
- 3

template:

{{ test[0] }}
{{ test[1] }}
{{ test[2] }}

After rendering you get:

1
2
3

How to use in CI/CD

Gitlab CI

Native integration

Use Gitlab CI integration with Kubernetes (https://docs.gitlab.com/ee/user/project/clusters/index.html#adding-an-existing-kubernetes-cluster) .gitlab-ci.yaml:

deploy:
  image: 2gis/k8s-handle:latest
  script:
    - k8s-handle deploy --section <section_name> --use-kubeconfig

Through variables

Alternatively you can setup Gitlab CI variables:

  • K8S_TOKEN_STAGING = < serviceaccount token for staging >
  • K8S_TOKEN_PRODUCTION = < serviceaccount token for production >

Don't forget mark variables as protected

then add next lines to config.yaml

staging:
  k8s_master_uri: <kubenetes staging master uri>
  k8s_token: "{{ env='K8S_TOKEN_STAGING' }}"
  k8s_ca_base64: <kubernetes staging ca>
  
production:
  k8s_master_uri: <kubenetes production master uri>
  k8s_token: "{{ env='K8S_TOKEN_PRODUCTION' }}"
  k8s_ca_base64: <kubernetes production ca>

Now just run proper gitlab job(without --use-kubeconfig option):

deploy:
  image: 2gis/k8s-handle:latest
  script:
    - k8s-handle deploy --section <section_name>

Working modes

Dry run

If you want check templates generation and not apply changes to kubernetes use --dry-run function.

$ k8s-handle deploy -s staging --use-kubeconfig --dry-run
INFO:__main__:Using default namespace k8s-handle-test
INFO:templating:Trying to generate file from template "configmap.yaml.j2" in "/tmp/k8s-handle"
INFO:templating:File "/tmp/k8s-handle/configmap.yaml" successfully generated
INFO:templating:Trying to generate file from template "deployment.yaml.j2" in "/tmp/k8s-handle"
INFO:templating:File "/tmp/k8s-handle/deployment.yaml" successfully generated
INFO:templating:Trying to generate file from template "service.yaml.j2" in "/tmp/k8s-handle"
INFO:templating:File "/tmp/k8s-handle/service.yaml" successfully generated
$ cat /tmp/k8s-handle/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: example
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
  selector:
    app: example 

Sync mode

Works only with Deployment, Job, StatefulSet and DaemonSet

By default k8s-handle just apply resources to kubernetes and exit. In sync mode k8s-handle wait for resources up and running

$ k8s-handle deploy --section staging  --sync-mode
...
INFO:k8s.resource:Deployment "k8s-starter-kit" already exists, replace it
INFO:k8s.resource:desiredReplicas = 3, updatedReplicas = 3, availableReplicas = 1
INFO:k8s.resource:Deployment not completed on 1 attempt, next attempt in 5 sec.
INFO:k8s.resource:desiredReplicas = 3, updatedReplicas = 3, availableReplicas = 2
INFO:k8s.resource:Deployment not completed on 2 attempt, next attempt in 5 sec.
INFO:k8s.resource:desiredReplicas = 3, updatedReplicas = 3, availableReplicas = 3
INFO:k8s.resource:Deployment completed on 3 attempt
...

You can specify number of tries before k8s-handle exit with non zero exit code and delay before checks:

--tries <tries> (360 by default)
--retry-delay <retry-delay in seconds> (5 by default)

Strict mode

In some cases k8s-handle warn you about ambiguous situations and keep working. With --strict mode k8s-handle warn and exit with non zero code. For example when some used environment variables is empty.

$ k8s-handle-os deploy -s staging --use-kubeconfig --strict
ERROR:__main__:RuntimeError: Environment variable "IMAGE_VERSION" is not set
$ echo $?
1

Destroy

In some cases you need to destroy early created resources(demo env, deploy from git branches, testing etc.), k8s-handle support destroy subcommand for you. Just use destroy instead of deploy. k8s-handle process destroy as deploy, but call delete kubernetes api calls instead of create or replace.

Sync mode available for destroy too