feat: implementation of simple alerting interface (#70)

Co-authored-by: Philipp Belitz <philipp.belitz@securesystems.de>
Co-authored-by: annekebr <44376590+annekebr@users.noreply.github.com>

This allows to configure external entities to receive notifications either if connaisseur admits or if it rejects a deployment. There are templates for Slack, OpsGenie as well as Keybase, but it is built to allow custom extensions easily. Notifications are implemented as simple POST requests with a configurable JSON payload and extendable headers.

Co-authored-by: Philipp Belitz <philipp.belitz@securesystems.de>

.github/workflows/cicd.yaml

@@ -161,6 +161,11 @@ jobs:
   integration-test:
     runs-on: ubuntu-latest
     needs: [build]
+    services:
+      alerting-endpoint:
+        image: securesystemsengineering/alerting-endpoint
+        ports:
+          - 56243:56243
     steps:
       - uses: actions/checkout@v2
       - uses: actions/download-artifact@v2
@@ -185,6 +190,9 @@ jobs:
         run: |
           kind load docker-image $(yq e '.deployment.image' helm/values.yaml)
           kind load docker-image $(yq e '.deployment.helmHookImage' helm/values.yaml)
-
       - name: Run actual integration test
-        run: bash connaisseur/tests/integration/integration-test.sh
+        run: |
+          SERVICE_CONTAINER_ID=$(docker container ls --no-trunc --format "{{json . }}" | jq ' . | select(.Image|match("alerting-endpoint"))' | jq -r .ID)
+          docker network connect kind ${SERVICE_CONTAINER_ID}
+          export ALERTING_ENDPOINT_IP=$(docker network inspect kind | jq -r --arg container_id ${SERVICE_CONTAINER_ID}  '.[].Containers[$container_id].IPv4Address' | sed s+/.*++g)
+          bash connaisseur/tests/integration/integration-test.sh

.github/workflows/dockerhub-check.yml

@@ -17,3 +17,5 @@ jobs:
         run: DOCKER_CONTENT_TRUST=1 docker pull docker.io/securesystemsengineering/testimage:signed
       - name: Check unsigned test image
         run: DOCKER_CONTENT_TRUST=0 docker pull docker.io/securesystemsengineering/testimage:unsigned
+      - name: Check alerting endpoint image
+        run: DOCKER_CONTENT_TRUST=0 docker pull docker.io/securesystemsengineering/alerting-endpoint

.gitignore

@@ -18,3 +18,5 @@ samples/*
 demo/
 tls-*.conf
 .venv/
+venv/
+.idea/

.pylintrc

@@ -150,7 +150,6 @@ disable=print-statement,
         W0703,
         W0511,
 
-
 # Enable the message, report, category or checker with the given id(s). You can
 # either give multiple identifier separated by comma (,) or put this option
 # multiple time (only on the command line, not in the configuration file where

Makefile

@@ -1,6 +1,7 @@
 NAMESPACE = connaisseur
 IMAGE := $(shell yq e '.deployment.image' helm/values.yaml)
 HELM_HOOK_IMAGE := $(shell yq e '.deployment.helmHookImage' helm/values.yaml)
+CLUSTER := $(shell CONTEXT=`kubectl config current-context` && kubectl config view -ojson | jq --arg CONTEXT $$CONTEXT '.contexts[] | select(.name==$$CONTEXT) | .context.cluster')
 
 .PHONY: all docker certs install unistall upgrade annihilate
 
@@ -23,7 +24,7 @@ install: certs
 	#
 	#=============================================
 	#
-	helm install connaisseur helm --atomic
+	helm install connaisseur helm --atomic --set alerting.cluster=$(CLUSTER)
 
 uninstall:
 	kubectl config set-context --current --namespace $(NAMESPACE)
@@ -32,7 +33,7 @@ uninstall:
 
 upgrade:
 	kubectl config set-context --current --namespace $(NAMESPACE)
-	helm upgrade connaisseur helm --wait
+	helm upgrade connaisseur helm --wait --set alerting.cluster=$(CLUSTER)
 
 annihilate:
 	kubectl delete all,mutatingwebhookconfigurations,clusterroles,clusterrolebindings,configmaps,imagepolicies -lapp.kubernetes.io/instance=connaisseur

README.md

@@ -17,6 +17,7 @@ Connaisseur is an admission controller for Kubernetes that integrates Image Sign
   * [Makefile and Helm](#makefile-and-helm)
   * [Image Policy](#image-policy)
   * [Detection Mode](#detection-mode)
+  * [Alerting](#alerting)
 - [Threat Model](#threat-model)
   * [(1) Developer/User](#-1--developer-user)
   * [(2) Connaisseur Service](#-2--connaisseur-service)
@@ -140,6 +141,38 @@ A detection mode is available in order to avoid interruptions of a running clust
 
 To activate the detection mode, set the `detection_mode` flag to `true` in `helm/values.yaml`.
 
+### Alerting
+
+You can receive notification on the admission decisions made by Connaisseur on basically every REST endpoint that accepts JSON payload. For Slack, Opsgenie and Keybase we have preconfigured payloads that are ready to use, but you can also use the existing payload templates as an example how to create your own custom one. It is also possible to configure multiple different interfaces you want to receive alerts at for each event category (admit/deny) in the `helm/values.yaml`.
+
+If you for example would like to receive notifications in Keybase whenever Connaisseur admits a request to your cluster, your alerting configuration would look similar to to following snippet:
+
+
+```
+alerting:
+  admit_request:
+    templates:
+      - template: keybase
+        receiver_url: https://bots.keybase.io/webhookbot/<Your-Keybase-Hook-Token>
+```
+
+For each element in `templates`, both `template` as well as `receiver_url` are required. The value for `template` needs to match an existing file of the pattern `helm/alert_payload_templates/<template>.json`; so if you want to use a predefined one it needs to be one of `slack`, `keybase` or `opsgenie`. For some REST endpoints like e.g. Opsgenie you need to configure a additional headers which you can pass to `custom_headers` as a list of plain string like e.g. `["Authorization: GenieKey <Your-Genie-Key>"]`. Setting `fail_if_alert_sending_fails` to `True` will make Connaisseur deny images if the corresponding alert cannot be successfully sent. This, obviously, makes sense only for requests that Connaisseur would have admitted as other requests would have been denied in the first place. The setting can come handy if you want to run Connaisseur in detection mode but still make sure that you get notified about what is going on in your cluster. *However, this setting will block everyone from contributing if the alert sending fails permanently. It could occur that your developers are blocked if the third party interface is down because somebody accidentally deleted your Slack Webhook App or your configuration has expired because your GenieKey expired!*
+
+| key                                                |  accepted values                                      | required               |
+| -------------------------------------------------- | ----------------------------------------------------  | ---------------------  |
+| `alerting.<category>.template`                     | `opsgenie`, `slack`, `keybase` or custom <sup>*</sup> | yes                    |
+| `alerting.<category>.receiver_url`                 | string                                                | yes                    |
+| `alerting.<category>.priority`                     | int                                                   | no (default: `3`)      |
+| `alerting.<category>.custom_headers`               | list[string]                                          | no                     |
+| `alerting.<category>.payload_fields`               | subyaml                                               | no                     |
+| `alerting.<category>.fail_if_alert_sending_fails`  | bool                                                  | no (default: `False`)  |
+
+<sup>* the basename of a custom template file in `helm/alerting_payload_templates` without file extension </sup>
+
+Along the lines of the templates that are already there you can easily define your custom template for your own endpoint. The following variables can be rendered during runtime into your payload: `alert_message`, `priority`, `connaisseur_pod_id`, `cluster`, `timestamp`, `request_id` and `images`. Rendering is done by Jinja2; so if you want to have a timestamp in your payload later on you can use `{{ timestamp }}` in your template. You can update your payload dynamically by adding payload fields in `yaml` presentation in the `payload_fields` key which will be translated to JSON by helm as is. If your REST endpoint requires particular headers, your can specify them as described above in `custom_headers`.
+
+Feel free to open a PR if you add new neat templates for other third parties!
+
 ## Threat Model
 
 The STRIDE threat model has been used as a reference for threat modeling. Each of the STRIDE threats were matched to all entities relevant to Connaisseur, including Connaisseur itself. A description of how a threat on an entity manifests itself is given as well as a possible counter measure.

connaisseur/alert.py

@@ -0,0 +1,185 @@
+import json
+import logging
+import os
+from datetime import datetime
+
+import requests
+from jinja2 import Template, StrictUndefined
+from jsonschema import validate as json_validate
+
+from connaisseur.util import safe_path_func
+from connaisseur.exceptions import AlertSendingError, ConfigurationError
+from connaisseur.image import Image
+from connaisseur.mutate import get_container_specs
+
+
+class Alert:
+    """
+    Class to store image information about an alert as attributes and a sending functionality as method.
+    Alert Sending can, depending on the configuration, throw an AlertSendingError causing Connaisseur
+    responding with status code 500 to the request that was sent for admission control,
+    causing a Kubernetes Error event.
+    """
+
+    template: str
+    receiver_url: str
+    payload: dict
+    headers: dict
+
+    context: dict
+    admission_request: dict
+    throw_if_alert_sending_fails: bool
+
+    def __init__(self, alert_message, receiver_config, admission_request):
+        self.context = {
+            "alert_message": alert_message,
+            "priority": str(receiver_config.get("priority", 3)),
+            "connaisseur_pod_id": os.getenv("POD_NAME"),
+            "cluster": os.getenv("CLUSTER_NAME"),
+            "timestamp": datetime.now(),
+            "request_id": admission_request.get("request", {}).get(
+                "uid", "No given UID"
+            ),
+            "images": (str(get_images(admission_request)) or "No given images"),
+        }
+        self.admission_request = admission_request
+        self.receiver_url = receiver_config["receiver_url"]
+        self.template = receiver_config["template"]
+        self.throw_if_alert_sending_fails = receiver_config.get(
+            "fail_if_alert_sending_fails", False
+        )
+        self.payload = self._construct_payload(receiver_config)
+        self.headers = self._get_headers(receiver_config)
+
+    def _construct_payload(self, receiver_config):
+        try:
+            alert_templates_dir = f'{os.getenv("ALERT_CONFIG_DIR")}/templates'
+            with safe_path_func(
+                open,
+                alert_templates_dir,
+                f"{alert_templates_dir}/{self.template}.json",
+                "r",
+            ) as templatefile:
+                template = json.load(templatefile)
+        except Exception as err:
+            raise ConfigurationError(
+                "Template file for alerting payload is either missing or invalid JSON: {}".format(
+                    str(err)
+                )
+            ) from err
+        payload = self._render_template(template)
+        if receiver_config.get("payload_fields") is not None:
+            payload.update(receiver_config.get("payload_fields"))
+        return json.dumps(payload)
+
+    def _render_template(self, template):
+        if isinstance(template, dict):
+            for key in template.keys():
+                template[key] = self._render_template(template[key])
+        elif isinstance(template, list):
+            template[:] = [self._render_template(entry) for entry in template]
+        elif isinstance(template, str):
+            template = Template(template).render(
+                self.context, undefined=StrictUndefined
+            )
+        return template
+
+    def send_alert(self):
+        try:
+            response = requests.post(
+                self.receiver_url, data=self.payload, headers=self.headers
+            )
+            response.raise_for_status()
+            logging.info("sent alert to %s", self.template)
+        except Exception as err:
+            if self.throw_if_alert_sending_fails:
+                raise AlertSendingError(str(err)) from err
+            logging.error(err)
+        return response
+
+    @staticmethod
+    def _get_headers(receiver_config):
+        headers = {"Content-Type": "application/json"}
+        additional_headers = receiver_config.get("custom_headers")
+        if additional_headers is not None:
+            for header in additional_headers:
+                key, value = header.split(":", 1)
+                headers.update({key.strip(): value.strip()})
+        return headers
+
+
+def load_config():
+    try:
+        alert_config_dir = f'{os.getenv("ALERT_CONFIG_DIR")}'
+        with safe_path_func(
+            open, alert_config_dir, f"{alert_config_dir}/alertconfig.json", "r"
+        ) as configfile:
+            alertconfig = json.load(configfile)
+        schema = get_alert_config_validation_schema()
+        json_validate(instance=alertconfig, schema=schema)
+    except Exception as err:
+        raise ConfigurationError(
+            "Alerting configuration file either not present or not valid."
+            "Check in the 'helm/values.yml' whether everything is correctly configured. {}".format(
+                str(err)
+            )
+        ) from err
+    return alertconfig
+
+
+def get_images(admission_request):
+    relevant_spec = get_container_specs(
+        admission_request.get("request", {}).get("object", {})
+    )
+    return list(map(lambda x: x.get("image"), relevant_spec))
+
+
+def send_alerts(admission_request, *, admitted, reason=None):
+    alert_config = load_config()
+    event_category = "admit_request" if admitted else "reject_request"
+    if alert_config.get(event_category) is not None:
+        for receiver in alert_config[event_category]["templates"]:
+            message = (
+                "CONNAISSEUR admitted a request."
+                if admitted
+                else "CONNAISSEUR rejected a request: {}".format(reason)
+            )
+            alert = Alert(message, receiver, admission_request)
+            alert.send_alert()
+
+
+def call_alerting_on_request(admission_request, *, admitted):
+    normalized_hook_image = Image(os.getenv("HELM_HOOK_IMAGE"))
+    hook_image = "{}/{}/{}:{}".format(
+        normalized_hook_image.registry,
+        normalized_hook_image.repository,
+        normalized_hook_image.name,
+        normalized_hook_image.tag,
+    )
+    images = []
+    for image in get_images(admission_request):
+        normalized_image = Image(image)
+        images.append(
+            "{}/{}/{}:{}".format(
+                normalized_image.registry,
+                normalized_image.repository,
+                normalized_image.name,
+                normalized_image.tag,
+            )
+        )
+    if images == [hook_image]:
+        return False
+    return not no_alerting_configured_for_event(admitted)
+
+
+def no_alerting_configured_for_event(admitted):
+    config = load_config()
+    templates = (
+        config.get("admit_request") if admitted else config.get("reject_request")
+    )
+    return templates is None
+
+
+def get_alert_config_validation_schema():
+    with open("connaisseur/res/alertconfig_schema.json") as schemafile:
+        return json.load(schemafile)

connaisseur/exceptions.py

@@ -55,3 +55,23 @@ class UnknownVersionError(Exception):
 
 class AmbiguousDigestError(BaseConnaisseurException):
     pass
+
+
+class AlertingException(Exception):
+
+    message: str
+
+    def __init__(self, message: str):
+        self.message = message
+        super().__init__()
+
+    def __str__(self):
+        return str(self.__dict__)
+
+
+class ConfigurationError(AlertingException):
+    pass
+
+
+class AlertSendingError(AlertingException):
+    pass