Skip to content

Commit

Permalink
Provide a working helm chart (#1)
Browse files Browse the repository at this point in the history
* Make helm chart runnable

* Fix envoy endpoint

* init Dashboard

* Initialized grafana dashboards

* Extract istio envoy plugin in own helm-chart

* Add release chart workflow

* Change version

* test

* Add  CR_SKIP_EXISTING: true

* Added grafana provisioning setup

* Added the grafana configuration to the helm chart.

* Removed legacy grafana config files

* Prepare for merge: Moved hawk service definition to deployment

* Push to other repository

* Change to master and set target directory to charts/

* Added all dashboards from the report and created config map.

* Extended Readme and added Screenshots

* Revert "Resolved merge confict"

This reverts commit 602f512, reversing
changes made to 3e15175.

* Fixed helm deployment

* Cleaned up grafana folder and extended the readme

* Added Grafana to .helmignore

* Realigned with working commit c0f4422

* Update README and add submodule for integrations

* Add grafana configMap
* Add `API_URL` to hawk-monitor

---------

Co-authored-by: ciphersmaug <39407479+ciphersmaug@users.noreply.github.com>
  • Loading branch information
tomijange and curiousjaki committed Oct 17, 2023
1 parent 290c07a commit 400e557
Show file tree
Hide file tree
Showing 41 changed files with 4,919 additions and 551 deletions.
67 changes: 27 additions & 40 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
@@ -1,49 +1,36 @@
name: Build and deploy to GitHub Packages
name: Commit to Helm Charts Repository

env:
CHART_NAME: "hawk"


on:
push:
tags:
- '*.*.*'
branches:
- master

jobs:
build:
push-to-repo:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Checkout source code
uses: actions/checkout@v2

- name: Helm Installation
uses: azure/setup-helm@v1.1
with:
version: v3.7.0
- name: Package Helm Chart
run: helm package .
- name: Index packaged Helm chart
run: helm repo index .
- name: Commit and push to another repository
run: |
mkdir -p hawk-helm-charts/${{ env.CHART_NAME }}
cp -r README.md values.yaml Chart.yaml Chart.lock templates .helmignore "hawk-helm-charts/${{ env.CHART_NAME }}"
- name: Create Release
id: create_release
uses: actions/create-release@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
tag_name: ${{ github.ref }}
release_name: Hawk Helm Chart ${{ github.ref }}
draft: false
prerelease: true
- name: Upload Package to Release
uses: svenstaro/upload-release-action@v2
with:
repo_token: ${{ secrets.GITHUB_TOKEN }}
file: '*.tgz'
tag: ${{ github.ref }}
overwrite: true
file_glob: true
- name: Upload Index to Release
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: index.yaml
asset_name: index.yaml
asset_content_type: application/x-yaml
- name: Pushes to public repository
id: push_directory
uses: cpina/github-action-push-to-another-repository@ssh-deploy-key
env:
SSH_DEPLOY_KEY: ${{ secrets.SSH_DEPLOY_KEY }}
with:
source-directory: hawk-helm-charts/${{ env.CHART_NAME }}
destination-github-username: 'PrivacyEngineering'
destination-repository-name: 'hawk-helm-charts'
user-email: $GITHUB_ACTOR@users.noreply.github.com
commit-message: pushed from $GITHUB_REF
target-directory: charts/${{ env.CHART_NAME }}
target-branch: master
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
.idea
.idea

charts/
4 changes: 4 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[submodule "integrations/hawk-envoy-plugin"]
path = integrations/hawk-envoy-plugin
url = https://github.com/PrivacyEngineering/hawk-envoy-plugin.git
branch = v2
1 change: 1 addition & 0 deletions .helmignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,4 @@
.idea/
*.tmproj
.vscode/
grafana/
6 changes: 6 additions & 0 deletions Chart.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
dependencies:
- name: postgresql
repository: https://charts.bitnami.com/bitnami
version: 12.6.6
digest: sha256:eea00e724f08def137f8205b5a6170559a2ae6d8205be5cf3a713f8d1978c6ba
generated: "2023-07-18T17:26:54.029647604+02:00"
8 changes: 7 additions & 1 deletion Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,16 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 1.0.1
version: 1.0.3

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "1.16.0"


dependencies:
- name: postgresql
repository: https://charts.bitnami.com/bitnami
version: 12.6.6
File renamed without changes.
99 changes: 87 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,33 @@

![workflow](https://github.com/PrivacyEngineering/hawk/actions/workflows/main.yml/badge.svg)


Hawk has been published at the 16th IEEE International Conference on Cloud Computing 2023, IEEE Cloud 2023.
Please find its publication here: https://arxiv.org/abs/2306.02496

## BibTex citation:
```
@misc{grünewald2023hawk,
title={Hawk: DevOps-driven Transparency and Accountability in Cloud Native Systems},
author={Elias Grünewald and Jannis Kiesel and Siar-Remzi Akbayin and Frank Pallas},
year={2023},
eprint={2306.02496},
archivePrefix={arXiv},
primaryClass={cs.DC}
}
```

## Overview

The Hawk Framework provides a way of tracking the dataflow between applications and allows for GDPR
related tags to be added to the data references. It also features an analytical dashboard about the
GDPR related information and integration for using the ratio of GDPR-tagged data in e.g. Flagger
Canary releases.

## Concept
<details>
<summary>Concept and Goal</summary>

### Concept

The concept is to archive this goal is to intercept the traffic between the individual applications
/ services. This idea is called Hawk Core. It can be either done by (A) Framework Integration inside
Expand All @@ -16,7 +37,7 @@ the Framework integration allows to interact with the Hawk API directly inside t
the possibility to intercept encrypted and also external traffic, the application itself must be
modified. The Service Mesh solution can be installed without modifying any application. Both
solutions can be active in parallel. Currently the only integrations
are [EnvoyProxy / Istio Service Mesh Integration]() and [Java Framework Integration]() for HTTP and
are [EnvoyProxy / Istio Service Mesh Integration](integrations/hawk-envoy-plugin/README.md) and [Java Framework Integration]() for HTTP and
JSON bodies only.

When a Packet is intercepted it will be parsed, according to the protocol used. The parsing searches
Expand Down Expand Up @@ -62,31 +83,60 @@ of a service is changed. These changes can be then update in the Hawk Core Monit
Hawk Release can constantly validate the coverage of mapped endpoints to prevent deploying unmapped
endpoints.

## Problems solved
### Goal

The Hawk Framework helps the company to be compliant with the GDPR, to avoid fines. The data
protection officer can use this software to keep updated about privacy related information and
The Hawk Framework helps businesses to be compliant with the GDPR, to avoid fines. The data
protection officer can use this software to stay updated about privacy related information and
change the privacy policy accordingly.

## Deployment Guide
</details>

### Hawk Core
## Quickstart

### Deployment through Helm

1. Add the helm chart repository:
```
helm repo add hawk https://privacyengineering.github.io/hawk-helm-charts/
```
2. Modify values in [`values.yaml`](values.yaml) to your needs.
3. Install hawk core and all it's services:
```
helm dependency update
helm install hawk hawk/hawk --namespace hawk --create-namespace
```
4. Access the hawk-core-monitor and hawk-service via ingress:
```
kubectl get ingress -n hawk
```
5. Add an integration to the hawk framework (see [Integrations](#integrations) for more information)

### Deloyment alternatives

This repository feature a Helm Chart which can be used to install this software in a Kubernetes
environment using Helm. It's also possible to install the application in a Non-Kubernetes
It's also possible to install the application in a Non-Kubernetes
environment or configuring them more individual using the Docker Images of them. The Istio / Envoy
integration is only available in specific Kubernetes environments.
The [Java integration](https://github.com/PrivacyEngineering/hawk-integration-java) is available in
every environment. It needs a connection to the Hawk Service. When possible, the Envoy Integration
is preferred as it's less effort to install. You must choose at least one integration.

#### Helm
## Integrations

The Hawk Framework can be extended through integrations. Currently there are two integrations (for HTTP and JSON bodies only):
* [EnvoyProxy / Istio Service Mesh Integration](integrations/hawk-envoy-plugin/README.md)
* [Java Framework Integration]()

Both integrations communicate with the hawk-service via the exposed REST API.

<details><summary> Hawk Core</summary>

### Hawk Core

The Helm Chart (WIP), installs the Hawk-Service, a default PostgreSQL database, Hawk Core Monitor (
nginx + monitor + grafana) and the Istio / Envoy integration if selected.

```shell
helm repo add https://github.com/PrivacyEngineering/hawk/releases/download/1.0.1
helm repo add hawk https://github.com/PrivacyEngineering/hawk/releases/download/1.0.1
helm install PrviacyEngineering/hawk
```

Expand Down Expand Up @@ -124,13 +174,21 @@ two (or three) components. See [Nginx Deployment](templates/nginx-deployment.yam
and [Nginx Config](templates/nginx-config.yaml) for information on which environment variables and
which files to provide.

</details>

<details><summary>Hawk Release</summary>

### Hawk Release

To enable Hawk Release, you have to install [Flux](https://github.com/fluxcd/flux)
and [Flagger](https://github.com/fluxcd/flagger). Then you can configure to use the Metrics using
Prometheus, see [Hawk Service](https://github.com/PrivacyEngineering/hawk-service) for more
information on which mappings to use. You also need to configure Prometheus to scrape the Metrics.

</details>

<details><summary>Hawk Build</summary>

### Hawk Build

To enable Hawk Build you have to install and configure
Expand All @@ -139,8 +197,25 @@ the [OpenAPI Privacy Changes Service](https://github.com/TUB-CNPE-TB/openapi-pri
use [OpenAPI Privacy Alert GitHub Action](https://github.com/TUB-CNPE-TB/openapi-privacy-alert-action)
.

## Example
</details>

## Example Deployment

An example using the [WeaveWorks SockShop](https://github.com/microservices-demo/microservices-demo)
, integrated with some of Hawk components can be
found [here](https://github.com/PrivacyEngine/hawk-sockshop).


## Hawk Grafana Dashboard Evaluation

We provide 4 grafana dashboards:

- Dashboard
- Service Graph
- Field Details
- Endpoint Details

A detailed explanation of the dashboards can be found [here](./grafana/README.md).

### Hawk Monitor Overview Dashboard
[![Dashboard overview with four panels](./grafana/images/overview.png "Dashboard overview with four distinct panels")](./grafana/README.md)
33 changes: 33 additions & 0 deletions grafana/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Hawk Grafana Monitor

The Hawk grafana dashboards enable capable privacy insights.

We provide four dashboards in the hawk directory:
- Dashboard
- Service Graph
- Field Details
- Endpoint Details

![Grafana folder structure screenshot](./images/folder-structure.png)

## Dashboard screenshots
### Overview
This dashboard acts as a general overview over all hawk information.
![Overview dashboard screenshot](./images/overview.png)

### Service Graph
This service graph shows all observed services and the amount of API Requests between them.
![Service Graph dashboard screenshot](./images/service-graph.png)

### Field Details
This Dashboard shows detailed information on a specified field.
Select a privacy related field on the top left.
Click on an endpoint to open a detailed endpoint overview dashboard.
![Field Details dashboard screenshot](./images/field-details.png)


### Endpoints Details
This dashboard shows the amount of requests of a specific endpoint over time.
On the top right hand side, the time frame for this analysis is specified.
On the top left, the graph can be updated for each existing endpoint.
![Endpoints Details](./images/endpoint-details.png)
Loading

0 comments on commit 400e557

Please sign in to comment.