Skip to content

Commit

Permalink
Merge pull request #109 from ccremer/stiebeleltron-stack
Browse files Browse the repository at this point in the history
Add stiebeleltron-stack chart
  • Loading branch information
ccremer authored Jan 2, 2022
2 parents 9598e47 + fc888a0 commit c7c48ba
Show file tree
Hide file tree
Showing 47 changed files with 2,834 additions and 0 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
| [![chart downloads](https://img.shields.io/github/downloads/ccremer/charts/kubernetes-zfs-provisioner-1.1.2/total)](https://github.com/ccremer/charts/releases/tag/kubernetes-zfs-provisioner-1.1.2) | [kubernetes-zfs-provisioner](charts/kubernetes-zfs-provisioner/README.md) |
| [![chart downloads](https://img.shields.io/github/downloads/ccremer/charts/samba-0.1.1/total)](https://github.com/ccremer/charts/releases/tag/samba-0.1.1) | [samba](charts/samba/README.md) |
| [![chart downloads](https://img.shields.io/github/downloads/ccremer/charts/stiebeleltron-exporter-0.1.1/total)](https://github.com/ccremer/charts/releases/tag/stiebeleltron-exporter-0.1.1) | [stiebeleltron-exporter](charts/stiebeleltron-exporter/README.md) |
| [![chart downloads](https://img.shields.io/github/downloads/ccremer/charts/stiebeleltron-stack-0.1.0/total)](https://github.com/ccremer/charts/releases/tag/stiebeleltron-stack-0.1.0) | [stiebeleltron-stack](charts/stiebeleltron-stack/README.md) |
| [![chart downloads](https://img.shields.io/github/downloads/ccremer/charts/znapzend-0.5.4/total)](https://github.com/ccremer/charts/releases/tag/znapzend-0.5.4) | [znapzend](charts/znapzend/README.md) |

## Development
Expand Down
23 changes: 23 additions & 0 deletions charts/stiebeleltron-stack/.helmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
9 changes: 9 additions & 0 deletions charts/stiebeleltron-stack/Chart.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
dependencies:
- name: stiebeleltron-exporter
repository: https://ccremer.github.io/charts
version: 0.1.0
- name: influxdb2
repository: https://helm.influxdata.com
version: 2.0.4
digest: sha256:46ad65dd0a4eca266c542dc6ccf4c9f3f7f73c34cb6d85e2888b9dbbf4500c4b
generated: "2021-12-12T16:23:18.747836615+01:00"
39 changes: 39 additions & 0 deletions charts/stiebeleltron-stack/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
apiVersion: v2
name: stiebeleltron-stack
description: A Helm chart for installing stiebeleltron-exporter with long-term storage

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
#appVersion: "1.16.0"

sources:
- https://github.com/ccremer/stiebeleltron-exporter

dependencies:
- name: stiebeleltron-exporter
version: 0.1.1
repository: https://ccremer.github.io/charts
alias: stiebeleltron
condition: stiebeleltron.enabled
- name: influxdb2
alias: influxdb
version: 2.0.4
repository: https://helm.influxdata.com
condition: influxdb.enabled
110 changes: 110 additions & 0 deletions charts/stiebeleltron-stack/README.gotmpl.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
> ⚠️ **WARNING**: See [data retention](#data-retention) first!
## About

This Helm chart installs the stiebeleltron-exporter along with long-term storage and a Grafana dashboard.
The goal is to be able to do statistics of the photovoltaic installation.

I wanted to keep stiebeleltron-exporter Helm chart minimal in case you're only interested in the exporter or you're bringing your own metrics stack.

## Features

* stiebeleltron-exporter chart
* InfluxDB 2 chart
- Post-Install hook to configure archival bucket with downsampling
* Grafana datasource
* Grafana dashboard

## Minimal required parameters

You must configure following parameters in the minimum:

```yaml
influxdb:
adminUser:
token: changeme
password: changeme

stiebeleltron:
exporter:
isgUrl: http://your.isg.device.or.ip/
telegraf:
influxdb:
token: changeme # should be the same as `influxdb.adminUser.token`!
globalTags:
site: my-home
```
## Data retention
Before installing the chart, think about data retention.
By default, the chart installs InfluxDB 2.0 with an initial bucket that keeps metrics for 14 days.
Additionally, the chart features a one-time post-install hook to create an archival bucket that keeps data forever.
It also creates a tasks that periodically downsamples the high-precision data to 10-minute-averages into the archival bucket.
This bucket is meant to hold data for years.
You can adjust retention with the following parameters:
```console
influxdb.adminUser.retention_policy
archival.*
```

> ℹ️ **NOTE**: After installation, any changes in bucket namings, retention or downsampling settings will NOT be applied anymore.
> You'd need to manually change this with InfluxDB CLI within the pod.
## Grafana integration

The chart comes with a Grafana datasource and dashboard ConfigMaps enabled by default.
Please ensure Grafana sidecar is enabled and it searches in the release namespace, or set `grafana.*.namespace`.

The dashboard features some panels that calculate per-day metrics.
Make sure the stiebeleltron ISG device, InfluxDB and Grafana dashboard are roughly in the same timezone.
The stiebeleltron ISG console includes counters like "energy per day" that reset at midnight.

## Migrating from netdata plugin

If you have used the stiebeleltron plugin that is part of netdata, you can migrate the data to the exporter variant.

As this step is done manually once, it's not automated, but here's roughly how I did it:

1. Install this chart but don't start the exporter and influxdb yet (0 replicas each).
Also disable the archival feature for now.
2. Modify the InfluxDB 2.x StatefulSet, so that you can migrate using the Docker approach.
You'd need to set `DOCKER_INFLUXDB_INIT_MODE=upgrade` and add the volume mounts from the old DB.
Set `INFLUXD_ENGINE_PATH=/var/lib/influxdb2/migration` (we appended `migration`, otherwise there will be some permission errors).
Set `INFLUXD_BOLT_PATH=/var/lib/influxdb2/migration/influxd.bolt`.
3. You can now scale the replicas to 1.
4. Once the InfluxDB 2.x is running and the old DB upgraded, enter its shell and execute the following:
```
influx backup /var/lib/influxdb2/backup
```
5. Exit shell and scale replica to 0.
6. Reset `DOCKER_INFLUXDB_INIT_MODE`, `INFLUXD_ENGINE_PATH` and `INFLUXD_BOLT_PATH` to default values and set `replicas=1`.
7. Enter shell again and execute a restore:
```
influx bucket delete --name stiebeleltron_live
influx restore /var/lib/influxdb2/backup
influx bucket create --name stiebeleltron_archive --org stiebeleltron
rm -rf /var/lib/influxdb2/migration /var/lib/influxdb2/backup
```
8. You should have now `netdata/autogen`, `stiebeleltron_live` and `stiebeleltron_archive` bucket.
The last 2 should be empty for now.
9. Migrate & downsample the raw data from `netdata/autogen` bucket using the [scripts provided](files/influxdb/migration).
You might need to edit the query a bit if you are deviating from the defaults.
You can execute the scripts with `kubectl`, e.g.
```
cat migration/<the-file>.flux | kubectl -n stiebeleltron exec -i stiebeleltron-influxdb-0 -- influx query -o stiebeleltron
```
10. Each migration script can take a few minutes.
Check the Grafana dashboard if the values make sense.
11. Scale up the exporter and verify that values are stored in `stiebeleltron_live` bucket.
12. Create the downsampling task for `stiebeleltron_archive`, check out the ConfigMap in the Helm chart that would installed on first installation.
13. Delete the `netdata/autogen` bucket via CLI.

<!---
Common/Useful Link references from values.yaml
-->
[resource-units]: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes
[prometheus-operator]: https://github.com/coreos/prometheus-operator
[prom-relabel-config]: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#relabelconfig
167 changes: 167 additions & 0 deletions charts/stiebeleltron-stack/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
# stiebeleltron-stack

![Version: 0.1.0](https://img.shields.io/badge/Version-0.1.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square)

A Helm chart for installing stiebeleltron-exporter with long-term storage

## Installation

```bash
helm repo add ccremer https://ccremer.github.io/charts
helm install stiebeleltron-stack ccremer/stiebeleltron-stack
```
> ⚠️ **WARNING**: See [data retention](#data-retention) first!
## About

This Helm chart installs the stiebeleltron-exporter along with long-term storage and a Grafana dashboard.
The goal is to be able to do statistics of the photovoltaic installation.

I wanted to keep stiebeleltron-exporter Helm chart minimal in case you're only interested in the exporter or you're bringing your own metrics stack.

## Features

* stiebeleltron-exporter chart
* InfluxDB 2 chart
- Post-Install hook to configure archival bucket with downsampling
* Grafana datasource
* Grafana dashboard

## Minimal required parameters

You must configure following parameters in the minimum:

```yaml
influxdb:
adminUser:
token: changeme
password: changeme

stiebeleltron:
exporter:
isgUrl: http://your.isg.device.or.ip/
telegraf:
influxdb:
token: changeme # should be the same as `influxdb.adminUser.token`!
globalTags:
site: my-home
```
## Data retention
Before installing the chart, think about data retention.
By default, the chart installs InfluxDB 2.0 with an initial bucket that keeps metrics for 14 days.
Additionally, the chart features a one-time post-install hook to create an archival bucket that keeps data forever.
It also creates a tasks that periodically downsamples the high-precision data to 10-minute-averages into the archival bucket.
This bucket is meant to hold data for years.
You can adjust retention with the following parameters:
```console
influxdb.adminUser.retention_policy
archival.*
```

> ℹ️ **NOTE**: After installation, any changes in bucket namings, retention or downsampling settings will NOT be applied anymore.
> You'd need to manually change this with InfluxDB CLI within the pod.
## Grafana integration

The chart comes with a Grafana datasource and dashboard ConfigMaps enabled by default.
Please ensure Grafana sidecar is enabled and it searches in the release namespace, or set `grafana.*.namespace`.

The dashboard features some panels that calculate per-day metrics.
Make sure the stiebeleltron ISG device, InfluxDB and Grafana dashboard are roughly in the same timezone.
The stiebeleltron ISG console includes counters like "energy per day" that reset at midnight.

## Migrating from netdata plugin

If you have used the stiebeleltron plugin that is part of netdata, you can migrate the data to the exporter variant.

As this step is done manually once, it's not automated, but here's roughly how I did it:

1. Install this chart but don't start the exporter and influxdb yet (0 replicas each).
Also disable the archival feature for now.
2. Modify the InfluxDB 2.x StatefulSet, so that you can migrate using the Docker approach.
You'd need to set `DOCKER_INFLUXDB_INIT_MODE=upgrade` and add the volume mounts from the old DB.
Set `INFLUXD_ENGINE_PATH=/var/lib/influxdb2/migration` (we appended `migration`, otherwise there will be some permission errors).
Set `INFLUXD_BOLT_PATH=/var/lib/influxdb2/migration/influxd.bolt`.
3. You can now scale the replicas to 1.
4. Once the InfluxDB 2.x is running and the old DB upgraded, enter its shell and execute the following:
```
influx backup /var/lib/influxdb2/backup
```
5. Exit shell and scale replica to 0.
6. Reset `DOCKER_INFLUXDB_INIT_MODE`, `INFLUXD_ENGINE_PATH` and `INFLUXD_BOLT_PATH` to default values and set `replicas=1`.
7. Enter shell again and execute a restore:
```
influx bucket delete --name stiebeleltron_live
influx restore /var/lib/influxdb2/backup
influx bucket create --name stiebeleltron_archive --org stiebeleltron
rm -rf /var/lib/influxdb2/migration /var/lib/influxdb2/backup
```
8. You should have now `netdata/autogen`, `stiebeleltron_live` and `stiebeleltron_archive` bucket.
The last 2 should be empty for now.
9. Migrate & downsample the raw data from `netdata/autogen` bucket using the [scripts provided](files/influxdb/migration).
You might need to edit the query a bit if you are deviating from the defaults.
You can execute the scripts with `kubectl`, e.g.
```
cat migration/<the-file>.flux | kubectl -n stiebeleltron exec -i stiebeleltron-influxdb-0 -- influx query -o stiebeleltron
```
10. Each migration script can take a few minutes.
Check the Grafana dashboard if the values make sense.
11. Scale up the exporter and verify that values are stored in `stiebeleltron_live` bucket.
12. Create the downsampling task for `stiebeleltron_archive`, check out the ConfigMap in the Helm chart that would installed on first installation.
13. Delete the `netdata/autogen` bucket via CLI.

<!---
Common/Useful Link references from values.yaml
-->
[resource-units]: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes
[prometheus-operator]: https://github.com/coreos/prometheus-operator
[prom-relabel-config]: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#relabelconfig

## Source Code

* <https://github.com/ccremer/stiebeleltron-exporter>

## Requirements

| Repository | Name | Version |
|------------|------|---------|
| https://ccremer.github.io/charts | stiebeleltron(stiebeleltron-exporter) | 0.1.1 |
| https://helm.influxdata.com | influxdb(influxdb2) | 2.0.4 |

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| archival.bucket | string | `"stiebeleltron_archive"` | Name of the archival bucket to create after installation. |
| archival.enabled | bool | `true` | Whether long-term archival is enabled. Note: Disabling archival after installation (when enabled) does not remove the archival bucket. |
| archival.retention | string | `"0s"` | Retention of the archival bucket. `0s` means forever. |
| archival.window | string | `"10m"` | Fixed windows of time in which metrics are averaged. |
| fullnameOverride | string | `""` | |
| grafana.dashboard.enabled | bool | `true` | |
| grafana.dashboard.labels | object | `{"grafana_dashboard":"1"}` | The labels which the sidecar is filtering for dashboards. |
| grafana.dashboard.namespace | string | `""` | Override the namespace where the ConfigMap is installed, defaults to release namespace. |
| grafana.datasource.enabled | bool | `true` | |
| grafana.datasource.labels | object | `{"grafana_datasource":"1"}` | The labels which the sidecar is filtering for data sources. |
| grafana.datasource.namespace | string | `""` | Override the namespace where the ConfigMap is installed, defaults to release namespace. |
| influxdb.adminUser.bucket | string | `"stiebeleltron_live"` | |
| influxdb.adminUser.organization | string | `"stiebeleltron"` | |
| influxdb.adminUser.retention_policy | string | `"14d"` | |
| influxdb.enabled | bool | `true` | |
| influxdb.pdb.create | bool | `false` | |
| influxdb.persistence.size | string | `"2Gi"` | |
| influxdb.resources.limits.memory | string | `"256Mi"` | |
| influxdb.resources.requests.cpu | string | `"50m"` | |
| influxdb.resources.requests.memory | string | `"128Mi"` | |
| nameOverride | string | `"stiebeleltron"` | |
| stiebeleltron.enabled | bool | `true` | |
| stiebeleltron.nameOverride | string | `"exporter"` | |
| stiebeleltron.telegraf.enabled | bool | `true` | |
| stiebeleltron.telegraf.globalTags.site | string | `"home"` | The name of the site or environment. |
| stiebeleltron.telegraf.influxdb.bucket | string | `"stiebeleltron_live"` | The high-precision bucket name, needs to be equal to `influxdb.adminUser.bucket`. |
| stiebeleltron.telegraf.influxdb.token | string | `""` | The token to connect to InfluxDB, needs to be equal to `influxdb.adminUser.token`. |
| stiebeleltron.telegraf.influxdb.url | string | `"http://stiebeleltron-influxdb"` | |
| stiebeleltron.telegraf.interval | string | `"5s"` | Interval of sending metrics to InfluxDB. |
Loading

0 comments on commit c7c48ba

Please sign in to comment.