New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docs: add documentation for Backend Metadata and Timedata #2314
Changes from all commits
c5d38d8
3e9f7ac
c1c058b
0d06a03
5c86fa9
9fbadf2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
= Metadata | ||
:sectnums: | ||
:sectnumlevels: 4 | ||
:toc: | ||
:toclevels: 4 | ||
:experimental: | ||
:keywords: AsciiDoc | ||
:source-highlighter: highlight.js | ||
:icons: font | ||
:imagesdir: ../../assets/images | ||
|
||
The https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/Metadata.java[Metadata] interface defines a service, that coordinates OpenEMS Backend communication with one or more OpenEMS Edges and with the OpenEMS UI. | ||
It is responsible for identification and authorization of OpenEMS edges and of Users using the OpenEMS UI. | ||
Beside that the Metadata bundle is responsible for the state of https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/User.java[users] and https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/Edge.java[edges] and related processes. | ||
|
||
== Edges | ||
|
||
The Metadata service provides access to the https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/Edge.java[Edge] object - the digital twin representation of the OpenEMS Edge instance. | ||
[NOTE] | ||
==== | ||
Identification of OpenEMS Edges is done with a unique API key. | ||
Please take care, that the API key includes a lot of randomness, because it is implicitly used to authorize the Edge. | ||
==== | ||
|
||
== Users | ||
|
||
The https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/metadata/User.java[User] object includes meta information about a user. | ||
Furthermore it is used for identification/authorization of the user. | ||
The metadata service can be used to change various attributes which are associated with a user (e.g. the users default language). | ||
|
||
== Processes | ||
|
||
The metadata service provides methods to | ||
|
||
* add users to an Edge, | ||
* set and change alerting/notification settings, | ||
* set up new Edge devices. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
= Backend Services | ||
:sectnums: | ||
:sectnumlevels: 4 | ||
:toc: | ||
:toclevels: 4 | ||
:experimental: | ||
:keywords: AsciiDoc | ||
:source-highlighter: highlight.js | ||
:icons: font | ||
:imagesdir: ../../assets/images | ||
|
||
The following bundles are available in OpenEMS Backend. | ||
[NOTE] | ||
==== | ||
This list is not complete. | ||
==== | ||
|
||
include::service.adoc.d/_include.adoc[leveloffset=+0] | ||
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
*.adoc |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
= Timedata | ||
:sectnums: | ||
:sectnumlevels: 4 | ||
:toc: | ||
:toclevels: 4 | ||
:experimental: | ||
:keywords: AsciiDoc | ||
:source-highlighter: highlight.js | ||
:icons: font | ||
:imagesdir: ../../assets/images | ||
|
||
Live and historical data of an OpenEMS Edge are handled by a https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.common/src/io/openems/backend/common/timedata/Timedata.java[Timedata] service. | ||
It describes basically methods to write and read Edge data to/from a database. There are different kind of timedata providers within OpenEMS (see xref:service.adoc[Service] for concrete implementations). | ||
|
||
Within OpenEMS Backend the only component which uses the different timedata services directly is the https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.core/src/io/openems/backend/core/timedatamanager/TimedataManagerImpl.java[TimedataManager]. | ||
It passes Edge relevant data to **all** Timedata service and it reads the data from the **first** Timedata providers which can deliver it. | ||
|
||
[NOTE] | ||
==== | ||
Following paragraphs describe the new data handlng since OpenEMS version 2023.8. | ||
==== | ||
|
||
OpenEMS Edges will send different types of data to the OpenEMS Backend: | ||
|
||
* `TimestampedDataNotification` | ||
** channel values which have changed | ||
** every 5 minutes a full snapshot of all channel values is sent, including channels which haven't changed over this time period | ||
|
||
* `AggregatedDataNotification` | ||
** data is sent in a format that is optimized for fast querying from the database, to allow very fast responses in OpenEMS UI | ||
** aggregated (average or maximum) channel values are sent every 5 minutes | ||
** Backend services handle values differently according to their aggregation type, e.g. 5-minute-average is always stored; maximum is only stored if feasible for fast energy queries (e.g. at the end of a day, depending on a time-zone) | ||
|
||
* `ResendDataNotification` | ||
** historic data that is resent after connection problems between Edge and Backend | ||
** this data is always aggregated in the form of the Edge.Timedata service (e.g. RRD4j) | ||
|
||
Splitting the data enables OpenEMS to implement different timedata providers, | ||
which handle data differently. This gives more flexibility when scaling OpenEMS. | ||
Also due to performance reasons the computation of `AggregatedDataNotification` is done on the Edge side. | ||
This helps keeping CPU load on the database server low. | ||
|
||
To get a better understanding of the new Edge data concept, have a look at the | ||
xref:service.adoc.d/io.openems.backend.timedata.aggregatedinflux.adoc[Aggregated Influx] bundle. | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
= Metadata File | ||
|
||
OpenEMS Backend implementation for Metadata. | ||
|
||
Allows you to configure multiple edges by a single JSON file. | ||
Using this bundle enables you to easily set up an OpenEMS | ||
Backend for testing purposes. | ||
|
||
|
||
== Example configuration file | ||
|
||
``` | ||
{ | ||
edges: { | ||
edge0: { | ||
comment: "Edge #0", | ||
apikey: "edge0", | ||
setuppassword: "abcdefgh" | ||
}, | ||
edge1: { | ||
comment: "Edge #1", | ||
apikey: "edge1", | ||
setuppassword: "1234567" | ||
} | ||
} | ||
} | ||
``` | ||
|
||
https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.metadata.file[Source Code icon:github[]] |
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
= Metadata Odoo | ||
|
||
Metadata OpenEMS Backend implementation for https://github.com/odoo/odoo[Odoo]. | ||
|
||
Enables you to manage multiple edges with one backend. | ||
|
||
|
||
[NOTE] | ||
==== | ||
* This bundle works with Odoo 15. For a seamless integration of Odoo | ||
and OpenEMS the open-source https://github.com/OpenEMS/odoo-openems[Odoo-OpenEMS] module is needed. | ||
==== | ||
|
||
|
||
|
||
|
||
https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.metadata.odoo[Source Code icon:github[]] |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,122 @@ | ||
= Timedata Aggregated InfluxDB | ||
|
||
OpenEMS Backend implementation for aggregated InfluxDB access. | ||
|
||
[NOTE] | ||
==== | ||
Bundle is needed for highly optimized environments with | ||
thousands of connected edges only. If you do not need that functionality you can skip this chapter. | ||
==== | ||
|
||
A large number of OpenEMS edges connected to a single backend leads to a performance bottleneck. | ||
The bottleneck is mainly due to the access to the database. This `Aggregated InfluxDB` bundle in | ||
combination with the xref:../io.openems.backend.timedata.influx.adoc[Timedata Influx] | ||
bundle can mitigate this bottleneck. This happens by writing all data as usual via xref:../io.openems.backend.timedata.influx.adoc[Timedata Influx] into | ||
an Influx database (`DB 1`). Furthermore, all "aggregated data" (see xref:timedata.adoc[edge datatypes]) are written via the `Aggregated InfluxDB` bundle to | ||
another database (`DB 2`) located on another server. | ||
Thus `DB 1` holds all data in high resolution, while `DB 2` only holds very few data (5-minute averages for power values | ||
or only daily values for energy). Furthermore, a retention policy of e.g. 90 days is applied to `DB 2`. | ||
Thus `DB 1` provides relatively slow access to a huge amount of data | ||
and `DB 2` provides fast and responsive access to a relatively small database. | ||
|
||
Within OpenEMS backend access to both databases is managed by the | ||
https://github.com/OpenEMS/openems/blob/develop/io.openems.backend.core/src/io/openems/backend/core/timedatamanager/TimedataManagerImpl.java[TimedataManager]. | ||
The TimedataManager writes edge relevant data to **all** Timedata providers, | ||
whereas it reads data only from the **first** Timedata provider, | ||
which can deliver. | ||
Assuming correct configuration, TimedataManager first returns historical data | ||
from `DB 2` (fast and responsive). | ||
And only in a very few cases it gets the data from `DB 1` | ||
(probably slower). | ||
|
||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we also mention what availableSince is doing? |
||
|
||
|
||
[CAUTION] | ||
==== | ||
The `Timedata.AggregatedInfluxDB` includes a class `AllowedChannels.java`. | ||
This implementation includes a *hardcoded* list of channels: | ||
|
||
* `ALLOWED_AVERAGE_CHANNELS` are used to calculate the 5min average values (e.g. average power values). | ||
* `ALLOWED_CUMULATED_CHANNELS` are used to calculate daily values (e.g. energy values). | ||
|
||
This list must be adopted to a concrete usecase. It strongly depends on | ||
|
||
* your strategy to select component-IDs. | ||
* the components you are using within OpenEMS. | ||
|
||
If you detect some widgets within your OpenEMS-UI which hava empty values, | ||
it may have to do with an incorrect hardcoded list. | ||
|
||
==== | ||
|
||
== Configuration Setup | ||
|
||
If you expect your backend to handle thousands of connected edges, | ||
the following configuration may provide a good start setup: | ||
|
||
*Create database and set retention policy:* | ||
|
||
Before starting the OpenEMS backend and after intially setting up the influx servers, | ||
you need to create the databases `influx0` and `aggregated0` | ||
and some retention policies: | ||
|
||
[source,shell] | ||
---- | ||
|
||
##### Influx Server 1 ##### | ||
curl -i -XPOST http://127.0.0.1:8082/query --data-urlencode "q=CREATE DATABASE influx0" | ||
|
||
##### Influx Server 2 ##### | ||
curl -i -XPOST http://127.0.0.1:8081/query --data-urlencode "q=CREATE DATABASE aggregated0" | ||
|
||
curl -i -XPOST http://127.0.0.1:8081/query --data-urlencode "q=CREATE RETENTION POLICY rp_max ON aggregated0 DURATION 90d REPLICATION 1" | ||
curl -i -XPOST http://127.0.0.1:8081/query --data-urlencode "q=CREATE RETENTION POLICY rp_avg ON aggregated0 DURATION 90d REPLICATION 1" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe we should link to this documentation. And the more important configuration is the the shard duraction If someone has the same settings for both retention policies they dont have to create the same one twice. We are using two different retention policies: For max we used a shard duration fo 31 days because there is much less data in this measurement (only one per day) and to optimize queries for months only 2 shards need to be readed by the db. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks. This is a valid comment, but in order to release these docs more quickly I decided to leave it out for now. I am happy to accept future documentation PRs from you. ;-) |
||
|
||
---- | ||
|
||
*OpenEMS backend configuration:* | ||
|
||
* bundle `Timedata.AggregatedInfluxDB` | ||
** ComponentID: `timedata0` | ||
** QueryLanguage: `INFLUX_QL` | ||
** URL: http://127.0.0.1:8081 | ||
** Apikey: `influxuser:influxpassword` | ||
** Bucket: `aggregated0` | ||
** Retention policy for avg values: `rp_avg` | ||
** Retention policy for max values: `rp_max` | ||
** Measurement Avg: `avg` | ||
** Measurement for max values: `Europe/Berlin=max` | ||
|
||
* bundle `Timedata.InfluxDB` | ||
** ComponentID: `timedata1` | ||
** QueryLanguage: `INFLUX_QL` | ||
** URL: http://127.0.0.1:8082 | ||
** Org: `-` | ||
** Apikey: `influxuser:influxpassword` | ||
** Bucket: `influx0/autogen` | ||
** Measurement: `data` | ||
|
||
* bundle `Core Timedata-Manager` | ||
** Timedata-IDs: `[timedata0, timedata1]` | ||
|
||
|
||
|
||
|
||
[NOTE] | ||
==== | ||
* Note the different Bucket namings, one with retention policy `autogen` and one without. | ||
* Make sure that your influx databases are located on two different servers | ||
or that you can easily move one database to another server later. | ||
* Be sure that the sequence within the `Core Timedata-Manager` is | ||
configured correctly - the component-ID of the `Timedata.AggregatedInfluxDB` | ||
comes first. | ||
* Influx Database must have at least Version 1.8.10. | ||
* Take care of your edge to backend connections on the edge side. Be sure to understand and select the right `PersistencePriority` for the different datatypes. | ||
* Remember the hardcoded list in _AllowedChannels.java_ | ||
* Note that you can shift the load of your databases by choosing different retention policies. | ||
* Be sure you know exactly how Influx handles *reading* and *writing* data when using different retention policies on one database. | ||
==== | ||
|
||
|
||
https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.timedata.influx.aggregatedinflux[Source Code icon:github[]] |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
= Timedata Dummy | ||
|
||
OpenEMS Backend dummy implementation for a timeseries database. | ||
|
||
Using this bundle enables you to easily set up an OpenEMS Backend for testing purposes. | ||
|
||
|
||
https://github.com/OpenEMS/openems/tree/develop/io.openems.backend.timedata.influx[Source Code icon:github[]] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello, just saw this file is missing on develop, can you readd it?