Skip to content

Commit

Permalink
Initial working (#1)
Browse files Browse the repository at this point in the history 
* Initial working

* Optimized code lenght

* Added connection error handling
  • Loading branch information
@kubagrzelak
kubagrzelak committed Jan 23, 2023
1 parent 84e9665 commit 6898a26
Show file tree
Hide file tree
Showing 11 changed files with 243 additions and 229 deletions.
126 changes: 83 additions & 43 deletions Module.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,50 +1,90 @@
displayName: 'Module Boilerplate' # Display name for the module that will be displayed in the UI.
moduleName: 'module-boilerplate' # Module name that will be used to reference the module, and uniquely identify the module and their versions.
description: 'This is module boilerplate description.' # Module description that will be displayed in the UI.
versionName: 'v1.0.0' # Module version name that will be used to reference the module version, and uniquely identify the module version with a common name.
isPublic: true # Boolean value that indicates if the module is Public or Private.
categories: # Array of category that will be used to group the module in the UI.
- 'CATEGORY_1'
- 'CATEGORY_2'
type: 'Output' # Module type (Input, Processing, Output) that will be used to group the module in the UI.
displayName: 'DolphinDB'
moduleName: 'dolphindb'
description: 'Write data to your DolphinDB.'
versionName: 'v1.0.0'
isPublic: true
categories:
- 'Databases'
type: 'Output'
image:
name: 'weevenetwork/module-boilerplate' # Image name from the docker registry.
tag: 'latest' # Image tag from the docker registry.
homepage: 'https://hub.docker.com/weevenetwork/module-boilerplate' # Image homepage from the docker registry.
repository: 'https://github.com/weeve-modules/python-output-module-boilerplate' # Repository URL to source code.
envs: # Array of environment variables that will be used to configure the module. Also used to show related fields on the UI.
- name: Setting 1 # name of the field to be displayed on the UI.
key: ENVIRONMENT_1 # key to set in the environment variables.
description: Module setting 1 # Description/helper text for the field.
value: '' # Value to be set in the environment variables.
type: 'select' # Type of the field. Possible values are: ['text', 'select', 'boolean', 'date', 'time', 'dateTime-local', 'number', 'multiple', 'CSV', 'file']
options: # Options for the field if it is a select or multiple select. An array of string options
- option_A
- option_B
- option_C
dependencies: [] # List of dependencies for the field. This field is only displayed if all the dependencies are met. If
- name: Setting 2
key: ENVIRONMENT_2
description: Module setting 2
value: ''
type: 'number'
options: []
dependencies:
- key: ENVIRONMENT_1 # key of the dependency
value: option_B # value of the dependency
ports: # Array of ports that will be exposed by the module.
- container: 8080 # Port number from the container.
host: 8080 # Port number to map on the host.
name: 'weevenetwork/dolphindb'
tag: 'v1.0.0'
homepage: 'https://hub.docker.com/weevenetwork/dolphindb'
repository: 'https://github.com/weeve-modules/dolphindb'
envs:
- name: Host
key: HOST
description: Host of DolphinDB server.
value: '0.0.0.0'
type: 'text'
options: []
dependencies: []
- name: Port
key: PORT
description: Port of DolphinDB server.
value: '8848'
type: 'text'
options: []
dependencies: []
- name: Username
key: USERNAME
description: Your DolphinDB username.
value: '(optional)'
type: 'text'
options: []
dependencies: []
- name: Password
key: PASSWORD
description: Your DolphinDB password.
value: '(optional)'
type: 'text'
options: []
dependencies: []
- name: Database Path
key: DB_PATH
description: Your database path.
value: 'dfs://demo'
type: 'text'
options: []
dependencies: []
- name: Table Name
key: TABLE_NAME
description: Table to write data to.
value: 'my_table'
type: 'text'
options: []
dependencies: []
- name: Database Columns
key: COLUMNS
description: List of comma (,) separated database columns headers to write to. IMPORTANT - order must be the same as in the database table.
value: 'date, temperature, location'
type: 'text'
options: []
dependencies: []
- name: Data Labels
key: LABELS
description: List of comma (,) separated labels in passed data. Order of labels must match the order of provided corresponding columns.
value: 'dateStamp, filterTemp, sensorLocation'
type: 'text'
options: []
dependencies: []
- name: Date Column
key: DATE_COLUMN
description: Name of column in database that holds date or timestamp (optional).
value: 'date'
type: 'text'
options: []
dependencies: []
ports: []
envPorts: []
mounts:
- container: '/data' # Mount path from the container.
host: '/data' # Mount path to map on the host.
mounts: []
envMounts: []
devices: []
envDevices: []
tags: # Array of tags.
tags:
- 'Python'
- 'Output'
- 'Boilerplate'
- '1.0.0'
icon: 'https://some.url/to/icon.png' # Icon URL that will be displayed in the UI.
- 'DolphinDB'
- 'Database'
- 'Time-Series'
icon: 'https://icons-020-demo.s3.eu-central-1.amazonaws.com/mi_dolphindb_output.png'
165 changes: 75 additions & 90 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,106 +1,91 @@
# Python Output Module Boilerplate
# DolphinDB

| | |
| ------------ | ---------------------------------------------------------------- |
| name | Python Output Module Boilerplate |
| version | v1.0.0 |
| GitHub | [python-output-module-boilerplate](https://github.com/weeve-modules/python-egress-module-boilerplate) |
| authors | Jakub Grzelak, Nithin Saai |
| | |
| --------- | ------------------------------------------------------------------------- |
| Name | DolphinDB |
| Version | v1.0.0 |
| DockerHub | [weevenetwork/dolphindb](https://hub.docker.com/r/weevenetwork/dolphindb) |
| Authors | Jakub Grzelak |

***
## Table of Content

- [Python Output Module Boilerplate](#python-output-module-boilerplate)
- [Table of Content](#table-of-content)
- [DolphinDB](#dolphindb)
- [Description](#description)
- [Directory Structure](#directory-structure)
- [File Tree](#file-tree)
- [Module Variables](#module-variables)
- [As a module developer](#as-a-module-developer)
- [Environment Variables](#environment-variables)
- [Module Specific](#module-specific)
- [Set by the weeve Agent on the edge-node](#set-by-the-weeve-agent-on-the-edge-node)
- [Dependencies](#dependencies)
***

## Description

This is a Python Output Boilerplate module and it serves as a starting point for developers to build output modules for weeve platform and data services.
Navigate to [As a module developer](#as-a-module-developer) to learn how to use this module. You can also explore our weeve documentation on [weeve Modules](https://docs.weeve.engineering/concepts/edge-applications/weeve-modules) and [module tutorials](https://docs.weeve.engineering/guides/how-to-create-a-weeve-module) to learn more details.

## Directory Structure

Most important resources:

| name | description |
| ----------------- | ------------------------------------------------------------------------------------------------------ |
| src | All source code related to the module (API and module code). |
| src/main.py | Entry-point for the module. |
| src/api | Code responsible for setting module's API and communication with weeve ecosystem. |
| src/module | Code related to the module's business logic. This is working directory for module developers. |
| docker | All resources related to Docker (Dockerfile, docker-entrypoint.sh, docker-compose.yml). |
| example.env | Holds examples of environment variables for running the module. |
| requirements.txt | A list of module dependencies. |
| Module.yaml | Module's YAML file that is later used by weeve platform Data Service Designer |

### File Tree

```bash
├── src
│ ├── api
│ │ ├── __init__.py
│ │ ├── log.py # log configurations
│ │ ├── processing_thread.py # a separate thread responsible for triggering data outputting
│ │ └── request_handler.py # handles module's API and receives data from a previous module
│ ├── module
│ │ ├── main.py # [*] main logic for the module
│ │ └── validator.py # [*] validation logic for incoming data
│ └── main.py # module entrypoint
├── docker
│ ├── .dockerignore
│ ├── docker-compose.yml
│ ├── docker-entrypoint.sh
│ └── Dockerfile
├── example.env # sample environment variables for the module
├── Module.yaml # used by weeve platform to generate resource in Data Service Designer section
├── makefile
├── README.md
├── example.README.md # README template for writing module documentation
└── requirements.txt # module dependencies, used for building Docker image
```
- [Input](#input)
- [Output](#output)

## Description

## Module Variables
Write data to your chosen DolphinDB table.

There are 5 module variables that are required by each module to correctly function within weeve ecosystem. In development, these variables can overridden for testing purposes. In production, these variables are set by weeve Agent.
## Environment Variables

| Environment Variables | type | Description |
| --------------------- | ------ | ------------------------------------------------- |
| MODULE_NAME | string | Name of the module |
| MODULE_TYPE | string | Type of the module (Input, Processing, Output) |
| LOG_LEVEL | string | Allowed log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL. Refer to `logging` package documentation. |
| INGRESS_HOST | string | Host to which data will be received |
| INGRESS_PORT | string | Port to which data will be received |
### Module Specific

## As a module developer
The following module configurations can be provided in a data service designer section on weeve platform:

RECOMMENDED:
Make sure you have [virtual environment](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/)
| Name | Environment Variables | type | Description |
| ---------------- | --------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- |
| Host | HOST | string | Host of DolphinDB server. |
| Port | PORT | integer | Port of DolphinDB server. |
| Username | USERNAME | string | Your DolphinDB username. |
| Password | PASSWORD | string | Your DolphinDB password. |
| Database Path | DB_PATH | string | Your database path. |
| Table Name | TABLE_NAME | string | Table to write data to. |
| Database Columns | COLUMNS | string | List of comma (,) separated database columns headers to write to. |
| Data Labels | LABELS | string | List of comma (,) separated labels in passed data. Order of labels must match the order of provided corresponding columns. |
| Date Column | DATE_COLUMN | string | Name of column in database that holds date or timestamp (optional). |

A module developer needs to add all the configuration and business logic.
### Set by the weeve Agent on the edge-node

All the module logic can be written in the module package in `src/module` directory.
Other features required for establishing the inter-container communication between modules in a data service are set by weeve agent.

* The files can me modified for the module
1. `module/validator.py`
* The function `data_validation` takes the JSON data received from the previous module.
* Incoming data can be validated here.
* Checks if data is of type permitted by a module (i.e. `dict` or `list`)>
* Checks if data contains required fields.
* Returns Error if data are not valid.
2. `module/module.py`
* The function `module_main` takes the JSON data received from the previous module.
* All the business logic about modules are written here.
* Returns error message.
| Environment Variables | type | Description |
| --------------------- | ------ | ---------------------------------------------- |
| MODULE_NAME | string | Name of the module |
| MODULE_TYPE | string | Type of the module (Input, Processing, Output) |
| INGRESS_HOST | string | Host to which data will be received |
| INGRESS_PORT | string | Port to which data will be received |

## Dependencies

The following are module dependencies:
```txt
bottle
dolphindb
pandas
numpy
```

## Input

Input to this module is:

* JSON body single object, example:

```json
{
"label-1": 12,
"label-2": "speed"
}
```

* array of JSON body objects, example:

```json
[
{
"label-1": 12,
"label-2": "speed"
},
{
"label-1": 15,
"label-2": "volume"
}
]
```

## Output

* bottle
Output of this module are records written to the selected DolphinDB table.
13 changes: 11 additions & 2 deletions docker/docker-compose.yml
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,22 @@
version: "3.0"
services:
boilerplate:
dolphindb:
build:
context: ../
dockerfile: docker/Dockerfile
environment:
MODULE_NAME: "boilerplate"
MODULE_NAME: "dolphindb"
MODULE_TYPE: "Output"
INGRESS_HOST: "0.0.0.0"
INGRESS_PORT: "80"
HOST: "0.0.0.0"
PORT: "8848"
USERNAME: "(optional)"
PASSWORD: "(optional)"
DB_PATH: "dfs://demo"
TABLE_NAME: "mytable"
COLUMNS: "date, temperature, location"
LABELS: "date, filterTemp, sensorLocation"
DATE_COLUMN: "date"
ports:
- 80:80
4 changes: 4 additions & 0 deletions docker/docker-entrypoint.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,9 @@ else
fi
echo "[ENTRYPOINT] Environment validated."

# Validate other environment variables
: "${COLUMNS:?Need to provide database Columns in module configuration pane in weeve IoT Platform.}"
: "${LABELS:?Need to provide data Labels in module configuration pane in weeve IoT Platform.}"

# CALL THE MAIN SCRIPT
python /app/src/main.py
Loading

0 comments on commit 6898a26

Please sign in to comment.