Switch branches/tags
Nothing to show
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
..
Failed to load latest commit information.
calibration
options
README.md
calibrator.sh
extractor.sh
harvester_influx.sh
harvester_last.sh
harvester_rivm.sh
harvester_whale.sh
refiner.sh
run.sh
sospublisher.sh
stapublisher.sh

README.md

ETL - Extract, Transform, Load for sensor data

Sources for ETL of the Smart Emission Platform. Originally this ETL was developed for the Smart Emission Project Nijmegen and the Intemo Josene Sensor Device (2015-2017). As to accommodate other sensor devices like the EU JRC AirSensEUR, the ETL-framework has been generalized (2018).

Uses host-specific variables for databases, passwords etc (not stored in GitHub).

All ETL is developed using Stetl. Stetl is a Python framework and programming model for any ETL process. The essence of Stetl is that each ETL process is a chain of linked Input, Filters and Output Python classes specified in a Stetl Config File.

The .sh files each invoke a Stetl ETL process via Docker.

Additional Python files implement specific ETL modules not defined in the Stetl Framework and are available under the Python smartem package. The Stetl config files for each ETL process are defined here.

All ETL is now migrated to the new SE GitHub repo: https://github.com/smartemission/docker-se-stetl

The scripts in this dir mainly call the Docker Image smartemission/se-stetl from DockerHub with the particular ETL Process as argument.