Skip to content

Commit

Permalink
#87 maintain and fix documentation and move CAdvisor usage
Browse files Browse the repository at this point in the history
  • Loading branch information
@justb4
justb4 committed May 8, 2018
1 parent dce02fe commit 6f08c54
Show file tree
Hide file tree
Showing 6 changed files with 78 additions and 179 deletions.
Binary file removed docker/.DS_Store
View file
Binary file not shown.
133 changes: 2 additions & 131 deletions docs/platform/code.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,136 +7,7 @@ ETL Processes
-------------

Python classes involved in ETL.
Code and config for all ETL can be found in the `SE GitHub <https://github.com/smartemission/smartemission/tree/master/etl>`_.
Code and config for all ETL can be found in the `SE GitHub <https://github.com/smartemission/docker-se-stetl>`_.
All Python ETL code is under the
`SE smartem Python Package <https://github.com/smartemission/smartemission/tree/master/etl/smartem>`_.
`SE smartem Python Package <https://github.com/smartemission/docker-se-stetl/tree/master/smartem>`_.

Harvesters and Extractors
~~~~~~~~~~~~~~~~~~~~~~~~~

.. automodule:: smartem.harvester.rawsensortimeseriesinput
:members:
:show-inheritance:

.. automodule:: smartem.harvester.rawsensorlastinput
:members:
:show-inheritance:

.. automodule:: smartem.harvester.harvestinfluxdb
:members:
:show-inheritance:

.. automodule:: smartem.harvester.rivmsosinput
:members:
:show-inheritance:

.. automodule:: smartem.harvester.extractor
:members:
:show-inheritance:

Devices
~~~~~~~

Device-specific metadata, conversion and support-functions.

.. automodule:: smartem.devices.device
:members:
:show-inheritance:

.. automodule:: smartem.devices.devicereg
:members:
:show-inheritance:

.. automodule:: smartem.devices.josene
:members:
:show-inheritance:

.. automodule:: smartem.devices.josenedefs
:members:
:show-inheritance:

.. automodule:: smartem.devices.josenefuncs
:members:
:show-inheritance:

Calibrator
~~~~~~~~~~

The calibration a.o. ANN learning system.

.. automodule:: smartem.calibrator.calibration
:members:
:show-inheritance:


.. automodule:: smartem.calibrator.calibration_input
:members:
:show-inheritance:

.. automodule:: smartem.calibrator.calibration_output
:members:
:show-inheritance:


.. automodule:: smartem.calibrator.mergerefdata
:members:
:show-inheritance:


.. automodule:: smartem.calibrator.calibration_parameters
:members:
:show-inheritance:


.. automodule:: smartem.calibrator.calibration_visualization
:members:
:show-inheritance:

Refinement
~~~~~~~~~~

.. automodule:: smartem.refiner.refinefilter
:members:
:show-inheritance:

.. automodule:: smartem.refiner.refiner
:members:
:show-inheritance:

Database Input/Output
~~~~~~~~~~~~~~~~~~~~~

.. automodule:: smartem.rawdbinput
:members:
:show-inheritance:

.. automodule:: smartem.refineddbinput
:members:
:show-inheritance:

.. automodule:: smartem.progresstracker
:members:
:show-inheritance:

.. automodule:: smartem.influxdbinput
:members:
:show-inheritance:

.. automodule:: smartem.influxdboutput
:members:
:show-inheritance:


SOS-T Publisher
~~~~~~~~~~~~~~~

.. automodule:: smartem.publisher.sosoutput
:members:
:show-inheritance:

SensorThings API Publisher
~~~~~~~~~~~~~~~~~~~~~~~~~~

.. automodule:: smartem.publisher.staoutput
:members:
:show-inheritance:
72 changes: 36 additions & 36 deletions docs/platform/components.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,39 +47,39 @@ of the Docker Image for K8s deployment:
* avail=available for deploy
* done=deployed in K8s

================ ============= =============================== ======== ======= ========
Name Categories Function Repos Prio Status
================ ============= =============================== ======== ======= ========
Home Apps Platform home/landing page GH DH 1 done
Admin Apps,Admin Admin access pages GH DH 2 done
Heron Apps Viewer with history GH DH 1 done
SmartApp Apps Viewer for last values GH DH 1 done
Waalkade Apps Viewer Nijmegen project GH DH 1 done
GostDashboard Apps,Admin Admin dashboard Gost GH DH 3 inprog
Grafana Apps View InfluxDB Data GH DH 3 inprog
Chronograf Apps,Admin Admin dashboard InfluxDB GH DH 3 avail
SOSEmu Services REST API SOS subset GH DH 1 done
GeoServer Services WMS (Time), WFS server GH DH 1 done
Gost Services SensorThings API (STA) server GH DH 2 done
SOS52N Services 52North SOS server GH DH 3 done
Mosquitto Services MQTT server coupled with Gost GH DH 2 done
PhpPgAdmin Apps,Admin Manager PostgreSQL GH DH 2 inprog
HarvesterLast ETL Harvester last sensor data GH DH 1 done
HarvesterWhale ETL Harvester historic sensor data GH DH 1 done
HarvesterInflux ETL Harvester InfluxDB sensor data GH DH 3 inprog
HarvesterRivm ETL Harvester RIVM ANN ref-data GH DH 4 avail
Extractor ETL Extract SE refdata for ANN ref GH DH 4 avail
Calibrator ETL ANN Learning engine GH DH 4 -
Refiner ETL Transformation/Calibration GH DH 1 done
SOSPublisher ETL Publish refined data to SOS GH DH 3 avail
STAPublisher ETL Publish refined data to STA GH DH 2 done
InfluxDB Datastore Calibration refdata/collector GH DH 3 -
PGPool Datastore PG Connection Pooling (only K8s) GH DH 1 done
Postgis Datastore Main database (not used in K8s) GH DH N.A. N.A.
Traefik Services Proxy server (not used in K8s) GH DH N.A. N.A.
Prometheus Mon,Apps Monitoring metrics collector GH DH 4 -
AlertManager Mon Prometheus (Prom.)alerter GH DH 4 -
CAdvisor Mon Prom. Docker metrics exporter GH DH 4 -
NodeExporter Mon Prom. host metrics exporter GH DH 4 -
GrafanaMon Mon,Apps Grafana Dashboards Prometheus GH DH 4 -
================ ============= =============================== ======== ======= ========
================ ============= ================================= ======== ======= ========
Name Categories Function Repos Prio Status
================ ============= ================================= ======== ======= ========
Home Apps Platform home/landing page GH DH 1 done
Admin Apps,Admin Admin access pages GH DH 2 done
Heron Apps Viewer with history GH DH 1 done
SmartApp Apps Viewer for last values GH DH 1 done
Waalkade Apps Viewer Nijmegen project GH DH 1 done
GostDashboard Apps,Admin Admin dashboard Gost GH DH 3 inprog
Grafana Apps View InfluxDB Data GH DH 3 inprog
Chronograf Apps,Admin Admin dashboard InfluxDB GH DH 3 avail
SOSEmu Services REST API SOS subset GH DH 1 done
GeoServer Services WMS (Time), WFS server GH DH 1 done
Gost Services SensorThings API (STA) server GH DH 2 done
SOS52N Services 52North SOS server GH DH 3 done
Mosquitto Services MQTT server coupled with Gost GH DH 2 done
PhpPgAdmin Apps,Admin Manager PostgreSQL GH DH 2 inprog
HarvesterLast ETL Harvester last sensor data GH DH 1 done
HarvesterWhale ETL Harvester historic sensor data GH DH 1 done
HarvesterInflux ETL Harvester InfluxDB sensor data GH DH 3 inprog
HarvesterRivm ETL Harvester RIVM ANN ref-data GH DH 4 avail
Extractor ETL Extract SE refdata for ANN ref GH DH 4 avail
Calibrator ETL ANN Learning engine GH DH 4 -
Refiner ETL Transformation/Calibration GH DH 1 done
SOSPublisher ETL Publish refined data to SOS GH DH 3 avail
STAPublisher ETL Publish refined data to STA GH DH 2 done
InfluxDB Datastore Calibration refdata/collector GH DH 3 -
PGPool Datastore PG Connection Pooling (only K8s) GH DH 1 done
Postgis Datastore Main database (not used in K8s) GH DH N.A. N.A.
Traefik Services Proxy server (not used in K8s) GH DH N.A. N.A.
Prometheus Mon,Apps Monitoring metrics collector GH DH 4 -
AlertManager Mon Prometheus (Prom.)alerter GH DH 4 -
CAdvisor Mon Prom. Docker metrics exporter GH DH 4 -
NodeExporter Mon Prom. host metrics exporter GH DH 4 -
GrafanaMon Mon,Apps Grafana Dashboards Prometheus GH DH 4 -
================ ============= ================================= ======== ======= ========
1 change: 1 addition & 0 deletions docs/platform/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Contents:
calibration.rst
services.rst
api.rst
code.rst
installation.rst
administration.rst
dissemination.rst
Expand Down
49 changes: 38 additions & 11 deletions docs/platform/intro.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,48 @@ Intro

This is the main (technical) documentation for the Smart Emission Data Platform.
It can always be found at `smartplatform.readthedocs.org <http://smartplatform.readthedocs.org/>`_.
A somewhat lighter introduction can be found in `this series of blogs <https://justobjects.nl/category/smartemission/>`_.

The home page for the Smart Emission project is http://www.smartemission.nl
The home page for the Smart Emission project and data platform is http://data.smartemission.nl

The home page for the Smart Emission Data Platform project is http://data.smartemission.nl
The home page for the Smart Emission Nijmegen project is http://smartemission.ruhosting.nl

The project GitHub repository is at https://github.com/smartemission/smartemission.

Background reading also via the SOSPilot project via http://sensors.geonovum.nl.

This is document version |release| generated on |today|.

Background
==========
History
=======

The Smart Emission Platform was initiated and largely developed within
the `Smart Emission Nijmegen project <http://smartemission.ruhosting.nl>`_ (see also below).

The Geonovum/RIVM `SOSPilot Project <http://sensors.geonovum.nl>`_, where RIVM LML
(Dutch national Air Quality Data) data was harvested and serviced via the OGC Sensor Observation Service (SOS), was
a precursor for the architecture and approach to ETL with sensor data.

In and after 2017 several other projects, web-clients and sensor-types started utilizing the platform hosted at
`data.smartemission.nl <http://data.smartemission.nl>`_. These include:

* the `Smart City Living Lab <https://slimstestad.nl/programma-2017-2018/>`_: around 7 major cities within NL deployed Intemo sensor stations
* `AirSensEUR <http://www.airsenseur.org/>`_ - a EU JRC initiative for an Open Sensor HW/SW platform

This put more strain on the platform and required a more structural development and maintenance approach (than project-based funding).

In 2018, the SE Platform was migrated to the Dutch National GDI infrastructure `PDOK <https://pdok.nl>`_ maintained
by the `Dutch Kadaster <https://www.kadaster.nl/>`_.
This gives a tremendous opportunity for long-term evolution and stability of the platform beyond the initial
and project-based fundings. This migration targeted hosting within a `Docker Kubernetes <https://kubernetes.io/>`_ environment.
All code was migrated to a dedicated `Smart Emission GitHub Organization <https://github.com/smartemission>`_ and
hosting of all Docker Images on an `SE DockerHub Organization <https://hub.docker.com/r/smartemission/>`_.

Smart Emission Nijmegen
=======================

The Smart Emission Platform was largely developed during the Smart Emission Nijmegen project
started in 2015 and still continuing.

Read all about the Smart Emission project via: `www.smartemission.nl <http://www.smartemission.nl>`_.
Read all about the Smart Emission Nijmegen project via: `smartemission.ruhosting.nl/ <http://smartemission.ruhosting.nl>`_.

An introductory presentation:
http://www.ru.nl/publish/pages/774337/smartemission_ru_24juni_lc_v5_smallsize.pdf
Expand All @@ -44,17 +71,17 @@ the project is described extensively.
*distribution of air pollution in the city, balanced against other spatial qualities. ...."*

The data from the Smart Emission sensors is converted and published as standard web services: OGC WMS(-Time), WFS, SOS
and SensorThings and FIWARE APIs. FIWARE is used as the supporting computing and component platform. Some web clients
and SensorThings APIs. Some web clients
(SmartApp, Heron) are developed to visualize the data. All this is part of the Smart Emission Data Platform whose technicalities
are the subject of this document.

Partners
========
SE Nijmegen Project Partners
----------------------------

.. figure:: _static/se-partners.jpg
:align: center

*Smart Emission Project Partners*
*Smart Emission Nijmegen Project Partners*

More on: http://smartemission.ruhosting.nl/over-ons/

Expand Down
2 changes: 1 addition & 1 deletion services/monitoring/docker-compose.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ services:

cadvisor:
# image: google/cadvisor
image: geonovum/cadvisor:v0.28.3
image: smartemission/se-cadvisor:v0.28.3
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
Expand Down

0 comments on commit 6f08c54

Please sign in to comment.