diff --git a/docs/_static/smartem/heron-viewer-o3-ts.jpg b/docs/_static/smartem/heron-viewer-o3-ts.jpg new file mode 100644 index 0000000..dc0fc44 Binary files /dev/null and b/docs/_static/smartem/heron-viewer-o3-ts.jpg differ diff --git a/docs/_static/smartem/heron-viewer-o3.jpg b/docs/_static/smartem/heron-viewer-o3.jpg new file mode 100644 index 0000000..04df0ca Binary files /dev/null and b/docs/_static/smartem/heron-viewer-o3.jpg differ diff --git a/docs/_static/smartem/jsclient-no2.jpg b/docs/_static/smartem/jsclient-no2.jpg new file mode 100644 index 0000000..1cc943a Binary files /dev/null and b/docs/_static/smartem/jsclient-no2.jpg differ diff --git a/docs/_static/smartem/measurements-stations-view.jpg b/docs/_static/smartem/measurements-stations-view.jpg new file mode 100644 index 0000000..cf9480c Binary files /dev/null and b/docs/_static/smartem/measurements-stations-view.jpg differ diff --git a/docs/_static/smartem/measurements-table.jpg b/docs/_static/smartem/measurements-table.jpg new file mode 100644 index 0000000..397afd3 Binary files /dev/null and b/docs/_static/smartem/measurements-table.jpg differ diff --git a/docs/_static/smartem/smartem-participants.jpg b/docs/_static/smartem/smartem-participants.jpg new file mode 100644 index 0000000..03e43ce Binary files /dev/null and b/docs/_static/smartem/smartem-participants.jpg differ diff --git a/docs/_static/smartem/smartem-sensor.jpg b/docs/_static/smartem/smartem-sensor.jpg new file mode 100644 index 0000000..530b098 Binary files /dev/null and b/docs/_static/smartem/smartem-sensor.jpg differ diff --git a/docs/_static/smartem/stations-table.png b/docs/_static/smartem/stations-table.png new file mode 100644 index 0000000..7e28021 Binary files /dev/null and b/docs/_static/smartem/stations-table.png differ diff --git a/docs/_static/smartem/stations-viewer.jpg b/docs/_static/smartem/stations-viewer.jpg new file mode 100644 index 0000000..f5c1d16 Binary files /dev/null and b/docs/_static/smartem/stations-viewer.jpg differ diff --git a/docs/data.rst b/docs/data.rst index 5d2fe9b..976fbfb 100644 --- a/docs/data.rst +++ b/docs/data.rst @@ -7,7 +7,7 @@ Data Management This chapter describes all technical aspects related to data within the SOSPilot project. The "data" as mentioned here is all related to LML RIVM data, i.e. Dutch Air Quality Data. In 2015 this setup was also used to handle air quality data from the `Smart Emission (Nijmegen) project `_. -This is described in a separate chapter. +This is described in a separate chapter: :ref:`smartemission`. * obtaining raw source data: measurements, stations etc. * data transformation (ETL) for various (OWS) services, INSPIRE and E-reporting diff --git a/docs/index.rst b/docs/index.rst index 197dc02..f3081d4 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -17,6 +17,7 @@ Contents: services.rst clients.rst administration.rst + smartemission.rst weather.rst weatherstation.rst raspberrypi-install.rst diff --git a/docs/smartemission.rst b/docs/smartemission.rst new file mode 100644 index 0000000..62003ef --- /dev/null +++ b/docs/smartemission.rst @@ -0,0 +1,570 @@ +.. _smartemission: + +====================== +Smart Emission Project +====================== + +In 2015 the SOSPilot platform was +extended to handle air quality data from +the `Smart Emission (Nijmegen) project `_. + +The same ETL components and services (WMS, WFS, SOS) as used for RIVM AQ data could be reused with some small modifications. + +Source code for the Smart Emission ETL can be found in +GitHub: https://github.com/Geonovum/sospilot/tree/master/src/smartem + +The AQ data was provided through `CityGIS `_. + +Background +========== + +Read more via the project page: `Smart Emission (Nijmegen) project `_. + +In the paper `Filling the feedback gap of place-related externalities in smart cities `_ +the project project is described extensively. + +*"...we present the set-up of the pilot experiment in project “Smart Emission”,* +*constructing an experimental citizen-sensor-network in the city of Nijmegen. This project, as part of* +*research program ‘Maps 4 Society,’ is one of the currently running Smart City projects in the* +*Netherlands. A number of social, technical and governmental innovations are put together in this* +*project: (1) innovative sensing method: new, low-cost sensors are being designed and built in the* +*project and tested in practice, using small sensing-modules that measure air quality indicators,* +*amongst others NO2, CO2, ozone, temperature and noise load. (2) big data: the measured data forms* +*a refined data-flow from sensing points at places where people live and work: thus forming a ‘big* +*picture’ to build a real-time, in-depth understanding of the local distribution of urban air quality (3)* +*empowering citizens by making visible the ‘externality’ of urban air quality and feeding this into a* +*bottom-up planning process: the community in the target area get the co-decision-making control over* +*where the sensors are placed, co-interpret the mapped feedback data, discuss and collectively explore* +*possible options for improvement (supported by a Maptable instrument) to get a fair and ‘better’* +*distribution of air pollution in the city, balanced against other spatial qualities. ...."* + +The following figures were taken from the Living Lab presentation, in June 2015: +http://www.ru.nl/publish/pages/774337/smartemission_ru_24juni_lc_v5_smallsize.pdf + +.. figure:: _static/smartem/smartem-participants.jpg + :align: center + + *Figure - Smart Emission Project - Participants* + +The Sensor (Sensor Jose) used was developed by Intemo with Server-connection by CityGIS. See below. + +.. figure:: _static/smartem/smartem-sensor.jpg + :align: center + + *Figure - Smart Emission Project - Sensors* + +The data from these sensors was used to convert and publish into standard OGC services: WMS(-Time), WFS and SOS. +This is described in the remainder of this chapter. For readers eager to see the results, these are presented +in the next chapter. + +Results +======= + +Results can be viewed in basically 3 ways: + +* as WMS and WMS-Time via the Heron Viewer: http://sensors.geonovum.nl/heronviewer +* as SOS-data via the SOS Web-Client: http://sensors.geonovum.nl/jsclient +* as raw SOS-data via SOS or easier via the SOS-REST API + +Below some guidance for each method. + +Heron Viewer +------------ + +Within the Heron Viewer at http://sensors.geonovum.nl/heronviewer one can view Stations and +Measurements. These are available as standard WMS layers organized as follows: + +* a Stations-layer: just the locations and info of all stations (triangles) +* Last Measurements Layers: per-component a Layer showing the last known measurements for each station +* Measurements History Layers: per-component a Layer showing measurements over time for each station + +To discern among RIVM LML and Smart Emission data, the latter Stations are rendered as pink icons. +The measurement data (circles) have a pink border. + +Stations Layer +~~~~~~~~~~~~~~ + +See Figure below (pink triangles). Clicking on a Station provides more detailed info via WMS ``GetFeatureInfo``. + +.. figure:: _static/smartem/stations-viewer.jpg + :align: center + + *Figure - Smart Emission Stations in Heron Viewer* + +Last Measurements Layers +~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the map/folder on the left called "Chemische Componenten (Current)". For each component +there are one or two (NO2, CO and O3)layers that can be enabled. NB the SmartEm data may not +be current. Click on a circle to see more detail. + +.. figure:: _static/smartem/heron-viewer-o3.jpg + :align: center + + *Figure - Heron Viewer showing latest known O3 Measurements* + +Measurements History Layers +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the map/folder on the left called "Chemische Componenten (Historie)". For each component +there are one or two (NO2, CO and O3)layers that can be enabled. Click on a circle to see more detail. +Use the TimeSlider on the upper right to go through the history. The image below +shows O3 at July 27, 2015, 10:00. Again pink-bordered circles denote Smart Emission measurements. + +.. figure:: _static/smartem/heron-viewer-o3-ts.jpg + :align: center + + *Figure - Heron Viewer showing O3 Measurements over time with timeslider* + +SOS Web-Client +-------------- + +The SOS Web Client by 52North: http://sensors.geonovum.nl/jsclient accesses the SOS directly. It is somewhat advanced and +takes some time to get used to. It is possible to get charts and other views. Best is to follow +the tutorial. Custom charts can be made by selecting stations on the map (Map View) and +adding these. The Smart Emission components are called O3, CO and NO2. The RIVM LML ones have +longer names starting with ``http:``. + +.. figure:: _static/smartem/jsclient-no2.jpg + :align: center + + *Figure - SOS Web Client showing NO2 Measurements in Chart* + +As can be seen the Smart Emission measurements are significantly higher. + +SOS-REST API +------------ + +Easiest is looking at results via the `SOS REST API `_. +The following are examples: + +* http://sensors.geonovum.nl/sos/api/v1/stations REST API, Stations +* http://sensors.geonovum.nl/sos/api/v1/phenomena REST API, Phenomena +* http://sensors.geonovum.nl/sos/api/v1/timeseries REST API, All Time Series List +* http://sensors.geonovum.nl/sos/api/v1/timeseries/260 REST API, Single Time Series MetaData +* http://sensors.geonovum.nl/sos/api/v1/timeseries/100/getData?timespan=PT48H/2014-09-06 REST API, Time Series Data +* http://sensors.geonovum.nl/sos/api/v1/timeseries/260/getData?timespan=2015-07-21TZ/2015-07-28TZ REST API, Time Series Data + + +Architecture +============ + +Figure 2 sketches the overall SOSPilot architecture with emphasis on the flow of data (arrows). +Circles depict harvesting/ETL processes. Server-instances are in rectangles. Datastores +the "DB"-icons. + +.. figure:: _static/sospilot-arch0.jpg + :align: center + + *Figure 2 - Overall Architecture* + +Figure 1 sketches the approach for RIVM LML AQ data, but this same approach was used voor Smart Emission. For "RIVM LML File Server" one should read: +"Raw Smart Emission Sample Data". + + +ETL Design +========== + +In this section the ETL is elaborated in more detail as depicted in the figure below. +Figure 3 sketches the approach for RIVM LML AQ data, but this same approach was used voor Smart Emission. +Also here: for "RIVM LML File Server" one should read: +"Raw Smart Emission Sample Data". Step 2 and Step 3 are identical, an advantage of the multi-step ETL process now pays back! +Step 1 was performed more manually, awaiting a similar automated approach. + +.. figure:: _static/sospilot-arch1.jpg + :align: center + + *Figure 3 - Overall Architecture with ETL Steps* + +The ETL design comprises three main processing steps and three datastores. The three ETL Steps are: + +#. File Harvester: manually (for now) via FTP +#. AQ ETL: transform this local source data to intermediate "Core AQ Data" in PostGIS +#. SOS ETL: transform and publish "Core AQ Data" to the 52N SOS DB via SOS-Transactions (SOS-T) + +The detailed dataflow from source to destination is as follows: + +#. AQ raw (text) files are placed on the file system (awaiting automated approach) +#. The AQ ETL process (files2measurements) reads these files Core AQ DB (Raw Measurements) +#. The Core AQ DB contains measurements + stations in regular tables 1-1 with original data, including a Time column +#. The Core AQ DB can be used for OWS (WMS/WFS) services via GeoServer (possibly using VIEW by Measurements/Stations JOIN) +#. The SOS ETL process transforms core AQ data to SOS Observations and publishes Observations using SOS-T InsertObservation +#. These three processes run continuously (via cron) +#. Each process always knows its progress and where it needs to resume, even after it has been stopped (by storing a progress/checkpoint info) + +These last two ETL processes manage their ``last sync-time`` using a separate ``progress table`` within the database. +The first (Harvester) only needs to check if a particular XML file (as they have a unique file name) has already been stored. + +Advantages of this approach: + +* backups of source data possible +* incrementally build up of history past the last month +* in case of (design) errors we can always reset the 'progress timestamp(s)' and restart anew +* simpler ETL scripts than “all-in-one", e.g. from “Core AQ DB” to "52N SOS DB" may even be in plain SQL +* migration with changed in 52N SOS DB schema simpler +* prepared for op IPR/INSPIRE ETL (source is Core OM DB) +* OWS server (WMS/WFS evt WCS) can directly use op Core OM DB (possibly via Measurements/Stations JOIN VIEW evt, see below) + +The Open Source ETL tool `Stetl, Streaming ETL `_ , is used for most of the transformation steps. +Stetl provides standard modules for building an ETL Chain via a configuration file. +This ETL Chain is a linkage of Input, Filter and Output modules. Each module is a Python class +derived from Stetl base classes. In addition a developer +may add custom modules where standard Stetl modules are not available or to specialize processing aspects. + +Stetl has been used sucessfully to publish BAG (Dutch Addresses and Buildings) to INSPIRE Addresses via +XSLT and WFS-T (to the ``deegree WFS server``) but also for transformation of Dutch topography (Top10NL and BGT) +to PostGIS. As Stetl is written in Python it is well-integrated with standard ETL and Geo-tools like GDAl/OGR, XSLT and +PostGIS. + +At runtime Stetl (via the ``stetl`` command) basically reads the config file, +creates all modules and links their inputs and outputs. This also makes for an easy programming model +as one only needs to concentrate on a single ETL step. + +ETL Step 1. - Harvester +----------------------- + +The Smart Emission FTP server provides measurements per sensor (unit) +in text files. See figure below. The raw data records per unit are divided +over multiple lines. See example below: :: + + 07/24/2015 07:25:41,P.UnitSerialnumber,1 + 07/24/2015 07:25:41,S.Longitude,5914103 + 07/24/2015 07:25:41,S.Latitude,53949942 + 07/24/2015 07:25:41,S.SatInfo,90889 + 07/24/2015 07:25:41,S.O3,163 + 07/24/2015 07:25:41,S.BottomSwitches,0 + 07/24/2015 07:25:41,S.RGBColor,16771990 + 07/24/2015 07:25:41,S.LightsensorBlue,92 + 07/24/2015 07:25:41,S.LightsensorGreen,144 + 07/24/2015 07:25:41,S.LightsensorRed,156 + 07/24/2015 07:25:41,S.AcceleroZ,753 + 07/24/2015 07:25:41,S.AcceleroY,516 + 07/24/2015 07:25:41,S.AcceleroX,510 + 07/24/2015 07:25:41,S.NO2,90 + 07/24/2015 07:25:41,S.CO,31755 + 07/24/2015 07:25:41,S.Altimeter,118 + 07/24/2015 07:25:41,S.Barometer,101101 + 07/24/2015 07:25:41,S.LightsensorBottom,26 + 07/24/2015 07:25:41,S.LightsensorTop,225 + 07/24/2015 07:25:41,S.Humidity,48618 + 07/24/2015 07:25:41,S.TemperatureAmbient,299425 + 07/24/2015 07:25:41,S.TemperatureUnit,305400 + 07/24/2015 07:25:41,S.SecondOfDay,33983 + 07/24/2015 07:25:41,S.RtcDate,1012101 + 07/24/2015 07:25:41,S.RtcTime,596503 + 07/24/2015 07:25:41,P.SessionUptime,60781 + 07/24/2015 07:25:41,P.BaseTimer,9 + 07/24/2015 07:25:41,P.ErrorStatus,0 + 07/24/2015 07:25:41,P.Powerstate,79 + 07/24/2015 07:25:51,P.UnitSerialnumber,1 + 07/24/2015 07:25:51,S.Longitude,5914103 + 07/24/2015 07:25:51,S.Latitude,53949942 + 07/24/2015 07:25:51,S.SatInfo,90889 + 07/24/2015 07:25:51,S.O3,157 + 07/24/2015 07:25:51,S.BottomSwitches,0 + +Each record starts on a line that contains ``P.UnitSerialnumber`` and runs to the next line +containing ``P.UnitSerialnumber`` or the end-of-file is reached. Each record contains +zero to three chemical component values named: ``S.CO`` (Carbon Monoxide), ``S.NO2`` (Nitrogen Dioxide) +or ``S.O3`` (Ozone), and further fields such as location (Latitude, Longitude) and +weather data (Temperature, Pressure). All fields have the same timestamp, e.g. ``07/24/2015 07:25:41``. +This is taken as the timestamp of the record. + +According to CityGIS the units are defined as follows. :: + + S.TemperatureUnit milliKelvin + S.TemperatureAmbient milliKelvin + S.Humidity %mRH + S.LightsensorTop Lux + S.LightsensorBottom Lux + S.Barometer Pascal + S.Altimeter Meter + S.CO ppb + S.NO2 ppb + S.AcceleroX 2 ~ +2G (0x200 = midscale) + S.AcceleroY 2 ~ +2G (0x200 = midscale) + S.AcceleroZ 2 ~ +2G (0x200 = midscale) + S.LightsensorRed Lux + S.LightsensorGreen Lux + S.LightsensorBlue Lux + S.RGBColor 8 bit R, 8 bit G, 8 bit B + S.BottomSwitches ? + S.O3 ppb + S.CO2 ppb + S.AudioMinus5 Octave -5 in dB(A) + S.AudioMinus4 Octave -4 in dB(A) + S.AudioMinus3 Octave -3 in dB(A) + S.AudioMinus2 Octave -2 in dB(A) + S.AudioMinus1 Octave -1 in dB(A) + S.Audio0 Octave 0 in dB(A) + S.AudioPlus1 Octave +1 in dB(A) + S.AudioPlus2 Octave +2 in dB(A) + S.AudioPlus3 Octave +3 in dB(A) + S.AudioPlus4 Octave +4 in dB(A) + S.AudioPlus5 Octave +5 in dB(A) + S.AudioPlus6 Octave +6 in dB(A) + S.AudioPlus7 Octave +7 in dB(A) + S.AudioPlus8 Octave +8 in dB(A) + S.AudioPlus9 Octave +9 in dB(A) + S.AudioPlus10 Octave +10 in dB(A) + S.SatInfo ? + S.Latitude *100 + Fractional degrees + S.Longitude *10 + Fractional degrees + +As stated above: this step, acquiring files is done via FTP. + +ETL Step 2 - Raw Measurements +----------------------------- + +This step produces raw AQ measurements, "AQ ETL" in Figure 2, from raw source (file) data harvested +in Step 1. The results of this step can be accessed via WMS and WFS, directly in the +project Heron viewer: http://sensors.geonovum.nl/heronviewer + +Two tables: ``stations`` and ``measurements``. This is a 1:1 transformation from the raw text. +The ``measurements`` refers to the ``stations`` by a FK ``unit_id``. + +Stations +~~~~~~~~ + +Station info has been assembled in a CSV file: +https://github.com/Geonovum/sospilot/tree/master/src/smartem/stations.csv :: + + UnitId,Name,Municipality,Lat,Lon,Altitude,AltitudeUnit + 1,Nijmegen-1,Nijmegen,51.94,5.90,30,m + 3,Nijmegen-3,Nijmegen,51.80,6.00,30,m + 5,Nijmegen-5,Nijmegen,51.85,5.95,30,m + 7,Nijmegen-7,Nijmegen,51.91,6.10,30,m + 8,Nijmegen-8,Nijmegen,51.87,5.80,30,m + 9,Nijmegen-9,Nijmegen,51.92,6.20,30,m + 10,Nijmegen-10,Nijmegen,51.89,5.85,30,m + +This info was deducted from the raw measurements files. NB: the Lat,Lon values +were inaccurate. This is still under investigation. +**For the sake of the project Lat,Lon values have been randomly altered here!**. +This will need to be corrected at a later stage. + +.. figure:: _static/smartem/stations-table.png + :align: center + + *Figure - Stations Read into Postgres/PostGIS* + +Test by viewing in http://sensors.geonovum.nl/heronviewer +See result (pink triangles). Clicking on a station provides more detailed info via WMS ``GetFeatureInfo``. + +.. figure:: _static/smartem/stations-viewer.jpg + :align: center + + *Figure - Smart Emission Stations in Heron Viewer* + +Measurements +~~~~~~~~~~~~ + +Reading raw measurements from the files is done with a ``Stetl`` +process. A specific Stetl Input module was developed to effect reading and parsing the files +and tracking the last id of the file processed. +https://github.com/Geonovum/sospilot/blob/master/src/smartem/raw2measurements.py + +Unit Conversion: as seen above the units for chemical components are in ``ppb`` (Parts-Per-Billion). +For AQ data the usual unit is ug/m3 (Microgram per cubic meter). The conversion +from ppb to ug/m3 is well-known and is dependent on molecular weight, temperature +and pressure. See more detail here: http://www.apis.ac.uk/unit-conversion. Some investigation: :: + + # Zie http://www.apis.ac.uk/unit-conversion + # ug/m3 = PPB * moleculair gewicht/moleculair volume + # waar molec vol = 22.41 * T/273 * 1013/P + # + # Typical values: + # Nitrogen dioxide 1 ppb = 1.91 ug/m3 bij 10C 1.98, bij 30C 1.85 --> 1.9 + # Ozone 1 ppb = 2.0 ug/m3 bij 10C 2.1, bij 30C 1.93 --> 2.0 + # Carbon monoxide 1 ppb = 1.16 ug/m3 bij 10C 1.2, bij 30C 1.1 --> 1.15 + # + # Benzene 1 ppb = 3.24 ug/m3 + # Sulphur dioxide 1 ppb = 2.66 ug/m3 + # + +For now a crude approximation as the measurements themselves are also not very accurate (another issue). +In `raw2measurements.py `_: :: + + record['sample_value'] = Record2MeasurementsFilter.ppb_to_ugm3_factor[component_name] * ppb_val + +with ``Record2MeasurementsFilter.ppb_to_ugm3_factor``: :: + + # For now a crude conversion (1 atm, 20C) + ppb_to_ugm3_factor = {'o3': 2.0, 'no2': 1.9, 'co': 1.15} + +The Stetl process is defined in +https://github.com/Geonovum/sospilot/blob/master/src/smartem/files2measurements.cfg + +The invokation of that Stetl process is via shell script: +https://github.com/Geonovum/sospilot/blob/master/src/smartem/files2measurements.sh + +The data is stored in the ``measurements`` table, as below. ``station_id`` is a foreign key +into the ``stations`` table. + +.. figure:: _static/smartem/measurements-table.jpg + :align: center + + *Figure - Smart Emission raw measurements stored in Postgres* + +Using a Postgres VIEW the two tables can be combined via an ``INNER JOIN`` to provide measurements +with location. This VIEW can be used as a WMS/WFS data source in GeoServer. + +.. figure:: _static/smartem/measurements-stations-view.jpg + :align: center + + *Figure - Postgres VIEW combining measurements and stations (units)* + +The VIEW is defined in https://github.com/Geonovum/sospilot/blob/master/src/smartem/db/db-schema.sql: :: + + CREATE VIEW smartem.measurements_stations AS + SELECT m.gid, m.station_id, s.name, s.municipality, m.component, m.sample_time, m.sample_value, + m.sample_value_ppb, s.point, s.lon, s.lat,m.insert_time, m.sample_id,s.unit_id, s.altitude + FROM smartem.measurements as m + INNER JOIN smartem.stations as s ON m.station_id = s.unit_id; + + +ETL Step 3 - SOS Publication +---------------------------- + +In this step the Raw Measurements data (see Step 2) is transformed to "SOS Ready Data", +i.e. data that can be handled by the 52North SOS server. This is done via +SOS Transaction (SOS-T) services using ``Stetl``. + +SOS Publication - Stetl Strategy +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As Stetl only supports WFS-T, not yet SOS, a SOS Output module ``sosoutput.py`` was developed derived +from the standard ``httpoutput.py`` module. +See https://github.com/Geonovum/sospilot/blob/master/src/smartem/sosoutput.py (this version was slightly +adapted from the version used for RIVM LML). + +Most importantly, the raw Smart Emission Measurements data +from Step 2 needs to be transformed to OWS O&M data. This is done via``substitutable templates``, like the +Stetl config itself also applies. This means we develop files with SOS Requests in which all variable parts get a +symbolic value like ``{sample_value}``. These templates can be found under +https://github.com/Geonovum/sospilot/tree/master/src/smartem/sostemplates in particular + +* https://github.com/Geonovum/sospilot/blob/master/src/smartem/sostemplates/insert-sensor.json InsertSensor +* https://github.com/Geonovum/sospilot/blob/master/src/smartem/sostemplates/delete-sensor.json DeleteSensor +* https://github.com/Geonovum/sospilot/blob/master/src/smartem/sostemplates/procedure-desc.xml Sensor ML +* https://github.com/Geonovum/sospilot/blob/master/src/smartem/sostemplates/insert-observation.json InsertObservation + +Note that we use JSON for the requests, as this is simpler than XML. The Sensor ML is embedded in the +``insert-sensor`` JSON request. + + +SOS Publication - Sensors +~~~~~~~~~~~~~~~~~~~~~~~~~ + +This step needs to be performed only once, or when any of the original Station data (CSV) changes. + +The Stetl config https://github.com/Geonovum/sospilot/blob/master/src/smartem/stations2sensors.cfg +uses a Standard Stetl module, ``inputs.dbinput.PostgresDbInput`` for obtaining Record data from a Postgres database. :: + + {{ + "request": "InsertSensor", + "service": "SOS", + "version": "2.0.0", + "procedureDescriptionFormat": "http://www.opengis.net/sensorML/1.0.1", + "procedureDescription": "{procedure-desc.xml}", + "observableProperty": [ + "CO", + "NO2", + "O3" + ], + "observationType": [ + "http://www.opengis.net/def/observationType/OGC-OM/2.0/OM_Measurement" + ], + "featureOfInterestType": "http://www.opengis.net/def/samplingFeatureType/OGC-OM/2.0/SF_SamplingPoint" + }} + +The SOSTOutput module will expand ``{procedure-desc.xml}`` with the Sensor ML template from +https://github.com/Geonovum/sospilot/blob/master/src/smartem/sostemplates/procedure-desc.xml. + +SOS Publication - Observations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Stetl config https://github.com/Geonovum/sospilot/blob/master/src/smartem/measurements2sos.cfg +uses an extended Stetl module (``inputs.dbinput.PostgresDbInput``) for obtaining Record data from a Postgres database: +https://github.com/Geonovum/sospilot/blob/master/src/smartem/measurementsdbinput.py. +This is required to track progress in the ``etl_progress`` table similar as in Step 2. +The ``last_id`` is remembered. + +The Observation template looks as follows. :: + + {{ + "request": "InsertObservation", + "service": "SOS", + "version": "2.0.0", + "offering": "SmartEmission-Offering-{unit_id}", + "observation": {{ + "identifier": {{ + "value": "{sample_id}", + "codespace": "http://www.opengis.net/def/nil/OGC/0/unknown" + }}, + "type": "http://www.opengis.net/def/observationType/OGC-OM/2.0/OM_Measurement", + "procedure": "SmartEmission-Unit-{unit_id}", + "observedProperty": "{component}", + "featureOfInterest": {{ + "identifier": {{ + "value": "SmartEmission-FID-{unit_id}", + "codespace": "http://www.opengis.net/def/nil/OGC/0/unknown" + }}, + "name": [ + {{ + "value": "{municipality}", + "codespace": "http://www.opengis.net/def/nil/OGC/0/unknown" + }} + ], + "geometry": {{ + "type": "Point", + "coordinates": [ + {lat}, + {lon} + ], + "crs": {{ + "type": "name", + "properties": {{ + "name": "EPSG:4326" + }} + }} + }} + }}, + "phenomenonTime": "{sample_time}", + "resultTime": "{sample_time}", + "result": {{ + "uom": "ug/m3", + "value": {sample_value} + }} + }} + }} + +It is quite trivial in ``sosoutput.py`` to substitute these values from the ``measurements``-table records. + +Like in ETL Step 2 the progress is remembered in the table ``rivm_lml.etl_progress`` by updating the ``last_id`` field +after publication, where that value represents the ``gid`` value of ``rivm_lml.measurements``. + +SOS Publication - Results +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Via the standard SOS protocol the results can be tested: + +* GetCapabilities: http://sensors.geonovum.nl/sos/service?service=SOS&request=GetCapabilities +* DescribeSensor (station 807, Hellendoorn): http://tinyurl.com/mmsr9hl (URL shortened) +* GetObservation: http://tinyurl.com/ol82sxv (URL shortened) + +Easier is looking at results via the `SOS REST API `_. +The following are examples: + +* http://sensors.geonovum.nl/sos/api/v1/stations REST API, Stations +* http://sensors.geonovum.nl/sos/api/v1/phenomena REST API, Phenomena +* http://sensors.geonovum.nl/sos/api/v1/timeseries REST API, All Time Series List +* http://sensors.geonovum.nl/sos/api/v1/timeseries/260 REST API, Single Time Series MetaData +* http://sensors.geonovum.nl/sos/api/v1/timeseries/100/getData?timespan=PT48H/2014-09-06 REST API, Time Series Data +* http://sensors.geonovum.nl/sos/api/v1/timeseries/260/getData?timespan=2015-07-21TZ/2015-07-28TZ REST API, Time Series Data + + + + + + diff --git a/src/smartem/readme.txt b/src/smartem/readme.txt index e37a3a0..f082d1e 100644 --- a/src/smartem/readme.txt +++ b/src/smartem/readme.txt @@ -4,12 +4,79 @@ Uses host-specific variables for databases, passwords etc. To use define a file on your host called options-.sh Take the example file options-sunda.sh for initial values and adapt to your host. -NB for now only raw files are processed, i.e. ETL Step 1 (Harvesting) is not yet performed. +NB for now only raw files are processed, i.e. ETL Step 1 (Harvesting, harvester.* files) is not yet performed. Only ETL Steps 2 and 3. Step 2: files2measurements: put files+stations in regular DB tables (DB sensors, schema: smartem) -Step 3: publish stationas and observations from tables in Step2 to SOS +Step 3: publish stations and observations from tables in Step2 to SOS + +------------------------------ +Data stats, op 30 sept 2015: + +- 175122 measurements in DB. Deze naar SOS gepubliceerd. + +- inlezen raw data in Postgres database: 5 minuten + +- publiceren naar SOS 10 minuten per 10000, plm 3 uur voor 175122 observaties + +- per station +SELECT count(*), station_id from Smartem.measurements group by station_id + +station_id count +3 7150 +8 24470 +9 53228 +1 63454 +5 58 +10 26762 + + +----------------------------- +Get data via REST +Zie doc http://sensorweb.demo.52north.org/sensorwebclient-webapp-stable/api-doc/ + +- Search (vindt procedures, timeseries etc gerelateerd aan Smart Emission) +http://sensors.geonovum.nl/sos/api/v1/search?q=SmartEmission + +- Stations +http://sensors.geonovum.nl/sos/api/v1/stations +http://sensors.geonovum.nl/sos/api/v1/stations/67 +http://sensors.geonovum.nl/sos/api/v1/stations/68 +http://sensors.geonovum.nl/sos/api/v1/stations/69 + +- Procedures (Sensor stations) +http://sensors.geonovum.nl/sos/api/v1/procedures +http://sensors.geonovum.nl/sos/api/v1/procedures/119 +http://sensors.geonovum.nl/sos/api/v1/procedures/120 +http://sensors.geonovum.nl/sos/api/v1/procedures/121 +http://sensors.geonovum.nl/sos/api/v1/procedures/122 +http://sensors.geonovum.nl/sos/api/v1/procedures/123 +http://sensors.geonovum.nl/sos/api/v1/procedures/124 +http://sensors.geonovum.nl/sos/api/v1/procedures/125 + +- Offerings +http://sensors.geonovum.nl/sos/api/v1/offerings +http://sensors.geonovum.nl/sos/api/v1/offerings/120 +http://sensors.geonovum.nl/sos/api/v1/offerings/121 +http://sensors.geonovum.nl/sos/api/v1/offerings/123 +http://sensors.geonovum.nl/sos/api/v1/offerings/125 + +- Timeseries Data +http://sensors.geonovum.nl/sos/api/v1/timeseries/ + +# metadata Unit-10 O3 procedure 125 +http://sensors.geonovum.nl/sos/api/v1/timeseries/259 + +# metadata Unit-10 NO2 procedure 125 +# Heeft data van: 2015-07-21 10:47:57 t/m 2015-07-28 07:11:08 +http://sensors.geonovum.nl/sos/api/v1/timeseries/260 + +# Get data in ISO_8601 timerange spec: https://en.wikipedia.org/wiki/ISO_8601#Time_intervals +http://sensors.geonovum.nl/sos/api/v1/timeseries/260/getData?timespan=2015-07-21TZ/2015-07-28TZ + +# CO SmartEmission-Unit-10 procedure 125 +http://sensors.geonovum.nl/sos/api/v1/timeseries/261 diff --git a/www/index.html b/www/index.html index 8da97db..13f2c5c 100644 --- a/www/index.html +++ b/www/index.html @@ -95,7 +95,9 @@

SOSPilot

Eionet. For this purpose a technical platform, "SOSPilot" has been developed. This website provides links to various elements of the SOSPilot platform. In addition, a weather station has been added to the platform. - + In 2015, Air Quality data from the + Smart Emission project + (via CityGIS.nl) was added to the platform.

Documentation

@@ -267,9 +269,7 @@

Web Services

52North SOS server

- Version: 20140521-SNAPSHOT with later patches from 52N
See - REST API doc - for the simpler REST API requests. + Version: 20140521-SNAPSHOT with later patches from 52N

  • @@ -312,17 +312,33 @@

    52North SOS server

    GetObservation (all Temperature Observations for Geonovum Weather Station, Amersfoort)
    • +
+

+ The REST API provides easier access to the SOS. + See REST API doc + for the REST API Syntax. Below some example REST API requests +

+ + REST API, Stations: http://sensors.geonovum.nl/sos/api/v1/stations + +
  • + REST API, Phenomena: http://sensors.geonovum.nl/sos/api/v1/phenomena/ +
    • +
  • + REST API, All Time Series List: http://sensors.geonovum.nl/sos/api/v1/timeseries +
    • +
  • + REST API, Single Time Series MetaData: http://sensors.geonovum.nl/sos/api/v1/timeseries/260 +
    • +
  • + REST API, Time Series Data: http://sensors.geonovum.nl/sos/api/v1/timeseries/100/getData?timespan=PT48H/2014-09-06 +
    • +
  • + REST API, Time Series Data: http://sensors.geonovum.nl/sos/api/v1/timeseries/260/getData?timespan=2015-07-21TZ/2015-07-28TZ +
    • +

    GeoServer