Update documentation

earthobservations · Dec 11, 2018 · 06de57e · 06de57e
1 parent 6e83de8
commit 06de57e
Show file tree

Hide file tree

Showing 3 changed files with 127 additions and 96 deletions.
diff --git a/CHANGES.rst b/CHANGES.rst
@@ -14,6 +14,8 @@ in progress
 - Add basic RDBMS adapter for storing station list and associated
   information to Postgres and other SQL databases supported by SQLAlchemy
 - Streamline station data schema
+- Add test harness for reverse geocoder subsystem
+- Improve robustness and quality of reverse geocoder
 - Make "sensors" data substructure an array
 
 

diff --git a/README.rst b/README.rst
@@ -8,44 +8,79 @@ About
 *****
 
 Features:
+
 - Request data from live API of luftdaten.info
 - Enrich location data with geospatial information from OSM/Nominatim
 - Display list of stations and readings
 - Export list of stations to RDBMS database
 - Forward readings to MQTT
 
 
+Description
+===========
+
+## Details
+How does this content get produced?
+1. [Luftdatenpumpe] acquires the current window of measurement readings from the livedata API of [luftdaten.info].
+2. While iterating the readings, it collects information about all stations and sensors they are originating from.
+3. Then, each stations location information gets enhanced by
+   - attaching its geospatial position as a [Geohash].
+   - attaching a synthetic real-world address resolved using the reverse geocoding service [Nominatim] by [OpenStreetMap].
+4. The resulting data is stored into a PostgreSQL database on `weather.hiveeyes.org` using the fine [dataset] package.
+   Being built on top of [SQLAlchemy], this supports all major [RDBMS] databases.
+
+[luftdaten.info]: http://luftdaten.info/
+[Luftdatenpumpe]: https://github.com/hiveeyes/luftdatenpumpe
+[Erneuerung der Luftdatenpumpe]: https://community.hiveeyes.org/t/erneuerung-der-luftdatenpumpe/1199
+[The Hiveeyes Project]: https://hiveeyes.org/
+
+[OpenStreetMap]: https://en.wikipedia.org/wiki/OpenStreetMap
+[Nominatim]: https://wiki.openstreetmap.org/wiki/Nominatim
+[Geohash]: https://en.wikipedia.org/wiki/Geohash
+[dataset]: https://dataset.readthedocs.io/
+[SQLAlchemy]: https://www.sqlalchemy.org/
+[RDBMS]: https://en.wikipedia.org/wiki/Relational_database_management_system
+
+
+
 ****
 Demo
 ****
+
+Live Daten
+==========
 - https://luftdaten.hiveeyes.org/grafana/d/bEe6HJamk/feinstaub-verlauf-berlin
 - https://luftdaten.hiveeyes.org/grafana/d/000000004/feinstaub-karte-deutschland
-- https://weather.hiveeyes.org/grafana/d/yDbjQ7Piz/stationslistengalerie
 
+Stationslisten
+==============
+- https://weather.hiveeyes.org/grafana/d/yDbjQ7Piz/amo-ldi-stations-1-select-by-name-country-and-state
+- https://weather.hiveeyes.org/grafana/d/Oztw1OEmz/amo-ldi-stations-2-cascaded-stations
+- https://weather.hiveeyes.org/grafana/d/lT4lLcEiz/amo-ldi-stations-3-cascaded-measurements
+- https://weather.hiveeyes.org/grafana/d/kMIweoPik/amo-ldi-stations-4-select-by-sensor-type
 
-**********
-References
-**********
 
-Upstream luftdaten.info
-=======================
-- http://luftdaten.info/
-- http://archive.luftdaten.info/
-- http://deutschland.maps.luftdaten.info/
+********
+Synopsis
+********
+
+List stations
+=============
+::
+
+    luftdatenpumpe stations --stations=28,297 --reverse-geocode
 
-Standing on the shoulders of giants
-===================================
-- https://github.com/vinsci/geohash/
-- https://github.com/openstreetmap/Nominatim
-- https://github.com/influxdata/influxdb
-- https://github.com/grafana/grafana
-- https://grafana.com/plugins/grafana-worldmap-panel
 
-Development
-===========
-- `Support for InfluxDB and MQTT as backend <https://github.com/opendata-stuttgart/sensors-software/issues/33#issuecomment-272711445>`_.
-- https://getkotori.org/docs/applications/luftdaten.info/
-- https://community.hiveeyes.org/t/erneuerung-der-luftdatenpumpe/1199
+
+MQTT forwarding
+===============
+::
+
+    luftdatenpumpe forward --stations=28,1071 --mqtt-uri mqtt://mqtt.example.org/luftdaten.info
+
+With authentication::
+
+    luftdatenpumpe forward --stations=28,1071 --mqtt-uri mqtt://username:password@mqtt.example.org/luftdaten.info
 
 
 *****
@@ -82,10 +117,10 @@ Redis cache
 This program extensively uses a runtime cache based on Redis.
 To make this work best, you should enable data durability with your Redis instance.
 
-> The append-only file is an alternative, fully-durable strategy for Redis. It became available in version 1.1.
-> You can turn on the AOF in your Redis configuration file (e.g. `/etc/redis/redis.conf`)::
->
->   appendonly yes
+    The append-only file is an alternative, fully-durable strategy for Redis. It became available in version 1.1.
+    You can turn on the AOF in your Redis configuration file (e.g. `/etc/redis/redis.conf`)::
+
+        appendonly yes
 
 
 Python module
@@ -96,44 +131,50 @@ Python module
     pip install luftdatenpumpe
 
 
-********
-Synopsis
-********
-
-List stations
-=============
-::
-
-    luftdatenpumpe stations --stations=28,297 --reverse-geocode
-
 
+**********
+References
+**********
 
-MQTT forwarding
-===============
-::
+Upstream luftdaten.info
+=======================
+- http://luftdaten.info/
+- http://archive.luftdaten.info/
+- http://deutschland.maps.luftdaten.info/
 
-    luftdatenpumpe forward --stations=28,1071 --mqtt-uri mqtt://mqtt.example.org/luftdaten.info
+Standing on the shoulders of giants
+===================================
+- https://github.com/vinsci/geohash/
+- https://github.com/openstreetmap/Nominatim
+- https://github.com/influxdata/influxdb
+- https://github.com/grafana/grafana
+- https://grafana.com/plugins/grafana-worldmap-panel
 
-With authentication::
+Development
+===========
+- `opendata-stuttgart/sensors-software: Support for InfluxDB and MQTT as backend <https://github.com/opendata-stuttgart/sensors-software/issues/33#issuecomment-272711445>`_.
+- https://getkotori.org/docs/applications/luftdaten.info/
+- https://community.hiveeyes.org/t/datenmischwerk/702
+- https://community.hiveeyes.org/t/environmental-metadata-library/1190
+- https://community.hiveeyes.org/t/erneuerung-der-luftdatenpumpe/1199
 
-    luftdatenpumpe forward --stations=28,1071 --mqtt-uri mqtt://username:password@mqtt.example.org/luftdaten.info
 
 
+*******
 License
-=======
-
-   This program is free software; you can redistribute it and/or modify
-   it under the terms of the GNU Affero General Public License as published by
-   the Free Software Foundation; either version 3 of the License, or
-   (at your option) any later version.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU General Public License for more details.
-
-   You should have received a copy of the GNU Affero General Public License
-   along with this program; if not, see:
-   <http://www.gnu.org/licenses/agpl-3.0.txt>,
-   or write to the Free Software Foundation,
-   Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301  USA
+*******
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation; either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program; if not, see:
+<http://www.gnu.org/licenses/agpl-3.0.txt>,
+or write to the Free Software Foundation,
+Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301  USA
diff --git a/doc/backlog.rst b/doc/backlog.rst
@@ -8,9 +8,7 @@ Prio 1
 ******
 - [x] Download cache for data feed (5 minutes)
 - [x] Write metadata directly to Postgres
-- [o] Refactor for handling multiple data sources and targets
 - [o] Redesign commandline interface
-- [o] Write measurement data directly to InfluxDB
 - [o] Create CHANGES.rst, update documentation and repository (tags)
 - [o] Add tooling for packaging
 - [o] Publish to PyPI
@@ -19,51 +17,41 @@ Prio 1
 ******
 Prio 2
 ******
+- [o] Write measurement data directly to InfluxDB
+- [o] Refactor for handling multiple data sources and targets
+- [o] Get English (or configurable) country labels from Nominatim
+- [o] Handle multiple languages with Nominatim. Use English as default.
+- [o] Make a sensor type chooser in Grafana. How would that actually select multiple(!) stations by id through Grafana?
 - [o] Run some metric about total count of measuremnts per feed action
 - [o] Export to tabular format: http://docs.python-tablib.org/
 - [o] Output data in tabular, markdown or rst formats
 - [o] Publish to MQTT with separate topics
-- [o] Store stations / data **while** processing
+- [/] Store stations / data **while** processing
+- [o] Store "boundingbox" attribute to RDBMS database
 
 
 ******
 Prio 3
 ******
+- [o] OSM: Why are some roads or towns empty?
+      weatherbase=# select * from ldi_osmdata where road is null limit 7;
+- [o] Add remark after "licence": "Data \u00a9 OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright" like
+      "remark": "The address information has been modified by luftdatenpumpe 0.4.0"
+- [o] OSM: English labels for e.g. Hercegovine, BA
+- [o] Database view
+      https://www.postgresql.org/docs/9.2/sql-createview.html
+      on top of
+      https://community.hiveeyes.org/t/erneuerung-der-luftdatenpumpe/1199/25
+
+
+******
+Prio 4
+******
 - [o] Write metadata directly to PostGIS
   https://dataset.readthedocs.io/en/latest/
 - [o] Add support for JSON and GIS data to "dataset" module
-
-
-****
-Misc
-****
-::
-
-    #result = self.db.query(self.stationtable.join(self.osmtable))
-
-    #statement = table.select(table.c.name.like('%John%'))
-    #result = self.db.query(statement)
-
-    #q = session.query(User).join(User.addresses)
-
-
-    # https://docs.sqlalchemy.org/en/latest/core/selectable.html#sqlalchemy.sql.expression.CompoundSelect.join
-    from sqlalchemy import select, join
-    from sqlalchemy.orm import aliased
-    table_stations = self.stationtable.table
-    table_osmdata = self.osmtable.table
-    rel = table_stations.join(table_osmdata, table_stations.c.location_id == table_osmdata.c.location_id)
-    stmt = select([table_stations]).select_from(rel)
-    result = self.db.query(stmt)
-    #result = self.db.query(select([rel]))
-    #result = self.db.query(table_stations.select(), table_osmdata).filter(table_stations.c.location_id == table_osmdata.location_id).all()
-    #result = self.db.query(self.stationtable, self.osmtable) #.filter(table_stations.c.location_id == table_osmdata.location_id).all()
-
-    fa = aliased(table_osmdata)
-    #j = join(table_stations, fa, table_stations.c.location_id == fa.c.location_id, isouter=True, full=True)
-    j = join(table_stations, table_osmdata, table_stations.c.location_id == table_osmdata.c.location_id)
-    stmt = select([table_stations, table_osmdata.c.display_name]).select_from(j)
-    stmt = table_stations.join(fa, table_stations.c.location_id == fa.c.location_id)
-    print(dir(stmt))
-    #result = self.db.query(stmt.select())
-    #result = self.db.query(table_stations).join(table_osmdata)
+- [o] OSM: Italia only has 3-letter state names like CAL, CAM, LOM, etc.
+- [o] Add PostgREST
+- [o] Import historical data from http://archive.luftdaten.info/
+- [o] Grafana: Link to https://www.madavi.de/sensor/graph.php somehow?
+- [o] After importing historical data, make a video from the expanding map