add documentation , ao SensorThings API

smartemission · Aug 2, 2018 · c95e129 · c95e129
1 parent b0c759a
commit c95e129
Show file tree

Hide file tree

Showing 4 changed files with 163 additions and 50 deletions.
diff --git a/docs/platform/data.rst b/docs/platform/data.rst
@@ -1133,6 +1133,24 @@ Finally the ``POST Observation`` STA template: ::
 	  }}
 	}}
 
+Entity Mapping
+--------------
+
+Data records produced by the Refinerare mapped to STA Entities by the STA Publisher.
+
+=======================     ======================  ===============================================================
+SE Artefact                 STA Entity              Example
+=======================     ======================  ===============================================================
+Station and location        `Thing`                 Intemo station  AirSensEUR Box at lat/lon
+Sensor Type/Metadata        `Sensor`                AlphaSense NO2B43F
+Type and unit (uom)         `ObservedProperty`      NO2 in ug/m3
+Value and time              `Observation`           42 ug/m3 on 1 aug 2018 13:42:45
+Combination of above        `Datastream`            Combines T, S, OP and O
+Station time+location       `HistoricalLocation`    AirSensEUR Box at lat/lon 52.35,4.92 on on 1 aug 2018 13:42:45
+Station Area                `FeatureOfInterest`     Location of Station 11820004
+=======================     ======================  ===============================================================
+
+
 Deleting STA Entities
 ~~~~~~~~~~~~~~~~~~~~~
 

diff --git a/docs/platform/index.rst b/docs/platform/index.rst
@@ -18,6 +18,7 @@ Contents:
    api.rst
    code.rst
    installation.rst
+   kubernetes.rst
    administration.rst
    dissemination.rst
    cookbook.rst

diff --git a/docs/platform/installation.rst b/docs/platform/installation.rst
@@ -4,7 +4,8 @@
 Installation
 ============
 
-This chapter describes the installation steps for the Smart Emission Data Platform.
+This chapter describes the installation steps for the Smart Emission Data Platform in a regular Docker environment.
+Note that installation and maintenance on Kubernetes is described in a separate chapter.
 
 Principles
 ==========
@@ -317,57 +318,18 @@ Uses the ``geonovum/stetl`` image with Stetl config from GitHub for all ETL proc
 The ``last.sh`` script is a wrapper to run the generic Docker ``geonovum/stetl`` with our
 local ETL-config and PostGIS:
 
-.. literalinclude:: ../../etl/last.sh
+.. literalinclude:: ../../etl/harvester_last.sh
     :language: bash
 
 
-web - Web Container
--------------------
-
-Uses the generic ``geonovum/apache2`` Docker Image from GitHub. It contains the standard Apache2 server with various
-modules enabled to be able to run Python and act as a proxy to backend services. To build: ::
-
-   # build apache2 image
-   cd ~/git/docker/apache2
-   sudo docker build -t geonovum/apache2 .
-
-The Bash-script at ``~/git/services/web/run.sh``  will re(run) the generic
-Apache2 Docker image with mappings to local directories of the host for the  Apache2 config, webcontent and logfiles.
-It will also link to the PostGIS Container (for the Flask Python app):
-
-.. literalinclude:: ../../services/web/run.sh
-    :language: bash
+webapps - Web Containers
+------------------------
 
-To run locally, e.g. with Vagrant, hardcode the DNS mapping in ``/etc/hosts`` : ::
+Each webapp has its own Docker image and is started via `docker-compose`.
 
-   127.0.0.1	local.smartemission.nl
-   127.0.0.1	local.data.smartemission.nl
+.. literalinclude:: ../../apps/home/docker-compose.yml
+    :language: YAML
 
-Inspect logfiles within the host ``/var/smartem/log/apache2`` : ::
-
-   tail -f   /var/smartem/log/apache2/data.smartem-error.log
-
-Debugging, start/stop Apache quickly within container: ::
-
-   # go into docker image named apache2 to bash prompt
-   sudo docker exec -it apache2 bash
-
-   # Kill running Apache parent process instance
-   root@ed393501ed58:/# ps -al
-   F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
-   4 S     0     8     1  0  80   0 - 15344 poll_s ?        00:00:00 sshd
-   4 S     0     9     1  0  80   0 - 23706 poll_s ?        00:00:00 apache2
-   5 S    33    10     9  0  80   0 - 23641 skb_re ?        00:00:00 apache2
-   5 S    33    11     9  0  80   0 - 96540 pipe_w ?        00:00:01 apache2
-   5 S    33    12     9  0  80   0 - 112940 pipe_w ?       00:00:01 apache2
-   0 R     0    94    81  0  80   0 -  1783 -      ?        00:00:00 ps
-   root@ed393501ed58:/# kill 9
-
-   # Start Apache from commandline
-   /bin/bash -c "source /etc/apache2/envvars && exec /usr/sbin/apache2 -DFOREGROUND"
-
-Securing access: create the file `htpasswd` under `config/admin` (see README there) with
-users for general site admin and SensorThings publication.
 
 geoserver - GeoServer
 ---------------------
@@ -545,10 +507,7 @@ Used for getting metrics in Prometheus from Docker components. See https://githu
     histograms of complete historical resource usage and network statistics. This data is exported by container and machine-wide."
 
 NB for now cAdvisor needs to be built because of `this bug <https://github.com/google/cadvisor/issues/1802>`_.
-Once that is resolved we can use official Docker Image. The Dockerfile :
-
-.. literalinclude:: ../../docker/cadvisor/Dockerfile
-    :language: guess
+Once that is resolved we can use official Docker Image. 
 
 NB cAdvisor via Docker on Ubuntu 14.04 has a `serious issue (like Node_exporter) <https://github.com/smartemission/smartemission/issues/73>`_
 and `this issue <https://github.com/google/cadvisor/issues/771>`_,

diff --git a/docs/platform/kubernetes.rst b/docs/platform/kubernetes.rst
@@ -0,0 +1,135 @@
+.. _kubernetes:
+
+==========
+Kubernetes
+==========
+
+This chapter describes the installation and maintenance for the Smart Emission Data Platform in a Kubernetes environment.
+Note that installation and maintenance on regular Docker is described in a separate chapter.
+
+Principles
+==========
+
+These are requirements and principles to understand and install an instance of the SE platform.
+It is required to have an understanding of `Docker <https://www.docker.com>`_ and Kubernetes (K8s)
+as that is the main environment in which the SE Platform is run.
+
+Services
+========
+
+InfluxDB
+--------
+
+Creation
+~~~~~~~~
+
+Create volumes via `PersistentVolumeClaim` (pvc.yml) , one for staorage, one for backup/restore: ::
+
+	# Run this once to make volumes
+	apiVersion: apps/v1beta2
+	kind: PersistentVolumeClaim
+	metadata:
+	  name: influxdb-backup
+	spec:
+	  accessModes:
+	  - ReadWriteOnce
+	  storageClassName: default
+	  resources:
+	    requests:
+	      storage: 2Gi
+
+	---
+
+	apiVersion: apps/v1beta2
+	kind: PersistentVolumeClaim
+	metadata:
+	  name: influxdb-storage
+	spec:
+	  accessModes:
+	  - ReadWriteOnce
+	  storageClassName: default
+	  resources:
+	    requests:
+	      storage: 5Gi
+
+
+Use these in `StatefulSet` deployment: ::
+
+	apiVersion: apps/v1beta2
+	kind: StatefulSet
+	metadata:
+	  name: influxdb
+	  namespace: smartemission
+	spec:
+	  selector:
+	    matchLabels:
+	      app: influxdb
+	  serviceName: "influxdb"
+	  replicas: 1
+	  template:
+	    metadata:
+	      labels:
+	        app: influxdb
+	    spec:
+	      terminationGracePeriodSeconds: 10
+	      containers:
+	      - name: influxdb
+	        image: influxdb:1.5.3
+	        ports:
+	        - containerPort: 8086
+	        volumeMounts:
+	        - mountPath: /var/lib/influxdb
+	          name: influxdb-storage
+	        - mountPath: /backup
+	          name: influxdb-backup
+	  volumeClaimTemplates:
+	  - metadata:
+	      name: influxdb-storage
+	    spec:
+	      accessModes: [ "ReadWriteOnce" ]
+	      storageClassName: default
+	      resources:
+	        requests:
+	          storage: 5Gi
+	  - metadata:
+	      name: influxdb-backup
+	    spec:
+	      accessModes: [ "ReadWriteOnce" ]
+	      storageClassName: default
+	      resources:
+	        requests:
+	          storage: 2Gi
+
+Backup and Restore
+~~~~~~~~~~~~~~~~~~
+
+Restoring in these steps:
+
+* copy backup files into `influxdb-backup` volume
+* stop/delete  `influxdb` container
+* run `job-restore` Job
+* re-create influxdb
+
+Here are the commands: ::
+
+	# All backup files are contained in local dir influxdb
+    # influxdb/smartemission.autogen.00101.00
+    # influxdb/meta.00
+    # influxdb/smartemission.autogen.00079.00
+    # etc
+	kubectl cp influxdb  smartemission/influxdb-0:/backup/
+    # NB files will reside remotely under /backup/influxdb/*.00 etc !
+
+	# Delete in Kubernetes the StateFulSet influxdb, YES DELETE!
+
+	# Job must run on specific node
+	$ kubectl get nodes
+	NAME                       STATUS    ROLES     AGE       VERSION
+	aks-agentpool-34284374-0   Ready     agent     35d       v1.10.3
+	aks-agentpool-34284374-1   Ready     agent     35d       v1.10.3
+	aks-agentpool-34284374-2   Ready     agent     35d       v1.10.3
+
+	$ kubectl -n smartemission get pvc
+	NAME                          STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
+	influxdb-backup-influxdb-0    Bound     pvc-f127f07a-958d-11e8-beac-0a58ac1f1ed2   2Gi        RWO            default        1h
+	influxdb-storage-influxdb-0   Bound     pvc-6c3a3d85-63fb-11e8-8f98-0a58ac1f0043   5Gi        RWO            default        63d