update documentation for Kubernetes deployment

smartemission · Aug 7, 2018 · 025ba75 · 025ba75
1 parent d528e64
commit 025ba75
Show file tree

Hide file tree

Showing 3 changed files with 57 additions and 9 deletions.
diff --git a/docs/platform/installation.rst b/docs/platform/installation.rst
@@ -5,7 +5,7 @@ Installation
 ============
 
 This chapter describes the installation steps for the Smart Emission Data Platform in a regular Docker environment.
-Note that installation and maintenance on Kubernetes is described in a separate chapter.
+Note that installation and maintenance on Kubernetes is described in the :ref:`kubernetes`_ chapter.
 
 Principles
 ==========

diff --git a/docs/platform/intro.rst b/docs/platform/intro.rst
@@ -20,9 +20,9 @@ History
 =======
 
 The Smart Emission Platform was initiated and largely developed within
-the `Smart Emission Nijmegen project <http://smartemission.ruhosting.nl>`_ (see also below).
+the `Smart Emission Nijmegen project <http://smartemission.ruhosting.nl>`_ (2015-2017, see also below).
 
-The Geonovum/RIVM `SOSPilot Project <http://sensors.geonovum.nl>`_, where RIVM LML
+The Geonovum/RIVM `SOSPilot Project <http://sensors.geonovum.nl>`_ (2014-2015) , where RIVM LML
 (Dutch national Air Quality Data) data was harvested and serviced via the OGC Sensor Observation Service (SOS), was
 a precursor for the architecture and approach to ETL with sensor data.
 
@@ -32,7 +32,8 @@ In and after 2017 several other projects, web-clients and sensor-types started u
 * the `Smart City Living Lab <https://slimstestad.nl/programma-2017-2018/>`_: around 7 major cities within NL deployed Intemo sensor stations
 * `AirSensEUR <http://www.airsenseur.org/>`_ - a EU JRC initiative for an Open Sensor HW/SW platform
 
-This put more strain on the platform and required a more structural development and maintenance approach (than project-based funding).
+This put more strain on the platform and required a more structural development and
+maintenance approach (than project-based funding).
 
 In 2018, the SE Platform was migrated to the Dutch National GDI infrastructure `PDOK <https://pdok.nl>`_ maintained
 by the `Dutch Kadaster <https://www.kadaster.nl/>`_.

diff --git a/docs/platform/kubernetes.rst b/docs/platform/kubernetes.rst
@@ -4,26 +4,55 @@
 Kubernetes
 ==========
 
-This chapter describes the installation and maintenance for the Smart Emission Data Platform in a Kubernetes environment.
-Note that installation and maintenance on regular Docker is described in a separate chapter.
+This chapter describes the installation and maintenance for the Smart Emission Data Platform in a
+`Kubernetes (K8s) <https://kubernetes.io/>`_ environment.
+Note that installation and maintenance in a Docker environment is described in
+the :ref:`installation` chapter. SE was initially (2016-2018) deployed as Containers on a single "bare Docker" machine.
+Later with the use of `docker-compose` and Docker Hub but still "bare Docker". In spring 2018 migration within Kadaster-PDOK
+to K8s started, deploying in the K8s environment on Azure.
 
 Principles
 ==========
 
 These are requirements and principles to understand and install an instance of the SE platform.
-It is required to have an understanding of `Docker <https://www.docker.com>`_ and Kubernetes (K8s)
+It is required to have an understanding of `Docker <https://www.docker.com>`_ a
+nd `Kubernetes (K8s) <https://kubernetes.io/>`_
 as that is the main environment in which the SE Platform is run.
 
+Links
+=====
+
+Links to the main artefacts related to Kubernetes deployment:
+
+* K8s deployment specs and Readme: https://github.com/smartemission/kubernetes-se
+* GitHub repositories for all SE Docker Images: https://github.com/smartemission
+* Docker Images repo: https://hub.docker.com/r/smartemission
+
 Services
 ========
 
+Most Smart Emission services are deployed as follows in K8s:
+
+* deployment.yml - specifies (`Pods` for) a K8s `Deployment`
+* service.yml - describes a K8s `Service` (internal network proxy access) for the `Pods` in the `Deployment`
+* ingress.yml - rules how to route outside requests to `Service` (only if the `Service` requires outside access)
+
+Only for InfluxDB, as it requires local storage a `StatefulSet` is deployed i.s.o. a regular `Deployment`.
+
+
 InfluxDB
 --------
 
+Links
+~~~~~
+
+* K8s deployment specs: https://github.com/smartemission/kubernetes-se/tree/master/smartemission/services/influxdb
+* GitHub repo/var specs: https://github.com/smartemission/docker-se-influxdb
+
 Creation
 ~~~~~~~~
 
-Create volumes via `PersistentVolumeClaim` (pvc.yml) , one for staorage, one for backup/restore: ::
+Create two volumes via `PersistentVolumeClaim` (pvc.yml) , one for storage, one for backup/restore: ::
 
 	# Run this once to make volumes
 	apiVersion: apps/v1beta2
@@ -132,4 +161,22 @@ Here are the commands: ::
 	$ kubectl -n smartemission get pvc
 	NAME                          STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
 	influxdb-backup-influxdb-0    Bound     pvc-f127f07a-958d-11e8-beac-0a58ac1f1ed2   2Gi        RWO            default        1h
-	influxdb-storage-influxdb-0   Bound     pvc-6c3a3d85-63fb-11e8-8f98-0a58ac1f0043   5Gi        RWO            default        63d
+	influxdb-storage-influxdb-0   Bound     pvc-6c3a3d85-63fb-11e8-8f98-0a58ac1f0043   5Gi        RWO            default        63d
+
+CronJobs
+========
+
+K8s `Cronjobs` are applied for all SE ETL.
+CronJobs run jobs on a time-based schedule. These automated jobs run like Cron tasks on a Linux or UNIX system.
+
+Implementation
+--------------
+
+All ETL is based on `the Stetl ETL framework <http://stetl.org>`_. A single Docker Image based on the official Stetl Docker Image
+contains all ETL processes. Design of the ETL is described in the :ref:`data` chapter.
+
+* GitHub repository: https://github.com/smartemission/docker-se-stetl
+* Docker Image: https://hub.docker.com/r/smartemission/se-stetl
+* K8s `CronJobs`: https://github.com/smartemission/kubernetes-se/tree/master/smartemission/cronjobs
+
+