succesful run of stack: Apache/Flask/PostGIS with ETL via Stetl

docker/apache2/Dockerfile

@@ -8,10 +8,17 @@ RUN export DEBIAN_FRONTEND=noninteractive TERM=linux && \
   apt-get update && \
   apt-get -y upgrade
 
-RUN apt-get install -y openssh-server apache2 supervisor
+RUN apt-get install -y openssh-server apache2 supervisor libapache2-mod-wsgi python-flask python-psycopg2
 
 RUN mkdir -p /var/lock/apache2 /var/run/apache2 /var/run/sshd /var/log/supervisor
 
+RUN a2enmod headers
+RUN a2enmod wsgi
+RUN a2enmod proxy
+RUN a2enmod proxy_ajp
+RUN a2enmod proxy_http
+RUN a2enmod cgi
+
 COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
 
 EXPOSE 22 80

docker/apache2/README.md

@@ -12,7 +12,7 @@ sudo docker build -t geonovum/apache2 .
 
      sudo docker run -p 2222:22 -p 80:80 -t -i  geonovum/apache2
 
-This runs the image and exposes ports 2222 and 8081 mappoing these to the standard
+This runs the image and exposes ports 2222 and 8081 mapping these to the standard
 ports 22 and 80 within the container.
 
 Mapping volumes can be handly to maintain all config and data outside the container. 

docker/stetl/README.md

@@ -14,4 +14,5 @@ Note that full paths are required.
 	cd test/1_copystd
 	docker run -v `pwd`:`pwd` -w `pwd`  -t -i geonovum/stetl -c etl.cfg   
 
-Many Stetl configs require a connection to PostGIS.
+Many Stetl configs require a connection to PostGIS. This can be effected with a linked container: ``--link postgis``, or
+better using Docker networking.

docs/platform/architecture.rst

@@ -5,3 +5,9 @@ Architecture
 ============
 
 This chapter describes the (software) architecture of the Smart Emission Data Platform.
+
+Docker
+======
+
+`Docker <https://www.docker.com>`_ **...allows you to package an application with all of its dependencies into a standardized unit for software development.**.
+Read more  on https://docs.docker.com.

docs/platform/installation.rst

@@ -296,10 +296,6 @@ Steps. ::
    $ sudo docker run hello-world
    $ sudo docker run -it ubuntu bash
 
-   # cleanup non-running images
-   $ sudo docker rm -v $(sudo docker ps -a -q -f status=exited)
-   $ sudo docker rmi $(sudo docker images -f "dangling=true" -q)
-
 Install Docker-compose, for later combining Docker-images, see https://docs.docker.com/compose/install.
 Easiest via Python ``pip``. ::
 
@@ -309,13 +305,65 @@ Easiest via Python ``pip``. ::
 See also CLI utils for ``docker-compose``: https://docs.docker.com/v1.5/compose/cli/
 Now our system is ready to roll out Docker images.
 
+Handy Commands
+~~~~~~~~~~~~~~
+
+Some handly Docker commands: ::
+
+   # cleanup non-running images
+   $ sudo docker rm -v $(sudo docker ps -a -q -f status=exited)
+   $ sudo docker rmi $(sudo docker images -f "dangling=true" -q)
+
+   # go into docker image named apache2 to bash prompt
+   sudo docker exec -it apache2 bash
+
 Docker Images
 -------------
 
 Apache2
 ~~~~~~~
 
-PostGIS from Kartoza, see https://hub.docker.com/r/kartoza/postgis/ and https://github.com/kartoza/docker-postgis  ::
+Uses the generic ``geonovum/apache2`` image from GitHub. It contains the standard Apache2 server with various
+modules enabled to be able to run Python and act as a proxy to bachend service. To build: ::
+
+   # build apache2 image
+   cd ~/git/docker/apache2
+   sudo docker build -t geonovum/apache2 .
+
+The Bash-script at ``~/git/services/apache2/run-apache2.sh``  will re(run) the generic
+Apache2 Docker image with mappings to local directories of the host for the  Apache2 config, webcontent and logfiles.
+It will also link to the PostGIS container (for the Flask Python app):
+
+.. literalinclude:: ../../services/apache2/run-apache2.sh
+    :language: bash
+
+To run locally, e.g. with Vagrant, hardcode the DNS mapping in ``/etc/hosts`` : ::
+
+   127.0.0.1	local.smartemission.nl
+   127.0.0.1	local.api.smartemission.nl
+
+Inspect logfiles within the host ``/var/smartem/log/apache2`` : ::
+
+   tail -f   /var/smartem/log/apache2/api.smartem-error.log
+
+Debugging, start/stop Apache quickly within container: ::
+
+   # go into docker image named apache2 to bash prompt
+   sudo docker exec -it apache2 bash
+
+   # Kill running Apache parent process instance
+   root@ed393501ed58:/# ps -al
+   F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
+   4 S     0     8     1  0  80   0 - 15344 poll_s ?        00:00:00 sshd
+   4 S     0     9     1  0  80   0 - 23706 poll_s ?        00:00:00 apache2
+   5 S    33    10     9  0  80   0 - 23641 skb_re ?        00:00:00 apache2
+   5 S    33    11     9  0  80   0 - 96540 pipe_w ?        00:00:01 apache2
+   5 S    33    12     9  0  80   0 - 112940 pipe_w ?       00:00:01 apache2
+   0 R     0    94    81  0  80   0 -  1783 -      ?        00:00:00 ps
+   root@ed393501ed58:/# kill 9
+
+   # Start Apache from commandline
+   /bin/bash -c "source /etc/apache2/envvars && exec /usr/sbin/apache2 -DFOREGROUND"
 
 PostGIS
 ~~~~~~~

etl/last.sh

@@ -6,5 +6,10 @@
 STETL_ARGS="-c last.cfg -a options/docker.args"
 WORK_DIR="`pwd`"
 PG_HOST=postgis
+NAME="stetl"
 
-sudo docker run --link ${PG_HOST}:${PG_HOST} -v ${WORK_DIR}:${WORK_DIR} -w ${WORK_DIR} -t -i geonovum/stetl ${STETL_ARGS}
+# Stop and remove possibly old containers
+sudo docker stop ${NAME} > /dev/null 2>&1
+sudo docker rm ${NAME} > /dev/null 2>&1
+
+sudo docker run --name ${NAME} --link ${PG_HOST}:${PG_HOST} -v ${WORK_DIR}:${WORK_DIR} -w ${WORK_DIR} -t -i geonovum/stetl ${STETL_ARGS}

services/apache2/run-apache2.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+#
+# Run the Apache2 server with mapping to local config
+#
+
+GIT="/opt/geonovum/smartem/git"
+WWW="${GIT}/www"
+CONFIG="${GIT}/services/apache2"
+LOG="/var/smartem/log"
+NAME="apache2"
+IMAGE="geonovum/apache2"
+
+VOL_MAP="-v ${CONFIG}/sites-enabled:/etc/apache2/sites-enabled -v ${GIT}:${GIT} -v ${LOG}/apache2:/var/log/apache2"
+PORT_MAP="-p 2222:22 -p 80:80"
+PG_HOST="postgis"
+
+# Stop and remove possibly old containers
+sudo docker stop ${NAME} > /dev/null 2>&1
+sudo docker rm ${NAME} > /dev/null 2>&1
+
+# Finally run
+sudo docker run --name ${NAME} --link ${PG_HOST}:${PG_HOST} -d ${PORT_MAP} ${VOL_MAP} -t -i ${IMAGE}

services/apache2/sites-enabled/000-default.conf

@@ -0,0 +1,65 @@
+# SmartEmission API Web Services
+<VirtualHost *:80>
+    ServerName api.smartemission.nl
+    ServerAlias api.test.smartemission.nl
+    ServerAlias api.local.smartemission.nl
+
+    DocumentRoot /opt/geonovum/smartem/git/services/sosrest
+    ServerAdmin just@justobjects.nl
+
+    DirectoryIndex index.html index.jsp index.cgi index.fcgi
+
+	# Try with worker MPM, WSGIDaemonProcess bad perf with prefork MP?
+#    WSGIDaemonProcess smartem_sosemu user=sadmin group=sadmin processes=4 threads=4
+#	WSGIProcessGroup smartem_sosemu
+#   	WSGIApplicationGroup %{GLOBAL}
+    WSGIScriptAlias /sosemu /opt/geonovum/smartem/git/services/sosrest/index.py
+
+#    WSGIScriptAlias /istsos /opt/istsos/latest/application.py
+#    Alias /istsos/admin /opt/istsos/latest/interface/admin
+#    Alias /istsos/modules /opt/istsos/latest/interface/modules
+
+    <Directory /opt/geonovum/smartem/git/services/sosrest/>
+         Require all granted
+    </Directory>
+
+#	<LocationMatch /istsos>
+#	  Options +Indexes +FollowSymLinks +MultiViews
+#	  AllowOverride all
+#	  Require all granted
+#	</LocationMatch>
+
+	# %D voor timing in microsecs
+    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D" mpcombined
+    CustomLog /var/log/apache2/api.smartem-access.log mpcombined
+    ErrorLog /var/log/apache2/api.smartem-error.log
+</VirtualHost>
+
+# SmartEmission Web Site and Apps
+<VirtualHost *:80>
+    ServerName smartemission.nl
+    ServerAlias www.smartemission.nl
+    ServerAlias test.smartemission.nl
+    ServerAlias local.smartemission.nl
+
+    DocumentRoot /opt/geonovum/smartem/git/www/site
+    ServerAdmin just@justobjects.nl
+
+   ScriptAlias /cgi-bin/ /opt/geonovum/smartem/git/www/cgi-bin/
+    <Directory "/opt/geonovum/smartem/git/www/cgi-bin">
+         AllowOverride None
+         Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
+         Require all granted
+    </Directory>
+
+    DirectoryIndex index.html
+
+    <Directory /opt/geonovum/smartem/git/www/site/>
+         Require all granted
+    </Directory>
+
+    # %D voor timing in microsecs
+    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D" mpcombined
+    CustomLog /var/log/apache2/smartem-access.log mpcombined
+    ErrorLog /var/log/apache2/smartem-error.log
+</VirtualHost>

services/sosrest/config.py

@@ -1,9 +1,9 @@
 # local config, change on server for real config
 config = {
-    'database': 'sensors',
-    'host': 'localhost',
+    'database': 'gis',
+    'host': 'postgis',
     'port': '5432',
     'schema': 'smartem_rt',
-    'user': 'postgres',
-    'password': 'postgres'
+    'user': 'docker',
+    'password': 'docker'
 }

services/sosrest/postgis.py

@@ -36,6 +36,7 @@ def connect(self):
             log.debug("Connected to database %s" % (self.config['database']))
         except Exception, e:
             log.error("Cannot connect to database '%s'" % (self.config['database']))
+            raise
 
     def disconnect(self):
         self.e = None
@@ -50,7 +51,7 @@ def disconnect(self):
     # Do the whole thing: connecting, query, and conversion of result to array of dicts (records)
     def do_query(self, query_str, table):
         self.connect()
-    
+
         column_names = self.get_column_names(table, self.config.get('schema'))
         # print('cols=' + str(column_names))