refactor for docker working

smartemission · Apr 22, 2016 · 69882d1 · 69882d1
1 parent 211372e
commit 69882d1
Show file tree

Hide file tree

Showing 8 changed files with 81 additions and 26 deletions.
diff --git a/docker/apache2/README.md b/docker/apache2/README.md
@@ -15,11 +15,8 @@ docker build -t geonovum/apache2 .
 This runs the image and exposes ports 2222 and 8081 mappoing these to the standard
 ports 22 and 80 within the container.
 
-NB on Mac OSX via docker machine the actual IP adress is not 127.0.0.1 but e.g. 192.168.99.100
-since Docker runs within VirtualBox VM.
-
 Mapping volumes can be handly to maintain all config and data outside the container. 
 Note that full paths are required.
 
-     docker run -p 2222:22 -p 8081:80 -v `pwd`/log:/var/log/apache2 -t -i  geonovum/apache2
+     docker run -p 2222:22 -p 80:80 -v `pwd`/log:/var/log/apache2 -t -i  geonovum/apache2
 
diff --git a/docs/platform/installation.rst b/docs/platform/installation.rst
@@ -209,6 +209,53 @@ Then access Apache from local system via ``localhost:8081``.
 
    *Access Apache running with Docker externally*
 
+Same for Stetl, build and test: ::
+
+   $ cd ~/git/docker/stetl
+   $ sudo docker build -t geonovum/stetl .
+   $ cd test/1_copystd
+   $ sudo docker run -v `pwd`:`pwd` -w `pwd`  -t -i geonovum/stetl -c etl.cfg
+
+   2016-04-22 19:09:29,705 util INFO Found cStringIO, good!
+   2016-04-22 19:09:29,774 util INFO Found lxml.etree, native XML parsing, fabulous!
+   2016-04-22 19:09:29,926 util INFO Found GDAL/OGR Python bindings, super!!
+   2016-04-22 19:09:29,952 main INFO Stetl version = 1.0.9rc3
+   2016-04-22 19:09:29,961 ETL INFO INIT - Stetl version is 1.0.9rc3
+   2016-04-22 19:09:29,965 ETL INFO Config/working dir = /home/vagrant/git/docker/stetl/test/1_copystd
+   2016-04-22 19:09:29,966 ETL INFO Reading config_file = etl.cfg
+   2016-04-22 19:09:29,968 ETL INFO START
+   2016-04-22 19:09:29,968 util INFO Timer start: total ETL
+   2016-04-22 19:09:29,969 chain INFO Assembling Chain: input_xml_file|output_std...
+   2016-04-22 19:09:29,987 input INFO cfg = {'class': 'inputs.fileinput.XmlFileInput', 'file_path': 'input/cities.xml'}
+   2016-04-22 19:09:29,993 fileinput INFO file_list=['input/cities.xml']
+   2016-04-22 19:09:29,995 output INFO cfg = {'class': 'outputs.standardoutput.StandardXmlOutput'}
+   2016-04-22 19:09:29,996 chain INFO Running Chain: input_xml_file|output_std
+   2016-04-22 19:09:29,996 fileinput INFO Read/parse for start for file=input/cities.xml....
+   2016-04-22 19:09:30,008 fileinput INFO Read/parse ok for file=input/cities.xml
+   2016-04-22 19:09:30,014 fileinput INFO all files done
+   <?xml version='1.0' encoding='utf-8'?>
+   <cities>
+       <city>
+           <name>Amsterdam</name>
+           <lat>52.4</lat>
+           <lon>4.9</lon>
+       </city>
+       <city>
+           <name>Bonn</name>
+           <lat>50.7</lat>
+           <lon>7.1</lon>
+       </city>
+       <city>
+           <name>Rome</name>
+           <lat>41.9</lat>
+           <lon>12.5</lon>
+       </city>
+   </cities>
+
+   2016-04-22 19:09:30,024 chain INFO DONE - 1 rounds - chain=input_xml_file|output_std
+   2016-04-22 19:09:30,024 util INFO Timer end: total ETL time=0.0 sec
+   2016-04-22 19:09:30,026 ETL INFO ALL DONE
+
 Install Docker
 --------------
 
@@ -261,3 +308,28 @@ Easiest via Python ``pip``. ::
 See also CLI utils for ``docker-compose``: https://docs.docker.com/v1.5/compose/cli/
 Now our system is ready to roll out Docker images.
 
+Docker Images
+-------------
+
+PostGIS
+~~~~~~~
+
+PostGIS from Kartoza, see https://hub.docker.com/r/kartoza/postgis/ and https://github.com/kartoza/docker-postgis  ::
+
+   $ sudo docker pull kartoza/postgis:9.4-2.1
+   $ sudo docker run --name "postgis" -p 5432:5432 -d -t kartoza/postgis:9.4-2.1
+   $ sudo apt-get install postgresql-client-9.3
+   $ psql -h localhost -U docker  -l
+   Password for user docker: (also 'docker')
+                                    List of databases
+          Name       |  Owner   | Encoding  | Collate | Ctype |   Access privileges
+   ------------------+----------+-----------+---------+-------+-----------------------
+    gis              | docker   | UTF8      | C       | C     |
+    postgres         | postgres | SQL_ASCII | C       | C     |
+    template0        | postgres | SQL_ASCII | C       | C     | =c/postgres          +
+                     |          |           |         |       | postgres=CTc/postgres
+    template1        | postgres | SQL_ASCII | C       | C     | =c/postgres          +
+                     |          |           |         |       | postgres=CTc/postgres
+    template_postgis | postgres | UTF8      | C       | C     |
+   (5 rows)
+
diff --git a/etl/last.cfg b/etl/last.cfg
@@ -11,7 +11,7 @@ chains = input_raw_sensor_last_api|output_postgres_insert
 
 # for reading files from CityGIS Sensor REST API
 [input_raw_sensor_last_api]
-class = rawsensorlastinput.RawSensorLastInput
+class = py.rawsensorlastinput.RawSensorLastInput
 url = {raw_device_url}
 
 # for testing/debugging

diff --git a/etl/last.sh b/etl/last.sh
@@ -3,16 +3,4 @@
 # ETL for reading last values from Smart Emission CityGIS Raw Sensor API
 #
 
-# Usually required in order to have Python find your package
-export PYTHONPATH=${PWD}/python:${PYTHONPATH}
-
-stetl_cmd=stetl
-
-# debugging
-# stetl_cmd=../../../../stetl/git/stetl/main.py
-
-# Host-specific options (args) file to be substitued in .cfg
-options_file=options/`hostname`.args
-
-${stetl_cmd} -c last.cfg -a ${options_file}
-
+sudo docker run --link postgis2:postgis2 -v `pwd`:`pwd` -w `pwd`  -t -i geonovum/stetl -c last.cfg -a options/docker.args
diff --git a/etl/options/README.txt b/etl/options/README.txt
@@ -1,4 +1 @@
-This dir contains host-specific options in order to have configs
-locally. To add your host make a file called <host>.args
-This file is used by Stetl (see ../*.sh) to substitute variables in a Stetl .cfg file.
-
+This dir contains host-specific options in order to have configs locally.
diff --git a/etl/options/sunda.args → etl/options/docker.args b/etl/options/sunda.args → etl/options/docker.args
@@ -2,11 +2,12 @@
 
 raw_device_url=http://whale.citygis.nl/sensors/v1
 
-pg_host=localhost
-pg_user=postgres
-pg_password=postgres
+# pg_host=172.17.0.1
+pg_host=postgis2
+pg_user=docker
+pg_password=docker
 pg_port=5432
-pg_database=sensors
+pg_database=gis
 pg_schema=smartem
 pg_schema_rt=smartem_rt
 

diff --git a/etl/python/rawsensorlastinput.py → etl/py/rawsensorlastinput.py b/etl/python/rawsensorlastinput.py → etl/py/rawsensorlastinput.py
diff --git a/etl/python/sensorconverters.py → etl/py/sensorconverters.py b/etl/python/sensorconverters.py → etl/py/sensorconverters.py