PR #86 from borrob/pr_python3 - THE SWITCH Py2 to Py3

Squash all commits for the python 3 development. This is the definitive switch from Python 2 to 3, no way back!

.travis.yml

@@ -1,13 +1,19 @@
 language: python
+
 sudo: true
 
+dist: xenial
+
 env:
   global:
    - CPLUS_INCLUDE_PATH=/usr/include/gdal
    - C_INCLUDE_PATH=/usr/include/gdal
 
 python:
-  - "2.7"
+  - "3.4"
+  - "3.5"
+  - "3.6"
+  - "3.7"
 
 before_install:
   - sudo apt-add-repository ppa:ubuntugis/ubuntugis-unstable -y

CONTRIBUTING.md

@@ -90,6 +90,8 @@ accurate comments, etc.) and any other requirements (such as test coverage).
 You can run the `nose` and `flake8` tools to check your code with respect to
 unit tests and coding style.
 
+### Getting started
+
 Follow this process if you'd like your work considered for inclusion in the
 project:
 
@@ -105,13 +107,19 @@ project:
    git remote add upstream https://github.com/<upstream-owner>/<repo-name>
    ```
 
+2. make a `virtualenv`  e.g. for Python 3.4.2 (e.g. `pyenv` on MacOSX Homebrew)
+
+2. install dependencies, GDAL may be tricky, usually `pip install gdal==<version>` with v from `gdalinfo --version`
+
 2. If you cloned a while ago, get the latest changes from upstream:
 
    ```bash
    git checkout <dev-branch>
    git pull upstream <dev-branch>
    ```
 
+3. Run ``python setup.py install`` to install the ``stetl`` command and Stetl code in the ``site-packages`` for ``venv``.
+
 3. Create a new topic branch (off the main project development branch) to
    contain your feature, change, or fix:
 
@@ -131,6 +139,12 @@ project:
    git pull [--rebase] upstream <dev-branch>
    ```
 
+5. Run ``flake8`` from the root directory to verify the syntax and coding standards.
+
+5. Run ``nose2`` from the root direcotry to verify all tests are ok.
+
+5. Check the basic examples still work: ``cd examples/basics; ./runall.sh > runall.log 2>&1``. Inspect ``runall.log`` for errors or strange outputs
+
 6. Push your topic branch up to your fork:
 
    ```bash
@@ -143,6 +157,14 @@ project:
 **IMPORTANT**: By submitting a patch, you agree to allow the project owner to
 license your work under the same license as that used by the project.
 
+### Docker
+
+To verify the Docker image build:
+
+*  build Docker Image: `docker build -t geopython/stetl:2.0 .`
+* `cd examples/basics; ./runall-docker.sh > runall-docker.log 2>&1` - run all basic examples from Docker Image
+* inspect `runall-docker.log` for errors or strange outputs
+
 ## Thanks
 
 This doc copied and adapted from original at:

Dockerfile

@@ -1,4 +1,4 @@
-FROM debian:stretch-slim
+FROM python:3.6-slim-stretch
 
 LABEL maintainer="Just van den Broecke <justb4@gmail.com>"
 
@@ -13,17 +13,16 @@ ARG ADD_PYTHON_DEB_PACKAGES=""
 # ARG ADD_PYTHON_PIP_PACKAGES="scikit-learn==0.18"
 ARG ADD_PYTHON_PIP_PACKAGES=""
 
+# Tricky: must match installed GDAL version (2.1.2 on Stretch)
+ARG GDAL_PYTHON_BINDINGS_VERSION="2.1.3"
+
 #
 # ENV settings
 #
 ENV TZ=${TIMEZONE} \
-   LC_ALL="${LOCALE}" \
-   LC_TYPE="${LOCALE}" \
-   LANG="${LOCALE}" \
-   LANGUAGE="${LOCALE}" \
    DEBIAN_FRONTEND="noninteractive" \
-   BUILD_DEPS="tzdata locales" \
-   PYTHON_CORE_PACKAGES="cython python-requests python-tz python-numpy python-pandas python-setuptools python-pip python-lxml python-gdal python-psycopg2 python-jinja2 gdal-bin" \
+   BUILD_DEPS="tzdata build-essential apt-utils libgdal-dev" \
+   PYTHON_CORE_PACKAGES="locales python3-requests python3-tz python3-numpy python3-pandas python3-setuptools python3-pip python3-lxml python3-psycopg2 python3-jinja2 gdal-bin" \
    PYTHON_EXTRA_DEB_PACKAGES="${ADD_PYTHON_DEB_PACKAGES}"  \
    PYTHON_EXTRA_PIP_PACKAGES="${ADD_PYTHON_PIP_PACKAGES}"
 
@@ -37,36 +36,35 @@ RUN \
 		${BUILD_DEPS} \
 	    ${PYTHON_CORE_PACKAGES} \
 	    ${PYTHON_EXTRA_DEB_PACKAGES} \
-
 	# Timezone
-	# echo "${TZ}" > /etc/timezone && \
 	&& cp /usr/share/zoneinfo/${TZ} /etc/localtime\
 	&& dpkg-reconfigure tzdata \
-
 	# Locale
-	&& echo "LANG=${LANG}" >/etc/default/locale  \
-	&& echo "${LANG} UTF-8" > /etc/locale.gen \
-    && locale-gen \
-    && dpkg-reconfigure locales \
-    && /usr/sbin/update-locale LANG=${LANG} \
-
+	&& sed -i -e "s/# ${LOCALE} UTF-8/${LOCALE} UTF-8/" /etc/locale.gen \
+    && dpkg-reconfigure --frontend=noninteractive locales \
+    && update-locale LANG=${LOCALE} \
+    # Install GDAL version matching installed binary - MESSY - need cleaner solution!
+    # && pip install GDAL==`gdalinfo --version | cut -d' ' -f2 | cut -d',' -f1` \
+    && export CPLUS_INCLUDE_PATH=/usr/include/gdal \
+    && export C_INCLUDE_PATH=/usr/include/gdal \
+    && pip install GDAL==${GDAL_PYTHON_BINDINGS_VERSION} \
     # Optional packages to install via Pip
 	&& if [ "x${PYTHON_EXTRA_PIP_PACKAGES}" = "x" ] ;\
 	    then \
 	        echo "No extra Pip packages to install" ;\
 	    else \
 	        pip install ${PYTHON_EXTRA_PIP_PACKAGES} ;\
 	    fi  \
-
 	# Install and Remove build-related packages for smaller image size
 	&& cd /stetl \
 		&& python setup.py install  \
+		&& apt-get remove --purge ${BUILD_DEPS} -y \
 		&& apt autoremove -y  \
-        && rm -rf /var/lib/apt/lists/* \
-        
-	&& echo "For ${TZ} date=`date`" \
-	&& echo "locale=`locale`"
+        && rm -rf /var/lib/apt/lists/* 
+
+ENV LANG="${LOCALE}" LANGUAGE="${LOCALE}"
+RUN echo "For ${TZ} date=`date`" && echo "Locale=`locale`"
 
-# Allow docker run
-# docker run --rm -it geopython/stetl  stetl
-# docker run --rm -it geopython/stetl  bash
+# Run examples
+# docker run --rm -it geopython/stetl stetl
+# docker run --rm -v $(pwd):/work -w /work geopython/stetl:2.0 stetl -c etl.cfg

bin/stetl

@@ -1,13 +1,12 @@
 #!/usr/bin/env python
-# -*- coding: utf-8 -*-
-#
 # Main Stetl program.
 #
 # Author: Just van den Broecke
 #
 from stetl.main import parse_args
 from stetl.etl import ETL
 from stetl.util import Util
+from stetl.version import __version__
 import sys
 
 log = Util.get_log('main')
@@ -20,19 +19,22 @@ def main():
        -c  --config <config_file>  the Stetl config file.
        -s  --section <section_name> the section in the Stetl config (ini) file to execute (default is [etl]).
        -a  --args <arglist> zero or more substitutable args for symbolic, {arg}, values in Stetl config file, in format -a arg1=foo -a arg2=bar etc.
+       -v  --version Show the current version of stelt and exit
        -h  --help <subject> Get component documentation like its configuration parameters, e.g. stetl doc stetl.inputs.fileinput.FileInput
 
     """
 
     args = parse_args(sys.argv[1:])
 
+    if args.version:
+        print('Stetl version: ', __version__)
+        exit()
+
     if args.config_file:
         # Do the ETL
         etl = ETL(vars(args), args.config_args)
         etl.run()
 
-    # elif args.doc_args:
-    #     print_doc(args.doc_args)
     else:
         print('Try stetl -h for help')
 

docs/index.rst

@@ -26,6 +26,7 @@ Contents:
 
    intro.rst
    install.rst
+   py3upgrade.rst
    background.rst
    using.rst
    cases.rst

docs/install.rst

@@ -5,6 +5,7 @@ Installation
 
 Stetl up to and including version 1.3 only runs with Python 2 (2.7+).
 Starting with Stetl v2.0 only Python 3 (3.4.2+) will be supported.
+You may want to read :ref:`py3upgrade` when upgrading from a Stetl pre-v2 version.
 
 Easiest is to first install the Stetl-dependencies (see below) and then
 install and maintain Stetl on your system as a Python package (`pip` is preferred). ::
@@ -50,6 +51,7 @@ Stetl depends on the following Python packages:
 * psycopg2 (PostgreSQL client)
 * lxml
 * Jinja2 templating
+* Deprecated
 
 ``GDAL`` Python binding requires the native GDAL/OGR libs and tools (version 2+) to be installed.
 
@@ -60,7 +62,9 @@ Stetl depends on the following Python packages:
 
 When using the ``Jinja2`` templating filter, ``Jinja2TemplatingFilter``, see http://jinja.pocoo.org:
 
-* Python Jinja2 package
+* Python ``Jinja2`` package
+
+``Deprecated`` is used to indicated deprecated functions and classes.
 
 Platform-specific guidelines for dependencies follow next.
 
@@ -76,17 +80,17 @@ choose to install the same packages via `pip` to have more recent versions like
 
 - Python dependencies: ::
 
-	apt-get install python-setuptools
-	apt-get install python-dev
-	apt-get install python-pip
+	apt-get install python3-setuptools
+	apt-get install python3-dev
+	apt-get install python3-pip
 	pip install --upgrade pip
 	
 - ``libxml2/libxslt`` libs are usually already installed. Together with Python ``lxml``, the total install for ``lxml`` is: ::
 
-	apt-get install python-libxml2
+	apt-get install python3-libxml2
 	apt-get install python-libxslt1
 	apt-get install libxml2-dev libxslt1-dev lib32z1-dev
-	apt-get install python-lxml
+	apt-get install python3-lxml
 
 - ``GDAL`` (http://gdal.org) version 2+ with Python bindings: ::
 
@@ -95,17 +99,17 @@ choose to install the same packages via `pip` to have more recent versions like
 	apt-get update
 	apt-get install gdal-bin
 	gdalinfo --version
-	# should show something like: GDAL 2.2.1, released 2017/06/23
+	# should show something like: GDAL 2.4.0, released 2019/03/04
 
 	apt-get install python-gdal
 
 - the PostgreSQL client library for Python ``psycopg2``: ::
 
-	apt-get install python-psycopg2
+	apt-get install python3-psycopg2
 
 - for ``Jinja2``: ::
 
-	apt-get install python-jinja2
+	apt-get install python3-jinja2
 
 Mac OSX
 ~~~~~~~
@@ -153,7 +157,7 @@ You should get meaningful output like ::
 	2013-09-16 18:25:12,122 main INFO Stetl version = 1.0.3
 	usage: stetl [-h] -c  CONFIG_FILE [-s  CONFIG_SECTION] [-a  CONFIG_ARGS]
 
-Especially check the Stetl version number.
+Especially check the Stetl version number. You can also use the `-v` or `--version` option for stetl.
 
 Try running the examples when running with a downloaded distro. ::
 

docs/py3upgrade.rst

@@ -0,0 +1,51 @@
+.. _py3upgrade:
+
+Upgrade to Python 3
+===================
+
+Stetl development started in Python 2. With `PEP 373
+<https://legacy.python.org/dev/peps/pep-0373/>`_ the EOL of python 2.7 was announced and python 2
+will not be officialy supported after 2020. Stetl was therefore upgraded to Python 3.
+
+Python 3
+--------
+
+Work started early 2019 to upgrade ``Stetl`` from Python 2 to Python 3. The last version of Stetl
+that supports Python 2 is version 1.3. This version *might* receive quick fixes and updates, but
+users are encouraged to upgrade to Stetl version 2 or higher and thus use Python 3.
+
+For the full discussion on the Python 2 to Python 3 migration: see the `conversation in pull
+request #81 <https://github.com/geopython/stetl/pull/81>`_ within the GitHub repository.
+
+Important changes for developers
+--------------------------------
+
+Python 2 and 3 are very similar, but there are a couple of important changes that developers need
+to keep in mind and are worth mentioning:
+
+- Stetl 2 supports Python 3.4.2 and higher (so unfortunately no `f strings <https://www.python.org/dev/peps/pep-0498/>`_)
+- Python 3 uses Unicode strings, meaning encoding/decoding is a bit different
+- ``stringIO`` and ``cstringIO`` were moved around
+- slight syntax change on calling ``next()`` for iterators
+- update on ``import`` statements
+- differences in ``urllib`` to make http-calls (although `issue 80 <https://github.com/geopython/stetl/issues/80>`_ might change it to the `requests` library).
+
+Important changes for users
+---------------------------
+
+The specification of the Stetl tool chain uses a configuration file. You can use the Inputs, Filters, and
+Outputs that are provided by Stetl, or write your own. If you use  Stetl Components in your configuration, you *must*
+specify the ``stetl.`` package prefix in the class specification. For example before Stetl version 2 the input XML
+file was specified as ::
+
+    [input_xml_file]
+    class = inputs.fileinput.XmlFileInput
+    file_path = input/cities.xml
+
+for Stetl version 2 this is changed to ::
+
+    [input_xml_file]
+    class = stetl.inputs.fileinput.XmlFileInput
+    file_path = input/cities.xml
+
+Note the extra ``stetl.`` part in the ``class`` specification.