Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.
Python Other
Permalink
Failed to load latest commit information.
bin fix import error: cannot import name print_doc Dec 8, 2014
docker #45 : change all links to old repo to new https://github.com/geopytho… Jul 13, 2016
docs #35 complete Chain Splitting also supporting sub-Chains Dec 14, 2016
examples update READMEs to reflect https://github.com/nlextract/NLExtract Jan 27, 2017
stetl Travis: execute nose2 tests in script and fix typo in dbinput.py Dec 17, 2016
tests XmlAssembler unit test improvements Sep 3, 2016
.coveragerc Added .coveragerc file to exclude tests directory while determining t… Aug 5, 2016
.gitignore Added .gitignore; updated tests/README.txt Jul 11, 2016
.travis.yml Travis: execute nose2 tests in script and fix typo in dbinput.py Dec 17, 2016
CHANGES.txt #45 : change all links to old repo to new https://github.com/geopytho… Jul 13, 2016
CONTRIBUTING.md enhance README and add CONTRIBUTING Feb 22, 2016
CREDITS.txt #45 : change all links to old repo to new https://github.com/geopytho… Jul 13, 2016
LICENSE.txt setup.py working now Jun 9, 2013
MANIFEST.in issue #36 first stab at Docker image for Stetl Jun 17, 2016
README.md update READMEs to reflect https://github.com/nlextract/NLExtract Jan 27, 2017
STETL.iml #35 provide code + example + doc for Output Splitting Dec 13, 2016
clean.sh more cleining clean.sh Sep 1, 2014
refresh-git.sh add git refresh cmdline util Oct 22, 2014
requirements-dev.txt Travis: remove GDAL Python pip and use Ubuntu python-gdal instead sigh Dec 17, 2016
setup.cfg refactoring directories to Python spirit, start adding tests Jun 3, 2013
setup.py Travis error resolve Dec 17, 2016

README.md

Stetl - Streaming ETL

Stetl, streaming ETL, pronounced "staedl", is a lightweight ETL-framework for geospatial data conversion.

Build Status Documentation Status Gitter Chat

Notice: the Stetl GH repo is now at the GeoPython GH organization.

License

Stetl is released under a GNU GPL v3 license (see LICENSE.txt).

Documentation

The Stetl website and documentation can be found via http://stetl.org. For a quick overview read the 5-minute Stetl-introduction, or a more detailed presentation. Stetl was presented at several events like the FOSS4G 2013 in Nottingham and GeoPython 2016.

Concepts

Stetl basically glues together existing parsing and transformation tools like GDAL/OGR, Jinja2 and XSLT with custom Python code. By using native libraries like libxml2 and libxslt (via Python lxml) Stetl is speed-optimized.

A configuration file, in Python config .ini format, specifies a chained sequence of transformation steps: typically an Input connected to one or more Filters, and finally to an Output. At runtime, this sequence is instantiated and run as a linked series of Python objects. These objects are symbolically specified (by their module/class name) and parameterized in the config file. Via the stetl -c <config file> command, the transformation is executed.

Stetl has been proven to handle 10's of millions of GML objects without any memory issues. This is achieved through a technique called "streaming and splitting". For example: using the OgrPostgisInput module an GML stream can be generated from the database. A component called the GmlSplitter can split this stream into manageable chunks (like 20000 features) and feed this upstream into the ETL chain.

Use Cases

Stetl has been found particularly useful for complex GML-related ETL-cases, like those found within EU INSPIRE Data Harmonization and the transformation of GML/XML-based National geo-datasets to for example PostGIS.

Most of the data conversions within the Dutch NLExtract Project apply Stetl.

Stetl also proved to be very effective in IoT-related transformations involving the SensorWeb/SOS.

Examples

Browse all examples under the examples dir. Best is to start with the basic examples

Installation

Stetl can be installed via PyPi pip install stetl and recently as a Stetl Docker image. More on installation in the documentation.

Contributing

Anyone and everyone is welcome to contribute. Please take a moment to review the guidelines for contributing.

Origins

Stetl originated in the INSPIRE-FOSS project: 2009-2013 now archived. Since then Stetl evolved into a wider use like transforming Dutch GML-based Open Datasets such as IMGEO/BGT (Large Scale Topography) and IMKAD/BRK (Cadastral Data).

Finally

The word "stetl" is also an alternative writing for "shtetl": http://en.wikipedia.org/wiki/Stetl : "...Material things were neither disdained nor extremely praised in the shtetl. Learning and education were the ultimate measures of worth in the eyes of the community, while money was secondary to status..."