#etlTest
##Installation
You can install etlTest by downloading the source and using the setup.py script as follows:
$ git clone git@github.com:OpenDataAlex/etlTest.git
$ cd etlTest
$ python setup.py install
This setup call installs all of the necessary python dependencies. There are a few external dependencies as well, so please see the section below labeled "Non-Python Dependencies".
Once you have done that, its ready to run!
Having come from software development and working with data integration tools, we always wondered why there wasn't some kind of uniform unit and integration testing tool specific to data integration. etlTest aims to fill that gap by providing an easy to use tool and data source agnostic testing tool. Testing is designed to be "black box" - which means that we aren't diving into the actual data integration code. Rather, we are executing the data integration process based on test data sets provided by the test writer and comparing the results using Python's unittest framework.
etlTest is based on the work and discussions that were started with etlUnit.
To actually use etlTest, you need a data and test file for it to act on. A most basic resource file can be found in the samples directory of the project (data/etlUnitTest/users.yml and test/dataMart/users_dim.yml). Executing the following will take that resource, generate some python code in the output directory specified, and run the code which will display the output of the tests executed to your terminal.
$ python etlTest/etltest/etlTest.py -f <path to users_dim.yml test file> -o /tmp/ -g -e
The documentation for etlTest can be found on Read the Docs here.
The only dependencies that are not handled in python currently are the ones for SQLAlchemy to connect to datasources. Documentation on how to install these is as follows:
We would love some feedback! Please do not hesitate to report any issues/questions/comments via the Github Issue Tracker.