Metapack, Metatab Data Packaging
Parse and manipulate structured data and metadata in a tabular format.
Metatab is a data format that allows structured metadata -- the sort you'd normally store in JSON, YAML or XML -- to be stored and edited in tabular forms like CSV or Excel. Metatab files look exactly like you'd expect, so they are very easy for non-technical users to read and edit, using tools they already have. Metatab is an excellent format for creating, storing and transmitting metadata. For more information about metatab, visit http://metatab.org.
Metapack is a packaging system that uses Metatab to create Zip, Excel and filesystem data packages.
Metapack only works with Python 3.5 or later, and you'll almost certainly want to install it into a virtual environment. To set up a virtual environment:
python3 -mvenv metapack cd metapack source bin/activate
Since we're stil in development, you'll get the latest code by installing package from github, but you can also install from pip. In either case, you should create the virtualenv, and afterward, you'll have to reinstall the six package because of an odd conflict
To install the package with pip:
pip install metapack
Because the fs package has an odd version requirement on six, you'll have to fix the version:
pip uninstall -y six pip install six==1.10.0
To run the tests, you'll also need to install some support modules;
$ pip install fiona shapely pyproj terminaltables geopandas
Then test parsing using a remote file with the
metatab program, from the
$ metatab -j https://raw.githubusercontent.com/CivicKnowledge/metatab-py/master/test-data/example1.csv
metatab -h to get other program options.
test-data directory has test files that also serve as examples to
parse. You can either clone the repo and parse them from the files, or from the
Github page for the file, click on the
raw button to get raw view of the
flie, then copy the URL.
The main program for metapack is mt, which has a number of ( extensible) sub
commands. See the commands with:
Clearing the Cache
Some tests can pass despite errors if the file the test is looking for is cached. The cache can be set with an evironmental variable and cleared before the tests to solve this problem
$ cache_dir=/tmp/some/dir $ rm -rf $cache_dir $ mkdir -p $cache_dir $ APPURL_CACHE=$cache_dir python setup.py test
Development Testing with Docker
Testing during development for other versions of Python is a bit of a pain, since you have to install the alternate version, and Tox will run all of the tests, not just the one you want.
One way to deal with this is to install Docker locally, then run the docker test container on the source directory. This is done automatically from the Makefile in appurl/tests
$ cd metapack/metapack/test $ make build # to create the container image $ make shell # to run bash the container
You now have a docker container where the /code directory is the appurl source dir. Since the Docker container is running code from your host machine, you can edit it normally.
Now, run tox to build the tox virtual environments, then enter the specific version you want to run tests for and activate the virtual environment.
To run one environment. for example, Python 3.4
# tox -e py34
To run one test in one environment environment. for example, Python 3.4
# tox -e py34 -- -s