Home

ETLLib

ETLLib is a command-line toolkit and Python library for munging JSON, TSV, and related data — using Apache Tika where field cleanup helps — and posting the result to Apache Solr.

It is not a workflow engine. You run the commands from a shell, import etl from Python, or call them from a Mnemosyne pipeline. BigTranslate does the last of those.

tsvtojson → repackage → poster
     TSV        split      Solr

Source and issues: https://github.com/chrismattmann/etllib

Website: https://chrismattmann.github.io/etllib/

Getting started

1. Getting Started — Python 3.9–3.13, libmagic, install
2. Tutorial — CSV/TSV to one JSON file per row
3. Commands — the six CLIs

Working deployments: BigTranslate.

Commands

Command	What it does
tsvtojson	TSV + column headers → aggregate JSON
repackage	Split an aggregate JSON into one file per record
poster	POST JSON documents to Solr
repackageandpost	Split and POST without writing the intermediate files
translatejson	Translate named JSON fields (Tika)
similarity	Jaccard similarity / clusters over a directory (Tika)

The same steps are functions in etl.etllib. Details: Commands.

Heritage

ETLLib started as Python scripts on DARPA XDATA corpora (Kiva JSON dumps, Computrabajo employment TSVs) and was wrapped by Apache OODT workflows. Those workflows now run on Mnemosyne. OODT-era install notes (buildout, Python 2.7, bootstrap.py) live under Old.