Commands

Commands

Six console scripts install on your PATH. Each is a thin wrapper around etl.etllib. Run command -h for the live usage string.

tsvtojson

TSV + column headers → one aggregate JSON file.

tsvtojson -t data.tsv -j out.json -c colheaders.txt -o employmentjobs -e encoding.txt -s 0.8 -v

Flag	Required	What
-t	yes	input TSV
-j	yes	output JSON; refuses to overwrite
-c	yes	column headers, one per line
-o	yes	wrapper key ({"employmentjobs": [ ... ]})
-s	yes	Jaccard threshold 0.0–1.0; higher keeps more
-e	no	encodings to try, one per line
-u	no	header whose value must be unique (later dupes skipped)
-v	no	verbose

Column-header tricks:

title
salary:
companyId*

salary: is skipped if the row has fewer cells than headers. companyId* is stored under both companyId* and id. If no * header is present, a UUID becomes id.

More on this command: TSV to JSON.

repackage

Pull the list under -o out of an aggregate JSON and write one {id}.json per record into the current directory.

cd json
repackage -j ../aggregate.json -o employmentjobs -v

Also: flatten nested structs, format dates, Tika-cleanse HTML body when Tika is importable.

poster

POST existing JSON documents to Solr's JSON update handler.

# stdin: one path per line
find . -name '*.json' | poster -u 'http://localhost:8983/solr/jobs/update/json?commit=true' -v

# or a directory
poster -u 'http://localhost:8983/solr/jobs/update/json?commit=true' -d ./json -v

repackageandpost

repackage then poster without writing {id}.json files. STDIN is a list of aggregate JSON paths.

echo /data/kiva/teams_pg1.json | repackageandpost \
    -u 'http://localhost:8983/solr/teams/update/json?commit=true' \
    -o teams -v

translatejson

Translate named fields in a JSON object from one language to another via tika.translate, with an rlite cache (-r).

translatejson -i in.json -j out.json -c translate.cols -f es -t en -r ./translation.rdb -v

translate.cols is one field name per line. -p / Bing credentials in older docs are gone; Tika does the translation.

This command needs hirlite (pip install -e ".[translate]"). That package often does not build on Python 3.10–3.13. For many-to-English at scale, use BigTranslate (Pantogloss), not this CLI.

similarity

Jaccard similarity over Tika metadata (keys or values) for files in a directory, then cluster. Not the same binary as the old imagesimilarity name in docs/USE.txt — the installed command is similarity.

similarity -f ~/Pictures -o ./similarity-out --threshold 0.01 --maxnode 10

Writes cluster JSON consumed by the D3 pages under html/ in the source tree.

Python library

from etl.etllib import prepareDocs, writeDoc, recoverMisdecoded
from etl.tsvtojson import detectEncoding, near_dedup_jaccard

The CLIs are the supported interface. Import the same functions if you would rather not shell out.