A near-duplicate detection pipeline for a multilingual news-article text corpus (Kurmanji Kurdish kmr_Latn, Sorani Kurdish ckb_Arab, Zazaki diq_Latn), built on top of datatrove's MinHash implementation. The corpus itself (CSV files with url/text/lang/publisher columns) lives outside this repo under data/.
pipenv install
Run the near-dedup pipeline:
pipenv run python -m dedup.dedup --input data/test_sample.csv --output-dir data/dedep_test
Build a small, outlet-diverse test sample from a full corpus dump:
pipenv run python dedup/sample_test_data.py --input data/output.csv --output data/test_sample.csv
CSV data-quality checks:
pipenv run python dedup/check_max_field_length.py data/output.csv
pipenv run python dedup/check_row_shape.py data/output.csv
Re-run language ID (the same GlotLID model used at scrape time in the sibling sorjin_scrapy project) against a deduped/ output, overwriting lang/lang_score in place:
pipenv run python -m dedup.relabel_lang data/dedep_test/deduped
Push a deduped/ output to the Hugging Face Hub, one language config at a time, via datasets.Dataset.push_to_hub() (requires pipenv run hf auth login first). This reads deduped/ directly and lets push_to_hub() build+upload its own Parquet shards, so it never stages a full extra copy of the corpus on local disk. The dataset card itself is written by hand on the Hub, not generated by a script:
pipenv run python -m dedup.upload_to_hf --repo-id <your-username>/<dataset-name> --input data/dedep_test/deduped
Run tests:
pipenv run pytest