GitHub - zwhe99/corpus-tools: Some useful scripts to process monolingual or parallel texts.

Corpus Clean

Some useful scripts to process monolingual or parallel texts.

Name		Name	Last commit message	Last commit date
Latest commit History 17 Commits
.gitignore		.gitignore
README.md		README.md
add_tag.py		add_tag.py
clean_histogram.py		clean_histogram.py
filter_lang.py		filter_lang.py
filter_lang_mono.py		filter_lang_mono.py
identity.py		identity.py
kenlm_normalized_score.py		kenlm_normalized_score.py
norm_unicode.py		norm_unicode.py
paste.py		paste.py
pivot_mine.py		pivot_mine.py
pivot_mine2.py		pivot_mine2.py
remove_duplicate.py		remove_duplicate.py
remove_duplicate_mono.py		remove_duplicate_mono.py
remove_from.py		remove_from.py
remove_from_mono.py		remove_from_mono.py
remove_identical.py		remove_identical.py
remove_invalid_char_from_tmx.py		remove_invalid_char_from_tmx.py
remove_noisy_pair.py		remove_noisy_pair.py
remove_specified_line.py		remove_specified_line.py
remove_too_much_punc.py		remove_too_much_punc.py
remove_too_much_punc_mono.py		remove_too_much_punc_mono.py
remove_url.py		remove_url.py
remove_url_mono.py		remove_url_mono.py
replace_repeat.py		replace_repeat.py
tag_unicode_norm.py		tag_unicode_norm.py
tmx2txt.py		tmx2txt.py
tmx2txt.sh		tmx2txt.sh
word2freq.py		word2freq.py