Skip to content

KurdishBLARK/InterdialectCorpus

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Kurdish Parallel Corpus

A parallel corpus of Kurdish (Sorani and Kurmanji) and English

This repository contains a parallel corpus of Sorani (ckb) and Kurmanji (kmr) dialects of Kurdish along with English (eng). The development of the corpus is described in our paper. Our approach is consisted of retrieving potentially-alignable news articles from multilingual websites, semi-automatically align sentences across dialects and languages based on lexical similarity and transliteration of scripts and, manually annotate correct translation pairs. For further information, please see the annotation guidelines.

Our parallel corpus contains three manually-aligned corpus in Sorani-Kurmanji, Sorani-English and Kurmanji-English in various formats, namely Translation Memory eXchange file format (.tmx), parallel annotated text useful for ParaConc and raw parallel texts (.txt). In the latter, each line corresponds to the same line in the other aligned file.

This corpus contains 12,327 translation pairs in the two major dialects of Kurdish, Sorani and Kurmanji. We also provide 1,797 and 650 translation pairs in English-Kurmanji and English-Sorani.

Download

You can clone this repository or download individual directories as follows:

In addition, the statistical models reported as the baseline in our paper are provided in the Moses-results directory.

Cite this paper

If you use any part of the data, please consider citing this paper as follows:

@misc{ahmadi2020leveraging,
      title={Leveraging Multilingual News Websites for Building a Kurdish Parallel Corpus}, 
      author={Sina Ahmadi and Hossein Hassani and Daban Q. Jaff},
      year={2020},
      eprint={2010.01554},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

The link to the published version of the paper will be also provided later.

License

This corpus is available under the Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license. For further information for commercial use, please get in touch with the contributors.