Word Corpus

Scripts that extract a word corpus from OpenStreetMap, Wikipedia, and Wikidata targeting South-East Asian and Indic languages.

Data License

• OpenStreetMap-derived data is licensed under the Open Data Commons Open Database License (ODbL). See https://www.openstreetmap.org/copyright
• Wikipedia-derived data is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA). See https://en.wikipedia.org/wiki/Wikipedia:Copyrights
• Wikidata-derived data is licensed under the Creative Commons CC0 License. See https://www.wikidata.org/wiki/Wikidata:Licensing

Downloads

Download corpus with duplicates. These files contain only non-Latin/Greek/Cyrillic/CJK text as defined in the file latin_greek_cyrillic_cjk.py.

• osm-corpus-with-duplicates.txt.zip (19M)
• wikipedia-corpus-with-duplicates.txt.zip (816M)
• wikidata-corpus-with-duplicates.txt.zip (118M)

Single Scripts

Download corpus for a single script without duplicates:

• Arabic
  • osm-arabic-corpus.txt.zip (2M)
  • wikipedia-arabic-corpus.txt.zip (80M)
  • wikidata-arabic-corpus.txt.zip (10M)
• Bengali
  • osm-bengali-corpus.txt.zip (79K)
  • wikipedia-bengali-corpus.txt.zip (22M)
  • wikidata-bengali-corpus.txt.zip (10M)
• Devanagari
  • osm-devanagari-corpus.txt.zip (197K)
  • wikipedia-devanagari-corpus.txt.zip (94M)
  • wikidata-devanagari-corpus.txt.zip (1.2M)
• Gujarati
  • osm-gujarati-corpus.txt.zip (11K)
  • wikipedia-gujarati-corpus.txt.zip (3.4M)
  • wikidata-gujarati-corpus.txt.zip (228K)
• Gurmukhi
  • osm-gurmukhi-corpus.txt.zip (11K)
  • wikipedia-gurmukhi-corpus.txt.zip (3.0M)
  • wikidata-gurmukhi-corpus.txt.zip (211K)
• Kannada
  • osm-kannada-corpus.txt.zip (119K)
  • wikipedia-kannada-corpus.txt.zip (1.7M)
  • wikidata-kannada-corpus.txt.zip (293K)
• Khmer
  • osm-khmer-corpus.txt.zip (73K)
  • wikipedia-khmer-corpus.txt.zip (12M)
  • wikidata-khmer-corpus.txt.zip (137K)
• Malayalam
  • osm-malayalam-corpus.txt.zip (76K)
  • wikipedia-malayalam-corpus.txt.zip (24M)
  • wikidata-malayalam-corpus.txt.zip (818K)
• Myanmar
  • osm-myanmar-corpus.txt.zip (206K)
  • wikipedia-myanmar-corpus.txt.zip (28M)
  • wikidata-myanmar-corpus.txt.zip (594K)
• Oriya
  • osm-oriya-corpus.txt.zip (7.5K)
  • wikipedia-oriya-corpus.txt.zip (2.2M)
  • wikidata-oriya-corpus.txt.zip (167K)
• Tamil
  • osm-tamil-corpus.txt.zip (53K)
  • wikipedia-tamil-corpus.txt.zip (27M)
  • wikidata-tamil-corpus.txt.zip (688K)
• Telugu
  • osm-telugu-corpus.txt.zip (25K)
  • wikipedia-telugu-corpus.txt.zip (27M)
  • wikidata-telugu-corpus.txt.zip (461K)

If you need some other language or script, please open an Issue on GitHub...

Steps

Download data sources:

cd osm/
python3 download.py
cd ../wikidata/
python3 download.py
cd ../wikipedia/
python3 download.py

Extract non-Latin/Greek/Cyrillic/CJK text from sources with:

python3 extract.py

Generate word corpus with duplicates with:

python3 generate_corpus.py

Filter the corpus for a single script:

python3 filter_by_script.py