Skip to content

Auto-generated stopwords for South African Bantu Languages

Notifications You must be signed in to change notification settings

ndamulelonemakh/our-stopwords

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 

Repository files navigation

Our Stopwords

  • Auto-generated stopwords for South African Languages

Introduction

Usage

  • The data is provided in JSON Lines format. Here is an example of using the stopwords in Python:

Python Example

import json

# Load the stop words from the JSON lines file
stop_words = []
with open('za_stopwords.main.jsonl', 'r', encoding='utf-8') as file:
    for line in file:
        stop_words.append(json.loads(line.strip()))

# Example: Print stop words in Zulu
for word in stop_words:
    print(f"English: {word['eng']}, Zulu: {word['zul']}")

Refer to training_example.py for a full working example on how you can use these stopwords in your model training scripts

Supported Languages

  • ven - Tshivenda
  • tso - Xitsonga
  • sot - Southern Sotho
  • nso - Northen Sotho
  • tsn - Setswana
  • zul - IsiZulu
  • xho - IsiXhosa
  • ...
  • Coming soon:
    • nbl
    • ssw

Contributions

  • Feel free to create a pull request if you have a suggestion

License

  • This work is published under the CC BY-NC 4.0 license

Citation

@misc{multilingual_stop_words,
    author = {Ndamulelo Nemakhavhani},
    title = {Autogenerated Stop Words for South African Bantu Languages},
    year = {2024},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/ndamulelonemakh/our-stopwords}}
}

Contact

  • If you have any questions or suggestions, please feel free to open an issue or contact us.