Skip to content
hULMonA (حلمنا)​: tHe first Universal Language MOdel iN Arabic
Jupyter Notebook Python
Branch: master
Clone or download
Pull request Compare This branch is even with ObeidaElJundi:master.
Latest commit 2744f37 Aug 4, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data Created README.md for data directory Jul 31, 2019
models adding README.md for models directory Jul 31, 2019
resources
.gitignore Initial commit Jul 23, 2019
README.md Update README.md Aug 4, 2019
arabic-wiki-stuff.ipynb adding code, datasets, and other resources Jul 31, 2019
build_arabic_language_model.ipynb
fine_tune_LM.ipynb

README.md

hULMonA: tHe first Universal Language MOdel iN Arabic

Paper: https://www.aclweb.org/anthology/W19-4608

Introduction

Recent state-of-the-art models in NLP (e.g., BERT, GPT, ULMFiT) utilize transfer learning by pre-training a language model on large curpos and then fine-tuning it on any downstream task. We developed the first Arabic specific universal language model, hULMonA, that can be fine-tuned for almost any Arabic text classification task. We evaluated hULMonA on Sentiment Analysis and achieved state-of-the-art on 4 Arabic datasets. hULMonA consists of three main stages:

1. General domain hULMonA pretraining

To capture the various properties of the Arabic language, we train the SOTA-ish language model AWD-LSTM on all Arabic Wikipedia.

This step is time consuming, but it should be done only once. We publish our pre-trained model, and it is availabe in models directory. To check the implementation details, or to pre-train your own LM, check build_arabic_language_model.ipynb

2. Target task hULMonA fine-tuning

The target task data (e.g., Twitter) will likely come from a different distribution than the general-domain data (Wikipedia). Therefore, fine-tuning the pretrained general-domain LM on the target task data is necessary for the LM to adapt to the new textual properties (e.g., dialects).

To fine-tune the pre-trained hULMonA on your own dataset, please check fine_tune_LM.ipynb

3. Target task classification

Finally, for downstream task classification, we augment the fine-tuned hULMonA with two fully connected layers with ReLU and Softmax activations respectively. Implementatoin details can be found here: fine_tune_LM.ipynb

How do I cite hULMonA?

Please cite this paper:

@inproceedings{eljundi2019hulmona,
  title={hULMonA: The Universal Language Model in Arabic},
  author={ElJundi, Obeida and Antoun, Wissam and El Droubi, Nour and Hajj, Hazem and El-Hajj, Wassim and Shaban, Khaled},
  booktitle={Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  pages={68--77},
  year={2019}
}

Contact information

For help, issues, or personal communication related to using hULMonA, please contact Obeida ElJundi (oae15@mail.aub.edu), Wissam Antoun (wfa07@mail.aub.edu), or Nour El Droubi (ngd02@mail.aub.edu).

You can’t perform that action at this time.