This is an project that investigates transfer learning in low-resourced languages, specifically in a named entity recognition (NER) task. This repository contains the code, the trained models can be found here and the paper (accepted to IJCNLP-AACL 2023) can be found here.
Further, many visualisations are available in ./analysis/. Finally, the raw predictions and results can be found in ./src/runs/v10/models/
- NER Transfer - Interpretation and Analysis
- About
- Contents
- Code Structure
- Get Started
- References / Sources / Libraries
- License
- Citation
- Model Cards
We mainly did the following in this project:
- Fine-tune many different pre-trained models on different languages from the MasakhaNER dataset.
- Using these fine-tuned models, we evaluated on all of the languages, to get an idea for zero-shot potential.
- Then we moved onto analysis, which is the bulk of our contributions and code here. The specific analysis we did can be found in
./src/analysis, with the following subfolders:v10: Analyse Pre-training effect on final performance.v20: Look at zero-shot transfer.v40: Investigate word embeddings, performing, among others, PCA on them and plotting them.v50: Statistically analysing the data overlap between the different languages, and investigating correlation between this and performance.
├── analysis -> This contains the bulk of our results, including plots, csv files and Latex tables.
│ ├── v10
│ ├── v20
│ ├── v40
│ └── v50
├── data
│ └── masakhane-ner
├── doc
│ ├── report.pdf -> Written Report
├── env.yml -> Environment File
├── logs -> Log files are written to here
├── run.sh -> script to run any code in this repository
└── src -> All of the source code
├── analysis -> Analyses the results we obtained
│ ├── utils.py -> General utilities
│ ├── v10 -> Effect of pre-trained models
│ ├── v20 -> Effect of fine-tuning and doing zero shot transfer
│ ├── v40 -> Interpretability, specifically word embedding analysis
│ └── v50 -> Statistical analysis including data overlap
├── results
│ └── v40 -> This contains some pickle files with precomputed embeddings.
├── run.sh -> Another run.sh
├── runs -> Code that ran all the fine-tuning. Also contains the results on the test/dev set.
│ ├── v10 -> Fine-tuning from different pre-trained models.
│ └── v20 -> Evaluating zero shot transfer.
├── slurms -> Job files are saved here.
│ └── v10
└── utils -> Overall utility code, lot of it from the MasakhaNER codebase
├── create_slurms.py -> Creates Slurm job files for the fine-tuning
├── evaluate_ner.py -> Evaluates a specific model on any language
├── train_ner.py -> Trains the models
└── utils_ner.py -> NER Utilities
To get started, you can use our conda environment by running the following command:
conda env create -f env.yml
Then run conda activate mner, and you should be ready to go. If that fails, you can try to instead use env_all.yml, with pinned versions for all packages.
Now, to run any python file in this project, the recommended way to do so is to use the ./run.sh script provided in the root of the project instead of python directly, as the former sets up the requisite paths, e.g. ./run.sh src/utils/create_slurms.py
If you want to train the models, then you can create the Slurm scripts used by running ./run.sh src/utils/create_slurms.py. Note that you will need to change this file's paths at the top of the file to be applicable to your cluster. Then, you should be able to see the slurm scripts in src/slurms/v10 and you can run all of them. Note, there are very many and your cluster scheduler may block you if you try running all of them at once.
Next up is how to do evaluation, i.e. take the models from before, and evaluate them on all of the other languages. To do so you can navigate to src/runs/v20 and run ./run_eval_all.sh. Note, this will run all of the inference (4 in parallel) in one step, so it might take a long time.
In practice, we do not recommend this training and evaluation is done from scratch as it uses significant compute and storage resources. We do provide downloadable models (albeit just one seed instead of 5), so it is recommended to just use those.
The analysis is mostly self contained, and the vast majority requires just the result .txt files present and not the models at all.
TL; DR: You can probably just run the following commands
./run.sh src/analysis/v10/basic_analyse_pretrained_model.py
./run.sh src/analysis/v10/analyse_per_category.py
./run.sh src/analysis/v20/base.py
./run.sh src/analysis/v50/statistical_overlap.py
and most of the analysis should run.
The embeddings code is a bit more involved, as that requires the actual word embeddings, which are quite large, and are stored in a separate repository. To download these, do the following:
mkdir -p src/results/v40
cd src/results/v40
git clone https://github.com/Michael-Beukman/NERTransfer-Embeddings .
cd -
Once those have been downloaded, you can run the code by simply running ./run.sh src/analysis/v40/embeddings.py.
Code is also contained inside there to actually create the word embeddings, although that does require all of the fine-tuned models.
In the repository, there are many images of embeddings from different combinations of models and languages, specifically found in analysis/v40, so the code does not need to be rerun to just view them.
In src/runs/v10/models there are many different folders, consisting of the results (metrics and predictions) from different fine-tuned models. src/runs/v10/models/v20 contains the results from the same models, when evaluating on different languages. There is lots of data here, so feel free to explore it.
Additionally, there are very many plots, images and tables in analysis/* that might be interesting.
The folder names might be slightly confusing, but, e.g. src/runs/v10/models/kin_v10_start_swa__kin__2_50/ means it was fine-tuned on Kinyarwanda NER data, starting from the Swahili pre-trained model, trained using seed = 2 and epochs = 50. Similarly, src/runs/v10/models/ibo_v10_start_base__ibo__0_50 is xlm-roberta-base fine-tuned on Igbo data.
For the results in v20, e.g. src/runs/v10/models/v20/lug_zero_shot_pcm_v10_start_swa__pcm__0_50_lug_50 is fundamentally the pcm_v10_start_swa__pcm__0_50 model (i.e. Swahili pre-trained and fine-tuned on Nigerian Pidgin) that was used to evaluate Luganda. Hence, these files follow the format src/runs/v10/models/v20/<eval>_zero_shot_<model>_<eval>_50.
- MasakhaNER (Github, paper). Their code and data can be found in
data/, and is licensed underMasakhane-LICENSE - Interspersed throughout the code are links to other sources that were used to solve a particular problem.
This code is licensed under the Apache License, Version 2.0.
If you find this useful, then please cite this work as follows:
@inproceedings{NerTransfer,
title={Analysing the effects of transfer learning on low-resourced named entity recognition performance},
author={Michael Beukman},
booktitle={3rd Workshop on African Natural Language Processing},
year={2021},
url={https://openreview.net/forum?id=HKWMFqfN8b5}
}
We release some models that were fine-tuned on the MasakhaNER dataset, using specific languages. We start with 3 different pre-trained models, and then fine-tune on the 10 languages in the dataset. The pre-trained models we start with are:
- xlm-roberta-base
- xlm-roberta-base-finetuned-swahili -> Domain adaptive fine-tuning on a large, unlabelled monolingual corpus.
- xlm-roberta-base-finetuned-X -> Domain adaptive fine-tuning on large, unlabelled monolingual corpora, for each of the languages under consideration.
Here we describe a joint model card for all of the models, as they were quite similar.
These models are transformer based and were all fine-tuned on the MasakhaNER dataset. It is a named entity recognition dataset, containing mostly news articles in 10 different African languages. The models were all fine-tuned for 50 epochs, with a maximum sequence length of 200, 32 batch size, 5e-5 learning rate. Each model was fine-tuned 5 times (with different random seeds), and the ones we upload are the ones that did the best for the language it was evaluated on.
These models were fine-tuned by me, Michael Beukman while doing a project at the University of the Witwatersrand, Johannesburg. This is version 1, as of 20 November 2021. These models are licensed under the Apache License, Version 2.0.
For more information about the models, including training scripts, detailed results and further resources, you can visit the the main Github repository. You can contact me by filing an issue on this repository.
In the interest of openness, and reporting resources used, we list here how long the training process took, as well as what the minimum resources would be to reproduce this. Fine-tuning each model on the NER dataset took between 10 and 30 minutes, and was performed on a NVIDIA RTX3090 GPU. To use a batch size of 32, at least 14GB of GPU memory was required, although it was just possible to fit these models in around 6.5GB's of VRAM when using a batch size of 1.
The train, evaluation and test datasets were taken directly from the MasakhaNER Github repository, with minimal to no preprocessing, as the original dataset is already of high quality. The motivation for the use of this data is that it is the "first large, publicly available, high quality dataset for named entity recognition (NER) in ten African languages" (source). The high-quality data, as well as the groundwork laid by the paper introducing it are some more reasons why this dataset was used. For evaluation, the dedicated test split was used, which is from the same distribution as the training data, so the models may not generalise to other distributions, and further testing would need to be done to investigate this. The exact distribution of the data is covered in detail here.
These models are intended to be used for NLP research into e.g. interpretability or transfer learning. Using these models in production is not supported, as generalisability and downright performance is limited. In particular, these models are not designed to be used in any important downstream task that could affect people, as harm could be caused by the limitations of the model, described next.
These models were only trained on one (relatively small) dataset, covering one task (NER) in one domain (news articles) and in a set span of time. The results may not generalise, and the model may perform badly, or in an unfair / biased way if used on other tasks. Although the purpose of this project was to investigate transfer learning, the performance on languages that the models were not trained for does suffer.
Because all of these models covered here used xlm-roberta-base as their starting point (potentially with domain adaptive fine-tuning on specific languages), this model's limitations can also apply here. These can include being biased towards the hegemonic viewpoint of most of its training data, being ungrounded and having subpar results on other languages (possibly due to unbalanced training data).
The embeddings we provide are meant purely for exploratory and analysis purposes, and might not be useful, or even dangerous when deployed in a high stakes environment that has tangible consequences.
As Adelani et al. (2021) showed, the models in general struggled with entities that were either longer than 3 words and entities that were not contained in the training data. This could bias the models towards not finding, e.g. names of people that have many words, possibly leading to a misrepresentation in the results. Similarly, names that are uncommon, and may not have been found in the training data (due to e.g. different languages) would also be predicted less often.
Additionally, these models have not been verified in practice, and other, more subtle problems may become prevalent if used without any verification that it does what it is supposed to.
The data comes from only publicly available news sources, the only available data should cover public figures and those that agreed to be reported on. See the original MasakhaNER paper for more details.
No explicit ethical considerations or adjustments were made during fine-tuning of these models.
As shown in our report, and in the MasakhaNER paper, the language adaptive models achieve (mostly) superior performance over starting with xlm-roberta-base. We also demonstrated some better transfer capabilities for some of these models. Our main metric was the aggregate F1 score for all NER categories.
These metrics are on the test set for MasakhaNER, so the data distribution is similar to the training set, so these results do not directly indicate how well these models generalise. We do find large variation in results when starting from different seeds (5 different seeds were tested), indicating that the fine-tuning process for transfer might be unstable.
The metrics used were chosen to be consistent with previous work, and to facilitate research. Other metrics may be more appropriate for other purposes.
In general, these models performed worse on the 'date' category compared to others, so if dates are a critical factor, then that might need to be taken into account and addressed, by for example collecting and annotating more data.
Here are some details regarding the specific models, like where to find them, what the starting points where and what they were evaluated on. Evaluation here is also the same language that the model was fine-tuned on. All of these metrics were calculated on the test set, and the seed was chosen that gave the best overall F1 score. The first three result columns are averaged over all categories, and the latter 4 provide performance broken down by category.
These models can predict the following label for a token (source):
| Abbreviation | Description |
|---|---|
| O | Outside of a named entity |
| B-DATE | Beginning of a DATE entity right after another DATE entity |
| I-DATE | DATE entity |
| B-PER | Beginning of a person’s name right after another person’s name |
| I-PER | Person’s name |
| B-ORG | Beginning of an organisation right after another organisation |
| I-ORG | Organisation |
| B-LOC | Beginning of a location right after another location |
| I-LOC | Location |
To use these models, you can do the following, just changing the model name (source):
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline
model_name = 'mbeukman/xlm-roberta-base-finetuned-hausa-finetuned-ner-hausa'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)
nlp = pipeline("ner", model=model, tokenizer=tokenizer)
example = "Donald Trump Ya Rufe Gwamnatin Amurka"
ner_results = nlp(example)
print(ner_results)
# [{'entity': 'B-PER', 'score': 0.9998982, 'index': 1, 'word': '▁Donald', 'start': 0, 'end': 6}, {'entity': 'I-PER', 'score': 0.9999061, 'index': 2, 'word': '▁Trump', 'start': 6, 'end': 12}, {'entity': 'B-LOC', 'score': 0.9999413, 'index': 7, 'word': '▁Amurka', 'start': 30, 'end': 37}]