Skip to content
Portuguese pre-trained BERT models
Branch: master
Clone or download

Latest commit

fabiocapsouza Updated README
Missing do_lower_case=False argument on tokenizer instantiation
Latest commit 54facdc Mar 16, 2020

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
LICENSE Added license Jan 22, 2020
README.md Updated README Mar 15, 2020
qualifying_exam-portuguese_named_entity_recognition_using_bert_crf.pdf Updated README and qualifying exam text Jan 22, 2020

README.md

** This is a work in progress **

Portuguese BERT

This repository contains pre-trained BERT models trained on the Portuguese language. BERT-Base and BERT-Large Cased variants were trained on the BrWaC (Brazilian Web as Corpus), a large Portuguese corpus, for 1,000,000 steps, using whole-word mask. Model artifacts for TensorFlow and PyTorch can be found below.

The models are a result of an ongoing Master's Program. The text submission for Qualifying Exam is also included in the repository in PDF format, which contains more details about the pre-training procedure, vocabulary generation and downstream usage in the task of Named Entity Recognition.

Download

Model TensorFlow checkpoint PyTorch checkpoint Vocabulary
bert-base-portuguese-cased Download Download Download
bert-large-portuguese-cased Download Download Download

NER Benchmarks

The models were benchmarked on the Named Entity Recognition task and compared to previous published results and Multilingual BERT. Reported results are for BERT or BERT-CRF architectures, while other results comprise distinct methods.

Test Dataset BERT-Large Portuguese BERT-Base Portuguese BERT-Base Multilingual Previous SOTA
MiniHAREM (5 classes) 83.30 83.03 79.44 82.26 [1], 76.27[2]
MiniHAREM (10 classes) 78.67 77.98 74.15 74.64 [1], 70.33[2]

PyTorch usage example

Our PyTorch artifacts are compatible with the 🤗Huggingface Transformers library and are also available on the Community models:

from transformers import AutoModel, AutoTokenizer

# Using the community model
# BERT Base
tokenizer = AutoTokenizer.from_pretrained('neuralmind/bert-base-portuguese-cased')
model = AutoModel.from_pretrained('neuralmind/bert-base-portuguese-cased')

# BERT Large
tokenizer = AutoTokenizer.from_pretrained('neuralmind/bert-large-portuguese-cased')
model = AutoModel.from_pretrained('neuralmind/bert-large-portuguese-cased')

# or, using BertModel and BertTokenizer directly
from transformers import BertModel, BertTokenizer

tokenizer = BertTokenizer.from_pretrained('path/to/vocab.txt', do_lower_case=False)
model = BertModel.from_pretrained('path/to/bert_dir')  # Or other BERT model class

Acknowledgement

We would like to thank Google for Cloud credits under a research grant that allowed us to train these models.

References

[1] Assessing the Impact of Contextual Embeddings for Portuguese Named Entity Recognition

[2] Portuguese Named Entity Recognition using LSTM-CRF

How to cite this work

@article{souza2019portuguese,
    title={Portuguese Named Entity Recognition using BERT-CRF},
    author={Souza, Fabio and Nogueira, Rodrigo and Lotufo, Roberto},
    journal={arXiv preprint arXiv:1909.10649},
    url={http://arxiv.org/abs/1909.10649},
    year={2019}
}
You can’t perform that action at this time.