Skip to content
BERT-NER (nert-bert) with google bert https://github.com/google-research.
Branch: master
Clone or download
Latest commit 6a5db94 Jul 6, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
exps update results Jul 6, 2019
modules update results Jul 6, 2019
.gitignore add new version Jul 6, 2019
README.md Update README.md Jul 6, 2019
requirements.txt add new version Jul 6, 2019

README.md

0. Papers

There are two solutions based on this architecture.

  1. BSNLP 2019 ACL workshop: solution and paper on multilingual shared task.
  2. The second place solution of Dialogue AGRR-2019 task and paper.

Description

This repository contains solution of NER task based on PyTorch reimplementation of Google's TensorFlow repository for the BERT model that was released together with the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.

This implementation can load any pre-trained TensorFlow checkpoint for BERT (in particular Google's pre-trained models).

Old version is in "old" branch.

2. Usage

2.1 Create data

from modules.data import bert_data
data = bert_data.LearnData.create(
    train_df_path=train_df_path,
    valid_df_path=valid_df_path,
    idx2labels_path="/path/to/vocab",
    clear_cache=True
)

2.2 Create model

from modules.models.bert_models import BERTBiLSTMAttnCRF
model = BERTBiLSTMAttnCRF.create(len(data.train_ds.idx2label))

2.3 Create Learner

from modules.train.train import NerLearner
num_epochs = 100
learner = NerLearner(
    model, data, "/path/for/save/best/model", t_total=num_epochs * len(data.train_dl))

2.4 Predict

from modules.data.bert_data import get_data_loader_for_predict
learner.load_model()
dl = get_data_loader_for_predict(data, df_path="/path/to/df/for/predict")
preds = learner.predict(dl)

2.5 Evaluate

from sklearn_crfsuite.metrics import flat_classification_report
from modules.analyze_utils.utils import bert_labels2tokens, voting_choicer
from modules.analyze_utils.plot_metrics import get_bert_span_report
from modules.analyze_utils.main_metrics import precision_recall_f1


pred_tokens, pred_labels = bert_labels2tokens(dl, preds)
true_tokens, true_labels = bert_labels2tokens(dl, [x.bert_labels for x in dl.dataset])
tokens_report = flat_classification_report(true_labels, pred_labels, digits=4)
print(tokens_report)

results = precision_recall_f1(true_labels, pred_labels)

3. Results

We didn't search best parametres and obtained the following results.

Model Data set Dev F1 tok Dev F1 span Test F1 tok Test F1 span
OURS
M-BERTCRF-IO FactRuEval - - 0.8543 0.8409
M-BERTNCRF-IO FactRuEval - - 0.8637 0.8516
M-BERTBiLSTMCRF-IO FactRuEval - - 0.8835 0.8718
M-BERTBiLSTMNCRF-IO FactRuEval - - 0.8632 0.8510
M-BERTAttnCRF-IO FactRuEval - - 0.8503 0.8346
M-BERTBiLSTMAttnCRF-IO FactRuEval - - 0.8839 0.8716
M-BERTBiLSTMAttnNCRF-IO FactRuEval - - 0.8807 0.8680
M-BERTBiLSTMAttnCRF-fit_BERT-IO FactRuEval - - 0.8823 0.8709
M-BERTBiLSTMAttnNCRF-fit_BERT-IO FactRuEval - - 0.8583 0.8456
- - - - - -
BERTBiLSTMCRF-IO CoNLL-2003 0.9629 - 0.9221 -
B-BERTBiLSTMCRF-IO CoNLL-2003 0.9635 - 0.9229 -
B-BERTBiLSTMAttnCRF-IO CoNLL-2003 0.9614 - 0.9237 -
B-BERTBiLSTMAttnNCRF-IO CoNLL-2003 0.9631 - 0.9249 -
Current SOTA
DeepPavlov-RuBERT-NER FactRuEval - - - 0.8266
CSE CoNLL-2003 - - 0.931 -
BERT-LARGE CoNLL-2003 0.966 - 0.928 -
BERT-BASE CoNLL-2003 0.964 - 0.924 -
You can’t perform that action at this time.