Assessing syntactic abilities of BERT
Clone or download
Latest commit a9fa1ac Jan 5, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
results initial Jan 6, 2019
LICENSE Initial commit Jan 6, 2019
README.md Update README.md Jan 6, 2019
eval_bert.py initial Jan 6, 2019
gen_gul_tbl.py initial Jan 6, 2019
gen_lgd_tbl.py initial Jan 6, 2019
gen_marvin_tbl.py initial Jan 6, 2019
generated.tab initial Jan 6, 2019
inflect.py initial Jan 6, 2019
lgd_dataset.tsv initial Jan 6, 2019
make_linzen_goldberg_testset.py initial Jan 6, 2019
marvin_linzen_dataset.tsv initial Jan 6, 2019
utils.py initial Jan 6, 2019

README.md

BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File Description
marvin_linzen_dataset.tsv stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
wiki.vocab from LGD, used for verb inflections (wiki.vocab)
lgd_dataset.tsv processed data from LGD
generated.tab data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py