BERT MULTI GPU

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

REQUIREMENT

python 3

tensorflow 1.14

0, edit the input and output file name in create_pretraining_data.py and run_pretraining_gpu.py

1, run create_pretraining_data.py

2, run run_pretraining_gpu_v2.py

Edit n_gpus in run_pretraining_gpu.py

In sample_text.txt, sentence is end by \n, paragraph is splitted by empty line.

Quora question pairs English dataset,

Official BERT: ACC 91.2, AUC 96.9

This BERT with pretrain loss 2.05: ACC 90.1, AUC 96.3

Name		Name	Last commit message	Last commit date
Latest commit History 40 Commits
tmp_data		tmp_data
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
__init__.py		__init__.py
bert_config.json		bert_config.json
create_pretraining_data.py		create_pretraining_data.py
extract_features.py		extract_features.py
modeling.py		modeling.py
modeling_test.py		modeling_test.py
multilingual.md		multilingual.md
optimization.py		optimization.py
optimization_gpu.py		optimization_gpu.py
optimization_test.py		optimization_test.py
requirements.txt		requirements.txt
run_classifier.py		run_classifier.py
run_pretraining.py		run_pretraining.py
run_pretraining_gpu.py		run_pretraining_gpu.py
run_pretraining_gpu_v2.py		run_pretraining_gpu_v2.py
run_squad.py		run_squad.py
sample_text.txt		sample_text.txt
tokenization.py		tokenization.py
tokenization_test.py		tokenization_test.py
vocab.py		vocab.py
vocab.txt		vocab.txt