Skip to content
repository for Publicly Available Clinical BERT Embeddings
Python Jupyter Notebook Perl Shell
Branch: master
Clone or download
Latest commit 5c7cb00 Oct 9, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
downstream_tasks add github links Oct 9, 2019
lm_pretraining add lm pretraining code Jun 29, 2019
.gitignore add lm pretraining code Jun 29, 2019
LICENSE add MIT license Aug 26, 2019
README.md Update README.md Oct 9, 2019
requirements.txt add requirements Jun 29, 2019

README.md

clinicalBERT

Repository for Publicly Available Clinical BERT Embeddings (NAACL Clinical NLP Workshop 2019)

Download Clinical BERT

The Clinical BERT models can be downloaded here, or via

wget -O pretrained_bert_tf.tar.gz https://www.dropbox.com/s/8armk04fu16algz/pretrained_bert_tf.tar.gz?dl=1

biobert_pretrain_output_all_notes_150000 corresponds to Bio+Clinical BERT, and biobert_pretrain_output_disch_100000 corresponds to Bio+Discharge Summary BERT. Both models are finetuned from BioBERT.

Reproduce Clinical BERT

Pretraining

To reproduce the steps necessary to finetune BERT or BioBERT on MIMIC data, follow the following steps:

  1. Run format_mimic_for_BERT.py - Note you'll need to change the file paths at the top of the file.
  2. Run create_pretrain_data.sh
  3. Run finetune_lm_tf.sh

Note: See issue #4 for ways to improve section splitting code.

Downstream Tasks

To see an example of how to use clinical BERT for the Med NLI tasks, go to the run_classifier.sh script in the downstream_tasks folder. To see an example for NER tasks, go to the run_i2b2.sh script.

Contact

Please post a Github issue or contact emilya@mit.edu if you have any questions.

Citation

Please acknowledge the following work in papers or derivative software:

Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.

@inproceedings{alsentzer-etal-2019-publicly,
    title = "Publicly Available Clinical {BERT} Embeddings",
    author = "Alsentzer, Emily  and
      Murphy, John  and
      Boag, William  and
      Weng, Wei-Hung  and
      Jin, Di  and
      Naumann, Tristan  and
      McDermott, Matthew",
    booktitle = "Proceedings of the 2nd Clinical Natural Language Processing Workshop",
    month = jun,
    year = "2019",
    address = "Minneapolis, Minnesota, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/W19-1909",
    doi = "10.18653/v1/W19-1909",
    pages = "72--78"
}
You can’t perform that action at this time.