Skip to content
Switch branches/tags


Failed to load latest commit information.
Latest commit message
Commit time


Repository for Publicly Available Clinical BERT Embeddings (NAACL Clinical NLP Workshop 2019)

Using Clinical BERT

UPDATE: You can now use ClinicalBERT directly through the transformers library. Check out the Bio+Clinical BERT and Bio+Discharge Summary BERT model pages for instructions on how to use the models within the Transformers library.

Download Clinical BERT

The Clinical BERT models can also be downloaded here, or via

wget -O pretrained_bert_tf.tar.gz

biobert_pretrain_output_all_notes_150000 corresponds to Bio+Clinical BERT, and biobert_pretrain_output_disch_100000 corresponds to Bio+Discharge Summary BERT. Both models are finetuned from BioBERT. We specifically use the BioBERT-Base v1.0 (+ PubMed 200K + PMC 270K) version of BioBERT.

bert_pretrain_output_all_notes_150000 corresponds to Clinical BERT, and bert_pretrain_output_disch_100000 corresponds to Discharge Summary BERT. Both models are finetuned from the cased version of BERT, specifically cased_L-12_H-768_A-12.

Reproduce Clinical BERT


To reproduce the steps necessary to finetune BERT or BioBERT on MIMIC data, follow the following steps:

  1. Run - Note you'll need to change the file paths at the top of the file.
  2. Run
  3. Run

Note: See issue #4 for ways to improve section splitting code.

Downstream Tasks

To see an example of how to use clinical BERT for the Med NLI tasks, go to the script in the downstream_tasks folder. To see an example for NER tasks, go to the script.


Please post a Github issue or contact if you have any questions.


Please acknowledge the following work in papers or derivative software:

Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.

    title = "Publicly Available Clinical {BERT} Embeddings",
    author = "Alsentzer, Emily  and
      Murphy, John  and
      Boag, William  and
      Weng, Wei-Hung  and
      Jin, Di  and
      Naumann, Tristan  and
      McDermott, Matthew",
    booktitle = "Proceedings of the 2nd Clinical Natural Language Processing Workshop",
    month = jun,
    year = "2019",
    address = "Minneapolis, Minnesota, USA",
    publisher = "Association for Computational Linguistics",
    url = "",
    doi = "10.18653/v1/W19-1909",
    pages = "72--78"


repository for Publicly Available Clinical BERT Embeddings







No releases published


No packages published