Skip to content


Folders and files

Last commit message
Last commit date

Latest commit



19 Commits

Repository files navigation

Machine reading comprehension on clinical case reports

This is the accompanying code of:

CliCR: A Dataset of Clinical Case Reports for Machine Reading Comprehension. Simon Šuster and Walter Daelemans. NAACL, 2018.

Alt text

Dataset availability

Thanks to an agreement with the publisher of BMJ Case Reports, we are allowed to freely distribute our dataset for research purposes. Please send us an email to, and we will provide you with the link.

Handling the dataset and baselines

Collect some statistics about the dataset:

python3 dataset-code/ -train_file TRAIN_PATH -dev_file DEV_PATH -test_file TEST_PATH


Neural readers (adapted to CliCR)

To train the Stanford Attentive Reader:

THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python --train_file PATH/TO/train1.0.json --dev_file PATH/TO/dev1.0.json --embedding_file PATH/TO/embeddings  --log_file best.log --att_output False

This will use default parameters, with hidden size and dropout rate optimized on the development set. It also by default removes those instances from the dataset for which the answers are not found in their exact form in the corresponding passage. To change any of these parameters, modify The model will be saved as best.model.

Test the SA model:

THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python --test_only True --pre_trained best.model --train_file PATH/TO/train1.0.json --dev_file PATH/TO/test1.0.json --embedding_file PATH/TO/embeddings --log_file best.test.log

To run evaluation separately:

python3 dataset-code/ -test_file PATH/TO/dev1.0.json -prediction_file predictions -embeddings_file PATH/TO/embeddings -downcase -extended

If you'd like to run an extended evaluation using BLEU and embedding-based metrics, use the option -extended. This embedding-based metrics will use the embeddings available in -embeddings_file.

To train the Gated Attention Reader with marked entities:

THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python3 --dataset clicr_plain --mode 1 --nhidden 67 --dropout 0.4 --use_feat 1 --data_path PATH/TO/dataset_plain/ent/gareader/ --experiments_path experiments/

Run python3 --help to see the full list of options.

To test the GA reader model:

THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python3 --dataset clicr_plain --mode 2 --nhidden 67 --dropout 0.4 --use_feat 1 --data_path PATH/TO/dataset_plain/ent/gareader/ --experiments_path experiments/