MORQA Evaluation

This is the code and data for Medical Open Response Question Answering (MORQA) Evaluation. Further experimental details can be found in our paper:

Citation:

@inproceedings{mediqa-eval-dataset,
author       = {Wen{-}wai Yim and Asma Ben Abacha and Zixuan Yu and Robert Doerning and Fei Xia and Meliha Yetisgen}, 
title        = {{MORQA:} Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering}, 
journal      = {Proceedings of the Fifteenth Language Resources and Evaluation Conference}, year         = {2026}
}

Code repo: https://github.com/wyim/morqa-eval Dataset repo: https://osf.io/kcv2n/overview

Dataset

Files

Human ratings can be found in the dataset repo: https://osf.io/kcv2n/overview Baselines system and evaluation scores from the paper can be found in the morqa-experiments-20250928 folder.

Underlying Consumer Health Datasets

dataset	split	#query	#gold-responses	#system-responses	lang	link
woundcare	valid	105	210	315	{en,zh}	https://osf.io/xsj5u/overview
woundcare	test	93	279	279	{en,zh}	https://osf.io/xsj5u/overview
iiyi	valid	56	417	158	{en,zh}	https://osf.io/72rp3/files/osfstorage/67c63d108afb6ebcf808b8c1
iiyi	test	100	926	300	{en,zh}	https://osf.io/72rp3/files/osfstorage/6694ba795a51c404a8e802d1
liveqa	test	40	62	233	{en}	https://osf.io/kcv2n/files/8xzg3
med_dialog	valid	236	236	708	{zh}	https://osf.io/kcv2n/files/n75yx
med_dialog	test	259	259	777	{zh}	https://osf.io/kcv2n/files/n75yx

(only selected liveqa questions were used - filtered for clinical questions)

Some liveqa and med_dialog from the underlying dataset are additionally skipped due to having relevant content, answers, or both. For liveqa use the SKIP attribute in the baseline file. For med_dialog, you can safely use the subset provided by the ratings file.

Human evaluations

Each system response for EN {woundcare/iiyi} datasets was rated according to the following description by a practicing medical doctor:

• disagree_flag: 1 if expert disgrees, 0 otherwise
• completeness: {0,0.5,1.0} 1 for complete answer to question, 0.5 partial, 0.0 inaccurate/missing critical information
• factual-accuracy: {0,0.5,1.0} 1 for factually acurate answer to question, 0.5 partial, 0.0 inaccurate/missing critical information
• relevance: {0,0.5,1.0} 1 for relevant question, 0.5 partially relevant, 0.0 irrelevant information
• writing-style: {0,0.5,1.0} 1 for appropriate writing style, 0.5 partial, 0 otherwise
• overall: 1 for complete answer to question, 0.5 partial, 0.0 inaccurate/missing critical information EN woundcare test dataset had 2 raters; where as EN woundcare valid had 1 rater.

These datasets also includes a seperate file for comments given at a response level.

Each system response for EN {liveqa} datasets was rated according to the following description by 2 medical NLM researchers:

• overall: normalized version of the original

Each system response for ZH {woundcare/iiyi} datasets was rated according to the following description by 1 domain expert trained at a Chinese Medical School:

• factual-consistency-wgold: {0,0.5,1.0} 1 for factual consistency with gold standard, 0.5 partial, 0 otherwise
• writing-style: {0,0.5,1.0} 1 for appropriate writing style, 0.5 partial, 0 otherwise

Total Judgements

EN

dataset	split	#system-responses	#raters
woundcare	valid	315	1
woundcare	test	279	2
iiyi	valid	158	2
iiyi	test	300	2
liveqa	test	232	-

EN judgement comments

dataset	split	#comments	#raters
woundcare	valid	122	1
woundcare	test	184	2
iiyi	valid	201	2
iiyi	test	465	2

ZH

dataset	split	#system-responses	#raters
woundcare	valid	315	1
woundcare	test	279	1
iiyi	valid	158	1
iiyi	test	300	1
med_dialog	valid	708	1
med_dialog	test	777	1

Code

Code for loading human ratings and baseline data, and calculating correlations {spearman,kendall,pearson} are located in the evaluate_rankings2.py.

You can run the evaluation with the following command:

python evaluate_rankings2.py <baseline-file> <ratings-file> <config-#>

To add your own metric, you can copy and modify the calculate_correlations.ipynb file.