Hierarchical Deep Multi-modal Network for MedicalVisual Question Answering
Visual Question Answering in Medical domain (VQA-Med) plays an important role in providing medical assistance to the end-users. These users are expected to raise either a straightforward question with a Yes/No answer or a challenging question that requires a detailed and descriptive answer. The existing techniques in VQA-Med fail to distinguish between the different question types sometimes complicates the simpler problems, or over-simplifies the complicated ones. It is certainly true that for different question types, several distinct systems can lead to confusion and discomfort for the end-users. To address this issue, we propose a hierarchical deep multi-modal network that analyzes and classifies end-user questions/queries and then incorporates a query-specific approach for answer prediction. We refer our proposed approach as Hierarchical Question Segregation based Visual Question Answering, in short HQS-VQA.
Dataset is taken from the paper available at https://www.nature.com/articles/sdata2018251
Input data format
Train/load the model and generate the predictions
Download glove.6B.300d.txt file and place it in the data/external/glove folder.
Evaluate and compare the models