Skip to content
master
Switch branches/tags
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Hierarchical Deep Multi-modal Network for MedicalVisual Question Answering

Introduction

Visual Question Answering in Medical domain (VQA-Med) plays an important role in providing medical assistance to the end-users. These users are expected to raise either a straightforward question with a Yes/No answer or a challenging question that requires a detailed and descriptive answer. The existing techniques in VQA-Med fail to distinguish between the different question types sometimes complicates the simpler problems, or over-simplifies the complicated ones. It is certainly true that for different question types, several distinct systems can lead to confusion and discomfort for the end-users. To address this issue, we propose a hierarchical deep multi-modal network that analyzes and classifies end-user questions/queries and then incorporates a query-specific approach for answer prediction. We refer our proposed approach as Hierarchical Question Segregation based Visual Question Answering, in short HQS-VQA.

Requirements

  • pandas
  • scikit-learn
  • matplotlib
  • pillow
  • nltk
  • keras

Dataset.

Dataset is taken from the paper available at https://www.nature.com/articles/sdata2018251

Input data format

image_id|question|answer
...

Train/load the model and generate the predictions

Download glove.6B.300d.txt file and place it in the data/external/glove folder.

python3 main.py

Evaluate and compare the models

python3 evaluate.py

License

MIT License

About

Hierarchical-Deep-Multi-modal-Network-for-MedicalVisual-Question-Answering

Resources

License

Releases

No releases published

Packages

No packages published