Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Medical MVP: VQA viability #44

Closed
andreimano opened this issue Jul 24, 2020 · 2 comments
Closed

Medical MVP: VQA viability #44

andreimano opened this issue Jul 24, 2020 · 2 comments
Assignees

Comments

@andreimano
Copy link
Collaborator

We should check if it's possible to solve some VQA tasks on medical data. VQA is especially nice when combined with #43 and #35 .

Depends on #33 .

@andreimano andreimano self-assigned this Jul 24, 2020
@andreimano
Copy link
Collaborator Author

andreimano commented Jul 24, 2020

MONAI seems to have no pretrained models for this task yet 1.

This paper 2 reports decent results when using deep convolutional network coupled with transformers or recurrent neural networks.

A bigger dataset can be found at 3, but it's a lot newer.

The example in 4 is from "Show, attend and tell", which is a pretty old paper. A simple CNN encoder - RNN decoder could be used as a baseline, it should also work fine with Captum. 3 reports good results with this type of network architecture.

@andreimano
Copy link
Collaborator Author

A demo should be possible, we should first train a baseline and make sure that it works, and then mix it up with Captum. See #45.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant