Skip to content

samarthramesh/multimodal-neuralnet

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Multimodal Neural Network

Multimodal recipe retrieval system built using PyTorch for text (instructions and ingredients) and images

Dataset

Utilized the MIT Recipe 1M dataset for recipe information

Text Side

The Infersent sentence embeddings were used to embed the recipe instructions

The individual ingredients were cleaned and embedded using TensorFlow

Image Side

The ResNet-50 CNN model was used for the image embeddings

Ingredient encodings were fed into a bi-directional LSTM and instruction embeddings were fed into a unidirectional LSTM. These encodings were concatenated and passed through a fully connected layer. Cosine similarity loss function was then used to train the model. First, the text side was trained and then, resnet-50 model was fine tuned.

Contributors

Ananya Gupta, Raghav Sriram, Samarth Ramesh, Samyak Jain

License

MIT

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Jupyter Notebook 100.0%