Keras implementation of a Siamese Neural Network for Joint Multimodal Text-Image Embedding
Switch branches/tags
Nothing to show
Clone or download
Latest commit 09bf967 Oct 18, 2017
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
README.md Update README.md Oct 17, 2017
multimodal-embedding-siamese-network.py Create multimodal-embedding-siamese-network.py Jul 31, 2017

README.md

Joint-Multimodal-Embedding

Keras implementation of a Siamese Neural Network for Joint Multimodal Text-Image Embedding

Tested with a Tensorflow backend

This embedding network was used in a larger deep learning project aimed at using GANs to generate video from text descriptions. Check out my blogpost on the results.

Tools Used:

Python 3
Keras
Keras' Xception V1 model pre-trained on ImageNet

Text data vectorized prior to running this network using a Variational Autoencoder and Facebook's fastText pre-trained Word2Vec. More information found here.