This is the pytorch implement for our paper S2IGAN: Speech-to-Image Generation via Adversarial Learning. More results can be seen in the project page.
step0: You can download the synthesized spoken caption database from the project page, and then go to "step3". Or, you can start from "step 1" and synthesize spoken caption by yourself.
step1: Download CUB and Oxford Image and Text Captions
step2: Using a TTS system to transfer text captions to speech captions. In our work, Tacotron2 pre-trained by NVIDIA was adopted. The original code released by NVIDIA doesn't provide the code for the batch inference. We made slight changes to make it can be used to perform batch inference directly. You can download it here.
step3: To speed up the training process, we transferred the wav audio to filter bank spectrogram in advance.
python data_processing/Audio_to_mel.py
step4: Download the train/test split files for CUB and Oxford
Directory tree
├── birds
│ ├── CUB_200_2011
│ │ ├── auido
│ │ ├── audio_mel
│ │ ├── images
│ ├── train
│ │ ├── filenames.pickle
│ │ ├── class_info.pickle
│ ├── test
│ │ ├── filenames.pickle
│ │ ├── class_info.pickle
├── flowers
│ ├── Oxford102
│ │ ├── auido
│ │ ├── audio_mel
│ │ ├── images
│ ├── train
│ │ ├── filenames.pickle
│ │ ├── class_info.pickle
│ ├── test
│ │ ├── filenames.pickle
│ │ ├── class_info.pickle
Download Places audio data. Images of Places-subset and split files can be downloaded here. Database files are organized as follows
├── places
│ ├── images
│ ├── audio
│ │ ├── mel
│ │ ├── wav
│ ├── train
│ │ ├── filenames.pickle
│ ├── test
│ │ ├── filenames.pickle
step1: Download Flickr8k Image and Audio Captions
step2: Transfer wav to spectrogram.
step2: Download the train/test split files.
├── Flickr8k
│ ├── images
│ ├── flickr_audio
│ │ ├── mel
│ │ ├── wavs
│ ├── train
│ │ ├── filenames.pickle
│ ├── test
│ │ ├── filenames.pickle
Note: Change the path in .sh files to your data path. If you use the speech embedding provided by us (see Step 2), you can start from step3.
step1: Train SEN
sh run/flickr/01_Pretrain.sh
step2: Extract speech embeddings
sh run/flickr/02_Extracting.sh
You can skip the first two steps by using our provided speech embeddings and pre-trained image encoder for CUB, Oxford, Flickr8k, and Places-subset. Then put these embeddings as follows:
├── outputs
│ ├── pre_train
│ │ ├── birds
│ │ │ ├── speech_embeddings_train.pickle
│ │ │ ├── speech_embeddings_test.pickle
│ │ │ ├── models
│ │ │ │ ├── best_image_model.pth
│ │ ├── flowers
│ │ │ ├── speech_embeddings_train.pickle
│ │ │ ├── speech_embeddings_test.pickle
│ │ │ ├── models
│ │ │ │ ├── best_image_model.pth
│ │ ├── flickr
│ │ │ ├── speech_embeddings_train.pickle
│ │ │ ├── speech_embeddings_test.pickle
│ │ │ ├── models
│ │ │ │ ├── best_image_model.pth
│ │ ├── places
│ │ │ ├── speech_embeddings_train.pickle
│ │ │ ├── speech_embeddings_test.pickle
│ │ │ ├── models
│ │ │ │ ├── best_image_model.pth
step3: Train the generator
sh run/flickr/03_TrainGAN.sh
step4: Generate images
sh run/flickr/04_GenImage.sh
step5: Calculate Insception Score (IS)
For Flickr and Places-subset, you can directly run the .sh files in the corresponding directory, such as
sh run/flickr/05_InsceptionScore_generally.sh
For CUB and Oxford, we use the fine-tuned model
step6: Semantic Consistency Evaluation
For Flickr and Places-subset:
sh run/flickr/06_Recall.sh
For CUB and Oxford:
sh run/birds/06_mAP.sh
step7: FID Download the code to calculate the FID
@article{wang2020s2igan,
title={S2IGAN: Speech-to-Image Generation via Adversarial Learning},
author={Wang, Xinsheng and Qiao, Tingting and Zhu, Jihua and Hanjalic, Alan and Scharenborg, Odette},
journal={arXiv preprint arXiv:2005.06968},
year={2020}
}