Skip to content

A project for telling stories according to images in some particular style

Notifications You must be signed in to change notification settings

seaweiqing/image2story

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

32 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

image2story

Written by LIN Jinghong, WEI Qing, SHI Siyuan

project introduction

We set up a model for generating a story of some specific style from pictures and named it "image2story". The model is based on the previous image caption model and we expand the image caption model to generate a complete paragraph instead of a single sentence. Our model can be roughly divided into three parts. First, the sentence description is extracted from the picture through the image caption network, at the same time, the skip - thought vectors model is trained to realize the conversion between sentence and vector. Then we feed the sentences extracted by image caption into the encoder of skip - thought vectors, and convert the image caption into a vector with its own text style. Then we use romantic novels as our data set and train a story generator, the principle of which is similar to that of skip - thought vectors. Finally, the vector is fed into the story generator to generate romantic paragraphs.

pretrained model

The pretrained model merges the image caption model from https://github.com/DeepRNN/image_captioning and a sentence-to-story model extracted from https://github.com/ryankiros/neural-storyteller. First, download the needed model this, put the caption.npy into the caption_model folder, put the stv_model folder as well as romance_models folder into the story_model folder. Then put the image you would like to test into the test/images folder and run the demo.py the generated stories will be stored in the variable -passages. Also, it will produce the image caption images each with 3 captions, which is stored in the test/results folder.

dependencies

  • Python 3
  • Theano
  • tensorflow

output samples

Result Result Result Result Result Result Result Result Result

Releases

No releases published

Packages

No packages published

Languages