No description or website provided.
Clone or download
bei21 Merge pull request #7 from wkcn/support_old_version_of_mxnet
Fix no `flatten` argument error. support old version of mxnet
Latest commit 5f395be Oct 28, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
code Merge pull request #7 from wkcn/support_old_version_of_mxnet Oct 28, 2018
data add dataset Jul 19, 2018
img update readme Oct 23, 2018
.gitignore update readme Oct 23, 2018
README.md update example Oct 25, 2018

README.md

Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training

An implementation of the image-to-poem model described in the paper: "Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training."

Bei Liu, Jianlong Fu, Makoto P. Kato, Masatoshi Yoshikawa

Full text available at: https://arxiv.org/abs/1804.08473

Contents

Model Overview

Introduction

The Img2poem model is a deep neural network that learns how to generate poems from images. For example:

Architecture

Getting Started

Install Required Packages

(It is recommended to install the dependencies under Conda environment.)

  • python2.7
  • tensorflow1.6
  • mxnet
  • opencv
  • scikit-image
  • tqdm
  • colorama
  • flask

Prepare the Training Data

Name #Poem #Line/poem #Word/line
MultiM-Poem 8,292 7.2 5.7
UniM-Poem 93,265 5.7 6.2
MultiM-Poem(Ex) 26,161 5.4 5.9

Both datasets are formatted in JSON files.

MultiM-Poem.json: image and poem pairs

[
    {
        "poem": str,
        "image_url": str,
        "id": int
    },
    ...
]

UniM-Poem.json: poem corpus

[
    {
        "poem": str,
        "id": int
    },
    ...
]

Download Trained Model

Please download models from https://1drv.ms/u/s!AkLgJBAHL_VFgSyyfpeGyGFZux56 and put it under "code/".

Generating Poems

The following command line will generate poem for an image.

python test.py

Type in the relative path to the test image in the console and the poem will be generated.

../images/test.jpg

Example output:

the sun is singing in the forest wind
and let us go to the wind of the sun
let the sun be free
let us be the storm of heaven
and let us be the slow sun
we keep our own strength together
we live in love and hate

Results

Here are some examples of poems generated by eight methods for an image.