Skip to content
RNN and Markov Chain-based Image Caption Generator
Branch: master
Clone or download
Pull request Compare This branch is 9 commits ahead, 1 commit behind hunkim:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

RNN and Markov Chain-based Caption Generation

basic img

This is the result of me being bored at 3 AM during exam week when I can't sleep because REM cycles or something.

Anyway, this is just a rough sketch of how captions can be generated from just an image.

How it works

First, the image gets processed through Clarifai and returns tags that describe the image. I then took hunkim's word-RNN repository to take a set of encouraging messages and generate a heuristic of a caption based on the relevant captions.

Once the RNN spits out a garbage caption, I combine that with a Markov chain trained on the original messages so that the grammar is not too bad, and there is slight relevancy to the original image.


Step 1:

Install the Clarifai Python client. Set it up according to their instructions, including getting your own Client ID and Secret.

Step 2:

Install the remaining dependencies, including markovify, numpy, and tensorflow if you don't have them already.

Step 3:

Change line 33 of to the URL of an image you want to caption.

Step 4:

Run to train the RNN, then run to test it out!

You can’t perform that action at this time.