No description, website, or topics provided.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
codec
data
generator
python
rnn
.gitignore Ignoring backup Oct 31, 2017
LICENSE
README.md
config.env
main.go
vocab.txt

README.md

This repository holds the basic implementation of a RNN.

It is based on the blog post The Unreasonable Effectiveness of Recurrent Neural Networks from Andrej Karpathy

The code is basically a transcript from his gist.

I also got some help from Daniel Whitenack's Building a Neural Net from Scratch in Go

For more information, please refer to this blog post

Configuration

Hyper parameters of the neural nerwork

RNN_INPUTNEURONS      Integer
RNN_OUTPUTNEURONS     Integer
RNN_HIDDENNEURONS     Integer    100        true
RNN_LEARNINGRATE      Float      1e-1       true
RNN_ADAGRADEPSILON    Float      1e-8       true
RNN_RANDOMFACTOR      Float      0.01

Parameters of the executable

MIN_CHAR_SAMPLESIZE         Integer    100        true
MIN_CHAR_SAMPLEFREQUENCY    Integer    1000       true
MIN_CHAR_INFOFREQUENCY      Integer    100        true
MIN_CHAR_BACKUPFREQUENCY    Integer    1000       true
MIN_CHAR_BACKUPPREFIX       String
MIN_CHAR_BACKUPSUFFIX       String

Parameters of the char codec

CHAR_CODEC_CHOICE     hard|soft (default hard)
CHAR_CODEC_EPOCH      100
CHAR_CODEC_VOCAB_FILE
CHAR_CODEC_INPUT_FILE
CHAR_CODEC_BATCHSIZE  default 25

Usage

Example:

This will train the RNN with Shakespeare inputs and save every now and then the model to shakespeare.bin

export CHAR_CODEC_INPUT_FILE=data/shakespeare/input.txt
export CHAR_CODEC_VOCAB_FILE=data/shakespeare/vocab.txt
export RNN_ADAGRADEPSILON=1e-8
export RNN_RANDOMFACTOR=0.1
export RNN_LEARNINGRATE=1e-1
export MIN_CHAR_CHOICE=hard
export RNN_HIDDENNEURONS=66
export MIN_CHAR_BATCHSIZE=25
export MIN_CHAR_SAMPLEFREQUENCY=1000
export MIN_CHAR_EPOCHS=100
export MIN_CHAR_SAMPLESIZE=500
export MIN_CHAR_BACKUPPREFIX=shakespeare
export MIN_CHAR_BACKUPFREQUENCY=1000
export CHAR_CODEC_CHOICE=soft
echo "starting sequence for the sampling" | ./min-char-rnn -train

To use the pre-train model:

echo "Initial sample" | ./min-char-rnn -restore shakespeare.bin