Skip to content

phueb/Preppy

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

A small Python package for preparing ordered language data for RNN language models.

Tokenization is not included.

Usage

from preppy import Prep

sentences = ['Hello World.', 'Hello World.']

prep = Prep(sentences,
            reverse=False,  # generate batches starting from last document
            batch_size=1,   # batch size 
            context_size=1, # number of back-prop-through-time steps
            sliding=False,  # windows slide over input text
            )
            
for batch in prep.generate_batches():
   pass  # train model on batch

Compatibility

Developed on Ubuntu 18.04 and Python 3.7