Permalink
Browse files

rnn language model with sentencepiece

  • Loading branch information...
kanoh-k committed Feb 3, 2018
1 parent 991dd62 commit 1cbec37d7361762a9f6e7942d8788026120db1c7
Showing with 465 additions and 0 deletions.
  1. +1 −0 WikiIterator.py
  2. +464 −0 rnnlm_sp.ipynb
@@ -107,6 +107,7 @@ def __init__(self, document):
self.sentences = self._break(document)

def _break(self, sentences):
sentences = sentences.replace('', '\n')
lines = sentences.split('\n')
return [line for line in lines if len(line) > 0]

Oops, something went wrong.

0 comments on commit 1cbec37

Please sign in to comment.