The sentence is split up to individual characters with no PoS tags

[kyu999]

Issue:

When the text is a bit long(in sample case, over 20000 letters), the sentence is split up to individual characters with no PoS tags.

bug.js

var fs = require('fs');
var RakutenMA = require("rakutenMA")

var model = JSON.parse(fs.readFileSync("../model_ja.json"));
rma = new RakutenMA(model, 1024, 0.007812);  // Specify hyperparameter for SCW (for demonstration purpose)
rma.featset = RakutenMA.default_featset_ja;
rma.hash_func = RakutenMA.create_hash_func(15);

var tokens = null
fs.readFile('sample.txt', 'utf8', function (err, text) {
    console.log(text);
    tokens = rma.tokenize(text)    
    fs.writeFile('tokens.txt', tokens , function (err) {
    console.log(err);
    });   
});

tokens.txt

著,,雍,,争,,奪,,戦,,で,,は,,、,,初,,日,,か,,ら,,前,,線,,へ,,出,,過,,ぎ,,た,,こ,,と,,で,,荀,,早,,隊,,に,,囚,,わ,,れ,,る,,失,,態,,を,,犯,,す,,も,,、,,咄,,嗟,,の,,判,,断,,で,,羌,,瘣,,に,,荀,,早,,を,,捕,,ら,,え,,さ,,せ,,た,,。,,そ,,の,,夜,,、,,魏,,国,,兵,,た,,ち,,に,,槍,,で,,小,,突,,き,,回,,さ,,れ,,る,,が,,、,,眼,,前,,の,,凱,,孟,,か,,ら,,の,,問,,い,,に,,臆,,せ,,ず,,素,,直,,に,,胸,,の,,内,,を,,語,,っ,,た,,こ,,と,,で,,、,,そ,,れ,,以,,上,,は,,粗,,略,,に,,さ,,れ,,ず,,、,,翌,,日,,の,,人,,質,,交,,換,,で,,飛,,信,,隊 (and so on)