Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

学習時にスペースを使っていたバグを修正; symbolも使わないようにした

  • Loading branch information...
commit 1f36d269e2ea5b7ad31f9a741d34c96f833bc6f4 1 parent 7024f8d
@nagadomi authored
Showing with 7 additions and 3 deletions.
  1. +7 −3 lib/nekoneko_gen/text_classifier_generator.rb
View
10 lib/nekoneko_gen/text_classifier_generator.rb
@@ -98,7 +98,9 @@ def self.predict(text)
private
def self.fv(text)
prev = nil
- BimyouSegmenter.segment(text).map do |word|
+ BimyouSegmenter.segment(text,
+ :white_space => false,
+ :symbol => false).map do |word|
if (prev)
if (NGRAM_TARGET =~ word)
nword = [prev + word, word]
@@ -145,7 +147,9 @@ def word2id(word)
def fv(text)
vec = Hash.new(0)
prev = nil
- words = BimyouSegmenter.segment(text, :white_space => true).map do |word|
+ words = BimyouSegmenter.segment(text,
+ :white_space => false,
+ :symbol => false).map do |word|
if (prev)
if (NGRAM_TARGET =~ word)
nword = [prev + word, word]
@@ -208,7 +212,7 @@ def under_sampling(a, n)
end
end
def safe_name(filename)
- File.basename(filename, ".*").gsub('-','_').gsub(/[^a-zA-Z_0-9]/, '')
+ File.basename(filename, ".*").gsub(/[\-\.]/,'_').gsub(/[^a-zA-Z_0-9]/, '')
end
def puts(s)
unless (@quiet)
Please sign in to comment.
Something went wrong with that request. Please try again.