Performance optimization for 'split' by 'partition'

[Mifrill]

Method split can be changed to partition to speed up the entire process.

lemmatizer/lib/lemmatizer/lemmatizer.rb

Line 118 in af70f99

w = line.split(/\s+/)[0]

require 'benchmark'

Benchmark.bm do |x|
  string = "'hood n 1 2 @ ; 1 0 08641944"
  x.report { 50000.times { string.split(/\s+/)[0] } }
  x.report { 50000.times { string.partition(/\s+/)[0] } }
end

[
 #<Benchmark::Tms:0x0000560ef6b6cc30 @label="", @real=0.18285021604970098, @cstime=0.0, @cutime=0.0, @stime=2.3999999999996247e-05, @utime=0.18282500000000024, @total=0.18284900000000023>, 
 #<Benchmark::Tms:0x0000560ef6b864a0 @label="", @real=0.05183851800393313, @cstime=0.0, @cutime=0.0, @stime=1.2000000000012001e-05, @utime=0.0518200000000002, @total=0.05183200000000021>
]

The change of real from 0.18284900000000023 to 0.05183200000000021 looks nice, @yohasebe what do you think?

Source:
https://stackoverflow.com/questions/7533479/ruby-string-search-which-is-faster-split-or-regex

[Mifrill]

It seems like this split method usage also might be a case for performance optimization, @yohasebe what do you think?

lemmatizer/lib/lemmatizer/lemmatizer.rb

Line 125 in af70f99

w, s = line.split(/\s+/)

[yohasebe]

I'm so sorry for my late reply. I needed a little time to refresh my memory about lemmatizer to respond to your suggestions and pull request. Your suggestion here looks good to me!