cache total word counts per category

[parkr]

Ref: cardmagic/classifier#8

/cc @dabble

[parkr]

From @dabble:

Original code recalculated total word count per category for each classification.
On large datasets, this is really expensive.

On my dataset with 58k records in training set and 11k records in classifying set,
classification times improved from 1560 seconds to 107 seconds, and training time
only went from 593 to 691 seconds.

and

Note, I also attempted pulling category.to_s out of the loop and changing
s = category_words.has_key?(word) ? category_words[word] : 0.1
TO
s = category_words[word] || 0.1
and neither change made a significant difference in performance

[parkr]

Rebased on master.

[Ch4s3]

Rebased on master.

#edit Is there a plan for bringing this into master?

[parkr]

Is there a plan for bringing this into master?

I just ported it over. Is it particularly beneficial to you?

[Ch4s3]

Is it particularly beneficial to you?

Not especially, but I can understand why it was originally done.

[macotouenaca]

ありがとうございます