natural.Tfidf.listTerms works incorrectly for custom-generated tokens (those passed as array to addDocument(...))

[senatet]

natural.Tfidf.addDocument accepts either a string or an array of pre-tokenized texts. When a document is added using an array of tokens, listTerms still applies the tokenization to the individual document tokens when computing the tfidf score, resulting in a tfidf score of 0, even though the tf and idf scores are > 0.

(natural version: ^5.1.11)
An example:

> var natural = require('natural')
> var tfidf = new natural.TfIdf()
> tfidf.listTerms(0)
[
  {
    term: 'domain',
    tf: 1,
    idf: 0.3068528194400547,
    tfidf: 0.3068528194400547
  },
  { term: 'google.com', tf: 1, idf: 0.3068528194400547, tfidf: 0 }
]

The second document should have a tfidf score of 0.306... (1 * .0.3068..), but it is 0.

The fix is simple.. Update the listTerms(...) function to pass an array in tfidf: _this.tfidf(term, d) call (change to:
tfidf: _this.tfidf([term], d) (line 174 here: https://github.com/NaturalNode/natural/blob/master/lib/natural/tfidf/tfidf.js ).

Thanks.

[DSchmidlin]

I ran into the same problem. The workaround I used was to set a custom tokenizer that does this work.

tfidf.setTokenizer( { tokenize(x) { return [x] } });

[Hugo-ter-Doest]

Solved in #748