Text tfidfvectorizer fit

TfidfVectorizer.Fit

Learn the vocabulary and the document frequencies from a corpus.

public TfidfVectorizer Fit(IEnumerable<string> documents)

Parameters — documents is the corpus to learn from, enumerated once.

Returns — TfidfVectorizer, the same instance, so a call can be chained.

Exceptions — ArgumentNullException when documents is null. A corpus that leaves no terms does not throw: it yields a model of zero columns, which every later transform will produce empty rows against.

Example — fit on training documents, weight a later one with those frequencies.

using Lodestar.Text.Vectorization;

var tv = new TfidfVectorizer();
tv.Fit(["the cat eats", "the dog eats"]);

CsrMatrix weighted = tv.Transform(["the cat sleeps"]);
int columns = weighted.ColumnCount;  // => 4

Remarks — two things are learned here where CountVectorizer.Fit learns one: the vocabulary, and the document frequency of each term in it. Both come from this corpus, which is what makes fitting on training data the correct order — a term's rarity is a property of the corpus it was measured on, and measuring it on the test set leaks information that will not exist at prediction time.

sleeps was never seen and is dropped, exactly as it would be by a count vectorizer.

Applies to — net10.0, netstandard2.0.

See also — TfidfVectorizer.Transform, TfidfVectorizer.FitTransform.