Text 0.4.0 countvectorizer

Lodestar.Text 0.4.0. This page is frozen at that release. Read the current documentation for what main says now. A link to a decision or a migration page follows main, and leaves the archive.

CountVectorizer

Term counts over a vocabulary learned from the corpus — the equivalent of sklearn.feature_extraction.text.CountVectorizer.

Fitting learns which terms exist and fixes their column order; transforming counts them. Column 7 means the same term in every row, and GetFeatureNames can say which.

public sealed class CountVectorizer

Constructor — CountVectorizer(CountVectorizerOptions? options = null). The default options are scikit-learn's defaults, so new CountVectorizer() is CountVectorizer().

Example — three documents, and the vocabulary they produce.

using Lodestar.Text.Vectorization;

string[] docs = ["the cat eats", "the dog eats", "the cat and the dog"];

var cv = new CountVectorizer();
CsrMatrix counts = cv.FitTransform(docs);

int features = counts.ColumnCount;  // => 5

Remarks — the vocabulary is sorted, as scikit-learn's is, so the column order depends only on the terms and not on the order the documents arrived in. Two fits over the same corpus give the same matrix, and a corpus shuffled before fitting gives the same matrix too.

What decides the counts is CountVectorizerOptions rather than anything here — the token pattern that drops single-letter words, the stop words, the n-gram range. This class only applies them.

To weight rare terms above common ones, TfidfVectorizer is this followed by TfidfTransformer. To skip the vocabulary entirely, HashingVectorizer trades naming for not having to hold one.

Applies to — net10.0, netstandard2.0.

See also — CountVectorizerOptions, CsrMatrix, TfidfVectorizer, the vectorization guide.

Members

Member	What it does
CountVectorizer.Fit	Learn the vocabulary, and return the same instance.
CountVectorizer.FitTransform	Learn it and count the same corpus in one pass.
CountVectorizer.GetFeatureNames	The term each column stands for, in column order.
CountVectorizer.Load	Read a fitted vectorizer back.
CountVectorizer.LoadAsync	The same, without blocking.
CountVectorizer.Save	Write a fitted vectorizer out.
CountVectorizer.SaveAsync	The same, without blocking.
CountVectorizer.Transform	Count a corpus against the vocabulary already learned.