Text vectorizers

Development build. This page describes main, not a released package. The latest published Lodestar.Text is 0.4.0 — read its documentation.

Vectorization — Lodestar.Text

You have a corpus of documents and you need numbers. Lodestar.Text.Vectorization reproduces sklearn.feature_extraction.text: it turns text into a sparse matrix of features, one row per document and one column per term.

Three vectorizers do that, and what separates them is what they need to know about the corpus first.

Which vectorizer?

flowchart TD
    A["What do you need?"] --> B{"Do rare words<br/>deserve more weight<br/>than common ones?"}
    B -->|"yes — this is search,<br/>or a classifier"| C["TfidfVectorizer"]
    B -->|"no, raw occurrence counts"| D{"Can you afford to hold<br/>the vocabulary in memory?"}
    D -->|yes| E["CountVectorizer"]
    D -->|"no — the corpus is a stream,<br/>or the vocabulary is unbounded"| F["HashingVectorizer"]
    A --> G{"Already have counts<br/>from somewhere else?"}
    G -->|yes| H["TfidfTransformer"]

Loading

CountVectorizer counts. It learns the vocabulary from the corpus during Fit, so column 7 means the same term in every row and GetFeatureNames can tell you which.

TfidfVectorizer counts and then weights each count by how rare the term is across the corpus. It is CountVectorizer followed by TfidfTransformer, and doing it in one step is the only difference.

HashingVectorizer does not learn anything. It hashes each term into one of a fixed number of columns, so it needs no Fit, holds no vocabulary, and works on a stream — at the price that two terms can collide in one column and no GetFeatureNames exists, because there is no vocabulary to name. That trade is the whole reason to choose it.

The three that need fitting, and the one that does not

Fit is where a vocabulary is learned, and it is why the order of calls matters:

Call	What it does
Fit(corpus)	Learn the vocabulary and, for TF-IDF, the document frequencies.
Transform(corpus)	Use what was learned. Refuses if nothing was.
FitTransform(corpus)	Both, on the same corpus — and not the same as Fit then Transform on different ones.

Transforming a document that holds a term the fit never saw drops that term silently: it has no column. That is scikit-learn's behaviour, and it is why fitting on the training corpus and transforming the test one is the correct order rather than a convenience.

HashingVectorizer has no Fit at all, and its Transform and FitTransform do the same thing — the second exists so the three vectorizers can be swapped for one another.

The matrix they return

Every one of them returns a CsrMatrix: compressed sparse row, the same layout scipy.sparse.csr_matrix uses. A corpus of ten thousand documents over fifty thousand terms is almost entirely zeros, and storing those zeros is what this layout exists to avoid.

Read ToDense only when you mean it: it allocates RowCount × ColumnCount doubles, which is exactly the array the sparse layout was avoiding.

The options carry the parity

Most of what makes these match scikit-learn lives in the three options records rather than in the vectorizers: CountVectorizerOptions, TfidfOptions and HashingVectorizerOptions. Their defaults are scikit-learn's defaults, and each property's page entry says which Python keyword it answers to.

Two defaults surprise people, and both are scikit-learn's:

• the token pattern is \b\w\w+\b, so single-letter words are dropped — "a" and "I" never become features;
• Lowercase is on, so Apple and apple are one term.

Types

Type	What it is
AnalyzerKind	Whether features are words or character n-grams.
CountVectorizer	Term counts, over a vocabulary learned from the corpus.
CountVectorizerOptions	Everything that decides what counts as a term.
CsrMatrix	The compressed-sparse-row matrix every vectorizer returns.
HashingVectorizer	Counts into a fixed number of columns, learning nothing.
HashingVectorizerOptions	The column count, and what the hashing does with signs.
SparseNorm	Which norm CsrMatrix.NormalizeRows divides each row by.
StopWords	The six built-in stop-word lists.
TfidfOptions	The four switches that decide how the weighting is computed.
TfidfTransformer	Counts in, TF-IDF weights out.
TfidfVectorizer	CountVectorizer and TfidfTransformer in one pass.
TfidfVectorizerOptions	The two halves above, as one options object.

See also

• From string to vector — the guide, which walks a corpus through all three rather than describing them one at a time.
• Python → C# equivalence — every sklearn.feature_extraction.text call and its counterpart here.