How to input tokenized sentences into sklearn.feature_extraction.text.TfidfVectorizer

[DachuanZhao]

I find the example in the document :

from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
    'This is the first document.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)

But in my real NLP project , I have tokenized sentences in my special way which means corpus is a corpus_list like this :

from sklearn.feature_extraction.text import TfidfVectorizer
corpus_list = [
    ['This', 'is', 'the', 'first', 'document', '.'],
    ['This', 'document', 'is', 'the', 'second', 'document', '.'],
    ['And', 'this', 'is', 'the', 'third', 'one', '.'],
    ['Is', 'this', 'the', 'first', 'document', '?']
]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus_list)

I think TfidfVectorizer should support this input ,because we often use a complex tokenize strategy .

[jnothman]

You're asking a usage question. The issue tracker is mainly for bugs and new features. For usage questions, it is recommended to try Stack Overflow or the Mailing List.

The answer, however is:

from sklearn.feature_extraction.text import TfidfVectorizer
corpus_list = [
    ['This', 'is', 'the', 'first', 'document', '.'],
    ['This', 'document', 'is', 'the', 'second', 'document', '.'],
    ['And', 'this', 'is', 'the', 'third', 'one', '.'],
    ['Is', 'this', 'the', 'first', 'document', '?']
]

def identity(x):
    return x

vectorizer = TfidfVectorizer(analyzer=identity)
X = vectorizer.fit_transform(corpus_list)

[DachuanZhao]

Thanks a lot . I think the answer should be written in the document . How can I contribute it ?

[jnothman]

Thanks @DachuanZhao, perhaps you can find a place to mention it in https://github.com/scikit-learn/scikit-learn/blob/master/doc/modules/feature_extraction.rst

[jnothman]

A similar statement is already there: "If documents are pre-tokenized by an external package, then store them in files (or strings) with the tokens separated by whitespace and pass analyzer=str.split"

[DachuanZhao]

A similar statement is already there: "If documents are pre-tokenized by an external package, then store them in files (or strings) with the tokens separated by whitespace and pass analyzer=str.split"

I find the document at https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py#L1705

    Examples
    --------
    >>> from sklearn.feature_extraction.text import TfidfVectorizer
    >>> corpus = [
    ...     'This is the first document.',
    ...     'This document is the second document.',
    ...     'And this is the third one.',
    ...     'Is this the first document?',
    ... ]
    >>> vectorizer = TfidfVectorizer()
    >>> X = vectorizer.fit_transform(corpus)
    >>> print(vectorizer.get_feature_names())
    ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']
    >>> print(X.shape)
    (4, 9)
    """

Can it be modified like this ? :

    Examples
    --------
    >>> from sklearn.feature_extraction.text import TfidfVectorizer
    >>> corpus = [
    ...     'This is the first document.',
    ...     'This document is the second document.',
    ...     'And this is the third one.',
    ...     'Is this the first document?',
    ... ]
    >>> vectorizer = TfidfVectorizer()
    >>> X = vectorizer.fit_transform(corpus)

    #You can also init X with a pre-tokenized data :
    >>> corpus_list = [
    ...     ['this', 'is', 'the', 'first', 'document'],
    ...     ['this', 'document', 'is', 'the', 'second', 'document'],
    ...     ['and', 'this', 'is', 'the', 'third', 'one'],
    ...     ['is', 'this', 'the', 'first', 'document']
    ... ]
    >>> vectorizer = TfidfVectorizer(analyzer=lambda x:x)
    >>> X = vectorizer.fit_transform(corpus_list)
    
    >>> print(vectorizer.get_feature_names())
    ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']
    >>> print(X.shape)
    (4, 9)
    """

[jnothman]

I'd rather not make that example so long. There are lots of variant invocations that could be covered there, and I'd rather it cover a main usage, and more details be left for the user guide and long-form examples.