Add tfidf retriever parameters

[go5paopao]

I would like to add some new features to the TfidfRetriever in the retriever library.

The TfidfRetriever currently uses TfidfVectorizer from scikit-learn, which has many optional parameters that can affect the result of the tfidf and retrieval processes.

For instance, if we want to use the TfidfRetriever with a different language, we may need to add an original tokenization process. In the case of Japanese, we need to pass tokenizer parameters to TfidfVectorizer as shown below:

import MeCab
from sklearn.feature_extraction.text import TfidfVectorizer

def mecab_tokenizer(text):
    mecab = MeCab.Tagger("-Owakati")
    return mecab.parse(text).split()

vectorizer = TfidfVectorizer(tokenizer=mecab_tokenizer)

I have submitted this pull request so that we can support this feature.

In addition, I have added a simple unit test code. Since there was no TfidfRetriever test code previously, I have created a new file.

This is my first time submitting a pull request, so if there is anything insufficient or incorrect, please let me know.

[dev2049]

tfidf_params = tfidf_params or {}
vectorizer = TfidfVectorizer(**tfidf_params)

[go5paopao]

@dev2049
Thank you for review comment.
It is smart way, I just updated!

[dev2049]

looks great, thanks @go5paopao!

[go5paopao]

I just fix import error of scikit-learn by adding scikit-learn library to poetry.
I think it is currently resolved.