PUBDEV-6938 Test TF-IDF #4668
Closed
PUBDEV-6938 Test TF-IDF #4668
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Basic implemenation of term frequency part of MRTask trainer which (for now) relies on getting whole data in one chunk at once.
Add helper MR task to get a unique index for each word from data.
TFIDF trainer class implementation now supports words counts computation for multiple chunks of data.
Implementation of TermFrequency computation (MRTask) and unit test to test its functionality.
Implementation of TF-IDF data preprocessing (MRTask) and unit test to test its functionality.
Add class for computing IDF values and corresponding unit tests.
TermFrequency MRTask now uses Frames with GroupBy operations instead of HashMaps.
Reduce output data of TF-IDF preprocessor to provide only necessary data.
Refactor TF task to simply delegate its logic to GroupyBy MRTask.
Add class for computing DocumentFrequency values via GroupBy MRTask.
Refactor IDF task to simply delegate its logic to GroupyBy MRTask.
Fix DF task to work correctly when there are multiple same words in the documents. Adjust unit test to test this behaviour.
Join all partial tasks and create a single AstPrimtive to compute TF-IDF.
Add bigdata test for TermFrequency task.
Add bigdata test for DocumentFrequency task.
Add bigdatat test for TF-IDF.
Add Python API allowing to compute TF-IDF values via wrapper function. Add unit test to test this Python API.
Remove previous model implementations of TF-IDF. Improve unit tests by adding assert messages and removing code duplications.
Allow already preprocessed input for TF-IDF. Add corresponding unit tests.
Add TF-IDF demo notebook showing how to compute TF-IDF with and without preprocessing.
Fix TF-IDF parts using groupby (TF, DF) to use single groupby columns array as an input.
Fix key leaks in the code, add input parameters checks, change some parts according to best pracice and remove old unit tests.
Add parameter allowing to specify whether input data should be treated as case sensitive. Add unit tests to test this option. Adjust demo notebook to demonstrate this option.
Accept column names/indices defining which columns from given frames to use instead of requiring specific frame format. Reuse output from TF task in DF task.
Add TF-IDF documentation page explaining basics of TF-IDF and showing how it should be used.
…oved unused test, resolved memory leaks
maurever
force-pushed
the
maurever_PUBDEV-6938_test_tfidf
branch
from
e9e6cde
to
09ece4e
Compare
Fix Python test for Python 2.7 - frame construction from OrderedDict instead of regular dictionary.
Merged
maurever
force-pushed
the
maurever_PUBDEV-6938_test_tfidf
branch
from
4b4d4fd
to
445352f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.