added documentation #21
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| self.doc_freqs = [] # list of dictionaries of term_frequency of each document | ||
| self.idf = {} # idf score of each word in whole corpus | ||
| self.doc_len = [] # list of length of each document in corpus | ||
| self.tokenizer = tokenizer # user input tokenizer, defaults to none | ||
|
|
||
| if tokenizer: | ||
| corpus = self._tokenize_corpus(corpus) | ||
|
|
||
| nd = self._initialize(corpus) |
There was a problem hiding this comment.
What is nd here? You should explain it
Comment on lines
+38
to
+40
| Example: | ||
| corpus = [['ram', 'is', 'a', 'good', 'boy'], ['ram', 'does', 'cycling', 'and', 'racing'], ['ram', 'is', 'healthy'], ['rita', 'likes', 'shyam'], ['good', 'luck']] | ||
| nd = {'ram': 3, 'is': 2, 'a': 1, 'good': 2, 'boy': 1, 'does': 1, 'cycling': 1, 'and': 1, 'racing': 1, 'healthy': 1, 'rita': 1, 'likes': 1, 'shyam': 1, 'luck': 1} |
There was a problem hiding this comment.
Shorten the examples so that I don't need to scroll. The functionality can also be explained only using 2 items in the list.
| for document in corpus: | ||
| self.doc_len.append(len(document)) | ||
| num_doc += len(document) | ||
| num_words += len(document) # total number of words in whole corpus |
There was a problem hiding this comment.
The function of variable num_words has already been explained.
| frequencies = {} | ||
| term_frequencies = ( | ||
| {} | ||
| ) # term frequency of each word in a document........ changed frequencies to term_frequencies |
There was a problem hiding this comment.
You don't need to comment that you changed the name of variable. git keeps track of it.
Comment on lines
+53
to
+54
| if word not in term_frequencies: | ||
| term_frequencies[word] = 0 |
There was a problem hiding this comment.
This block of code can be removed by using defaultdict instead of the normal dictionary.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
added inline comments and docstrings to explain what the code is actually doing.