Create local random generator for sample_text & add lenght #1422
Conversation
gensim/test/test_textcorpus.py
Outdated
| class TestTextCorpus(TextCorpus): | ||
| def __init__(self): | ||
| self.data = [["document1"], ["document2"]] | ||
| class DumyTextCorpus(TextCorpus): |
gensim/corpora/textcorpus.py
Outdated
| if n != 0: | ||
| # This means that length was set to be smaller than nuber of items in stream. | ||
| raise ValueError("length smaller than number of documents in stream") | ||
|
|
There was a problem hiding this comment.
Isn't this actually the opposite: that the generator has more elements than was declared?
Even so, I doubt this case should be a thrown-exception that must be handled. The user got their n docs, within the length declared – which might be exactly what they wanted/needed. So perhaps just a logged warning – "documents in excess of declared 'length' not considered for sampling".
There was a problem hiding this comment.
It should be: length was set to be greater than number of items in stream indeed.
It throws an exception because user received less than n docs. Note that if length is greater than actual length of the corpus, it will not trigger the exception on its own. The exception triggers when we still need to sample some documents, but the stream run out. I add it to the comment.
The case when length was smaller than the actual corpus length does not throw anything right now (the problem was only in the documentation). Printing a warning is awkward, because we would have to actually go through all of the length elements which is unnecessary (we may sample enough elements earlier).
gensim/corpora/textcorpus.py
Outdated
| """ | ||
| Yield n random texts from the corpus without replacement. | ||
| Yields n random documents from the corpus without replacement. |
There was a problem hiding this comment.
"Yields" => "Yield" (imperative mode).
gensim/corpora/textcorpus.py
Outdated
|
|
||
| Given the the number of remaingin elements in stream is remaining and we need | ||
| to choose n elements, the probability for current element to be chosen is n/remaining. | ||
| Given the number of remaining documents in corpus, we need to choose n elements. |
gensim/corpora/textcorpus.py
Outdated
| Given the the number of remaingin elements in stream is remaining and we need | ||
| to choose n elements, the probability for current element to be chosen is n/remaining. | ||
| Given the number of remaining documents in corpus, we need to choose n elements. | ||
| The probability for current element to be chosen is n/remaining. |
gensim/corpora/textcorpus.py
Outdated
| If we choose it, we just decreese the n and move to the next element. | ||
| Computing corpus length may be a costly operation so you can use optional paramter |
There was a problem hiding this comment.
"the corpus length"
"the optional parameter `length` instead"
gensim/corpora/textcorpus.py
Outdated
| Args: | ||
| n (int): number of documents we want to sample. | ||
| seed (int|None): if specified, use it as a seed for local random generator. | ||
| length (int|None): if specified, use it as guess of corpus length. |
gensim/corpora/textcorpus.py
Outdated
| length (int|None): if specified, use it as guess of corpus length. | ||
|
|
||
| Yeilds: | ||
| list[str]: document represented as list of tokens. See get_texts method. |
gensim/corpora/textcorpus.py
Outdated
| list[str]: document represented as list of tokens. See get_texts method. | ||
|
|
||
| Raises: | ||
| ValueError: then n is invalid or length was set incorrectly. |
There was a problem hiding this comment.
"then" => "when"?
Also, what does "incorrectly" mean? Above we say length is "a guess", so how can it be incorrect?
There was a problem hiding this comment.
I add some clarification to length parameter description.
gensim/corpora/textcorpus.py
Outdated
| length (int|None): if specified, use it as a guess of corpus length. | ||
| It must be positive and not greater than actual corpus length. | ||
|
|
||
| Yeilds: |
There was a problem hiding this comment.
*Yields. Spell checker can be useful.
There was a problem hiding this comment.
Ok.
I was wandering that I should finally install it (F6 in sublime).
|
Thank you @vlejd |
| if n != 0: | ||
| # This means that length was set to be greater than number of items in corpus | ||
| # and we were not able to sample enough documents before the stream ended. | ||
| raise ValueError("length greater than number of documents in corpus") |
There was a problem hiding this comment.
Can you make the message more concrete? Include the actual lengths.
|
|
||
| if not n <= length: | ||
| raise ValueError("n is larger than length of corpus.") |
There was a problem hiding this comment.
Can you make the message more specific? Include the actual lengths.
* Create local random generator for sample_text & add lenght * Fix typos
Changes regarding @gojomo comments at #308 . Add option to use local random and option to tell the function how long is the corpus (without running len and going through the whoe corpus).