RuntimeWarning: overflow encountered in exp2 topicmodeler.train

[dbonner]

Code:
trainclassdict = shorttext.data.nihreports(sample_size=None)
topicmodeler = shorttext.generators.LDAModeler()
topicmodeler.train(trainclassdict, 128)
Error message:
/lib/python2.7/site-packages/gensim/models/ldamodel.py:535: RuntimeWarning: overflow encountered in exp2
perwordbound, np.exp2(-perwordbound), len(chunk), corpus_words

Then the results are variable for
topicmodeler.retrieve_topicvec('stem cell research')

[stephenhky]

try topicmodeler.train(trainclassdict, 64)
128 is too much.

i will update the documentation

[dbonner]

Hi,
I tried:
topicmodeler.train(trainclassdict, 64)
I still get exactly the same error message.
By the way, there is no error message using Theano with 64 or 128.

[stephenhky]

This is weird. topic modeling should have nothing to do with theano and tensorflow as I am just invoking gensim. however, gensim itself uses keras, which require either of them. maybe they updated the source code.... weird

[stephenhky]

https://github.com/RaRe-Technologies/gensim/blob/300ce8c2ac56133972785c616db98bec29c11e8c/gensim/models/ldamodel.py#L535

In the process of training, gensim calculated the log (of base 2) of the perplexity. The computation to our purpose does not have overflow/underflow problem. But this line writes to the log, which causes this warning message.

But why this warning appears in tensorflow but not theano is a myth.

So all in all, this warning message can be ignored for the moment. However, we can report this to gensim.

[stephenhky]

Reported here: piskvorky/gensim#1692

[stephenhky]

See this: piskvorky/gensim#1692