Improving computational time in LDASeqModel

[Diego999]

Hi,

I was training a LDASeqModel on ~40k documents through 20 years (20 time slices in my case) and after 2 days of computation, still didn't finish.

I realized that only LDAModel is used in this class. Shouldn't be better to use LDAMulticore since it can also handle the case of single core ?

Thanks

[xelez]

Tried substituting LdaModel with LdaMulticore only to found that it doesn't change perfomance at all.

Started investigating, found several interesting things:

model = "DTM"
if model == "DTM":

2. LdaModel is used in:

• in __init__() to initialize DTM.. but you can train LDA yourself with LdaMulticore
• other places

It seems, that in other places it is used just as Data Transfer Object, while all calculations is mostly done inside LdaPost.

Although I can submit merge request that allows to choose between LdaModel and LdaMulticore, it's basically useless.

[mod-cpu]

Hi!

I may have a similar problem, but I don't know enough to be sure. I'm trying to run LDASeq over 221,261 short (reddit comment) documents. The model itself only takes about an hour or so to complete 10 passes on 5 topics, with 6 time slices, but it stalls after the passes are complete?

Is this model only appropriate for much smaller corpuses?

The model stalls after printing the lines below for reference:
2017-12-17 13:58:02,579 : INFO : E Step
2017-12-17 13:58:02,639 : INFO : using symmetric eta at 7.271933970839545e-06
2017-12-17 13:58:02,662 : INFO : using serial LDA version on this node

Thanks!

[menshikh-iv]

@Diego999 so, probably your Dictionary is very large, can you reduce the size of the dictionary and try again (based on log messages, it is still being initialized)

[ebergam]

Hi!
I have the same problem in scaling up ldaseq to large amounts of data.
Is there any update on this issue or another approach I should be considering?

Thank you!

[asaiyed10]

@matthewbaykenney Were you able to solve this by chance?

Having the same issue, with 200,000 documents and 50,000 words. I've tried topics ranging from 10-100, and set the em_min_iter to 0, with no luck.

[bensilver95]

I am having a similar issue! I am running ldaseq on millions of tweets, and it's taking quite a long time to finish running. If anyone has any advice or tips (such as how to reduce the dictionary size @menshikh-iv), that would be helpful.

Also, how do you get the model to print out logging information?

[nikchha]

I have a similar issue. Trying to run ldaseqmodel on 53,000 documents with 12,099 tokens remaining after preprocessing and 18 time slices. Further, I set the number of topics to 30. I used the wrapper class for the original DTM binary before, and it took merely 6 hours to complete. However, I would like to use the doc_topics method of the ldaseqmodel to find document clusters for each topic.

Is there a way to speed up training? Using the ldaseqmodel takes 10 hours just for the first EM iteration to complete.

Thank you!