Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gensim LdaMulticore can't work on cloud function #3422

Closed
tinac5519 opened this issue Jan 6, 2023 · 2 comments
Closed

Gensim LdaMulticore can't work on cloud function #3422

tinac5519 opened this issue Jan 6, 2023 · 2 comments

Comments

@tinac5519
Copy link

Problem description

I want to use gensim LDA module on cloud function, but it time out and show "/layers/google.python.pip/pip/lib/python3.8/site-packages/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses".

But the same code worked on colab (python 3.8.16) and I did't find any bug in it. It can print 'LDA1' and 'LDA2', then time out.

Steps/code/corpus to reproduce

1.I have tried diffierent python version like 3.10, 3.8, 3.7

2.ADD import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)

3.It works on colab and 300 text just cost 10 sec, but I need it work on cloud function

def LDA(corpus, dictionary, NumTopic):
    print('LDA1')
    time1 = time.time()
    print('LDA2')
    lda = gensim.models.LdaMulticore(corpus=corpus, id2word=dictionary, num_topics=NumTopic,  chunksize=1000, iterations=200, passes=20, per_word_topics=False,  random_state=100)
    print('LDA3')
    corpus_lda = lda[corpus] 
    print("LDA takes %2.2f seconds." % (time.time() - time1))
    return lda, corpus_lda

Versions

Please provide the output of:

from __future__ import unicode_literals
import base64
import importlib
import re
import os
import sys
import numpy as np
import pandas as pd
import gensim
import gensim.corpora as corpora
from gensim.utils import simple_preprocess
from gensim.models import CoherenceModel
from gensim import corpora, models, similarities
from google.cloud import bigquery
import pandas_gbq
import requests
import tqdm
import json
import pyLDAvis
import pyLDAvis.gensim_models
import matplotlib.pyplot as plt
import logging
import time
@piskvorky
Copy link
Owner

piskvorky commented Jan 6, 2023

Isn't that a question for the "cloud function" (whatever that is) software / service / support?

I don't see anything in the traceback to suggest this is a Gensim issue.

@gojomo
Copy link
Collaborator

gojomo commented Jan 10, 2023

Yes, seems like a limit in someone else's (Google's) peculiar service. I made a few followup Qs/suggestions to the same question posted at SO to help troubleshoot – https://stackoverflow.com/questions/75029088/gensim-ldamulticore-cant-work-on-cloud-function – with no responses yet. But I think this issue can & should be closed as no bug in Gensim.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants