New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lda2vec preprocess not good support about memory #59

Open

hxsnow10 opened this issue Oct 10, 2017 · 1 comment

hxsnow10 commented Oct 10, 2017 •

edited

Loading

when i use 6G pure text， 3 threads, max_len=4W, they run out my 120Gmeory+180Gswap.

i guess may be some point unfriendly with big data.

hxsnow10 changed the title ~~not good support about memory~~ lda2vec preprocess not good support about memory

ghost commented Feb 17, 2019

I've encountered MemoryError several times and now have to fall back to a smaller amount of data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment