Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak on corrector.LoadLangModel() #44

Open
rominf opened this issue Aug 22, 2018 · 3 comments
Open

Memory leak on corrector.LoadLangModel() #44

rominf opened this issue Aug 22, 2018 · 3 comments

Comments

@rominf
Copy link

rominf commented Aug 22, 2018

I've trained my model (I've tried versions from master and 0.0.11 branches) on 10 MiB plain text part of English Wikipedia (enwiki-latest-pages-articles_10MiB.txt) and got 41 MiB bin file (enwiki.bin.zip).

I'm loading it in Python, but it takes 12 GiB of memory to load it and still it doesn't load in foreseeable time.

@xdingo93x
Copy link

I'm facing the same issue. I've trained a (german) model using around 560MiB plain text from the Leipzig Corpora Collection. The model itself is 488MiB. Having a 16GB RAM 4CPU Linux Cloud, it takes 5-10 min to load the model. Is it possible to speed this up?

@mshakirDr
Copy link

March 16, 2022
I tried to use this on Windows 10, Python 3.9.7 but it keeps eating memory and nothing happens. Unable to load the 34 MB English model.

@bakwc
Copy link
Owner

bakwc commented Mar 16, 2022

Community version doesn't support loading linux models on windows.
You should either use a linux or train your own model on windows.

Or you can buy a PRO version, it supports all models on all operation systems. Also a PRO version reduced memory usage while training.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants