-
Notifications
You must be signed in to change notification settings - Fork 497
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixed the bug that joblib uses memory mapping when data size is too large #495
Conversation
Thanks for the PR. Hopefully this cleans up the problems. |
Can you explain how do you set the max_nbytes to None? Do you need to call joblib.Parallel or is there any other way? |
We also hit this issue and are currently using an git commit to fetch a working version of HDBSCAN. Could you consider releasing a new version of HDBSCAN with this fix? |
@ginward thanks for that, liked the way you documented everything - helped me to find these issues. Although I tried reinstalling HDBSCAN from git master with your fix, I still struggled with BERTopic with large amount of docs (2.9M) having the same error you had.
|
@yotammarton Maybe you can try cloning from the master branch, and check manually if the source code here and here has been updated, before installing directly from the clone?
Also note that if you are working on a Mac, you might need to change multiprocessing from Setting the number of jobs to 1 might fix the issue, but it might slow down the code, too. |
My steps were:
I was unable to locate the source code after installing the package so couldn't verify the change. Im on Ubuntu 18.04.5 LTS (GNU/Linux 5.3.0-1032-aws x86_64) |
…ternals using joblib. 🔥🐛 This is a somewhat known error as similar messages have been discussed [here](scikit-learn/scikit-learn#21685) and on the [hdbscan GH pull-#495](scikit-learn-contrib/hdbscan#495). The error messages is emitted from joblib.externals.loky.process_executor._RemoteTraceback and emits a ValueError: 'ValueError: buffer source array is read-only.' So far this has not been encountered with scikit-learn version 0.24
I got the same error. I tried uninstalling hdbscan and reinstall hdbscan from the patch (as per instructions above), but it didn't help. I am running Anaconda Python Python 3.8.5 on OS X Monterey 12.3.
|
After adjusting 'core_dist_n_jobs=1 in hdbscan.HDBSCAN,
|
@dbl001 Is it possible that your machine does not have enough memory RAM? |
It’s an iMac 27” with 128gb DRAM.
… On Apr 16, 2022, at 11:40 PM, Jinhua Wang ***@***.***> wrote:
@dbl001 <https://github.com/dbl001> Is it possible that your machine does not have enough memory RAM?
—
Reply to this email directly, view it on GitHub <#495 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXWFW47LLPFE7VXFGSJH63VFOW4PANCNFSM5FHMFFXQ>.
You are receiving this because you were mentioned.
|
This is to fix the bug mentioned in the following issues:
joblib/joblib#1225
scikit-learn/scikit-learn#21228
#494
MaartenGr/BERTopic#151
It turns out that joblib will use memoryviews in multiprocessing if the data size is beyond max_nbytes. However, this creates error in Cython, as discussed here: scikit-learn/scikit-learn#7981 (comment)
Therefore, I have set the max_nbytes to none to prevent the auto-memmapping, solving the above errors. The downside is that now the memory usage could be higher if you use multi-processing, as the numpy array will need to be copied into the new process.