[FEA] Support for HDBSCAN membership_vector and all_points_membership_vectors

[beckernick]

I'd like to be able to use HDBSCAN to calculate membership vectors for points, like I can with the CPU library. Per the CPU library documentation, this function "produces a vector for each point in [provided data] that gives a probability that the given point is a member of a cluster for each of the selected clusters of the clusterer."

HDBSCAN provides separate methods to calculating membership vectors for both new points and all points in the original dataset

This is useful in topic modeling workflows. Per the BERTopic documentation, "Calculating the probabilities is quite expensive and can significantly increase the computation time. Thus, only use it if you do not mind waiting a bit before the model is done running or if you have less than 50_000 documents." Accelerating this would be valuable, as these probabilities are necessary for evaluating confidence of topic assignments to documents. The existing probabilities_ we provide represent the probability of the ultimately assigned topic cluster, but it would be nice to provide the full vector of probabilities per point like CPU HDBSCAN for compatibility with tools that depend on it (like BERTopic).

import hdbscan
from sklearn.datasets import make_blobs
import numpy as np


np.random.seed(12)

data, _ = make_blobs(1000)

clusterer = hdbscan.HDBSCAN(min_cluster_size=10, prediction_data=True)
clusterer.fit(data)

new_points = np.array([[-9,7], [-3,4]])
hdbscan.membership_vector(clusterer, new_points)
array([[0.02848655, 0.07827696, 0.46968066],
       [0.0569429 , 0.05463597, 0.03344189]])

import hdbscan
from sklearn.datasets import make_blobs
import numpy as np

np.random.seed(12)


data, _ = make_blobs(1000)
clusterer = hdbscan.HDBSCAN(min_cluster_size=10, prediction_data=True)
clusterer.fit(data)

hdbscan.all_points_membership_vectors(clusterer)[:5].round(3)
array([[0.   , 0.   , 1.   ],
       [0.667, 0.11 , 0.05 ],
       [0.697, 0.059, 0.024],
       [0.   , 1.   , 0.   ],
       [0.546, 0.177, 0.047]])

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[beckernick]

With the merge of #4800, soft clustering the original dataset with all_points_membership_vectors is now available. Please give it a try and file issues if you run into any issues or have any feedback.

The outstanding request in this issue is soft-clustering a new set of points with membership_vector.

[ZhangzihanGit]

Hi,

Thanks for implementing this feature!

I notice that the all_points_membership_vectors method is written outside of the HDBSCAN class, is this intended?

In that case, we need to explicitly import this method to use it:

from cuml.cluster import HDBSCAN, all_points_membership_vectors

Thanks

[beckernick]

Good question. Yes. In the canonical CPU library, this function is accessible from the module level namespace rather than as a method of the class. We've matched this user experience.

import cuml
from cuml.cluster import hdbscan
import hdbscan as cpu_hdbscan

print(cpu_hdbscan.all_points_membership_vectors)
print(cuml.cluster.hdbscan.all_points_membership_vectors, hdbscan.all_points_membership_vectors)
<function all_points_membership_vectors at 0x7fa895c611f0>
<cyfunction all_points_membership_vectors at 0x7fa897a35520> <cyfunction all_points_membership_vectors at 0x7fa897a35520>

[ldsands]

Good question. Yes. In the canonical CPU library, this function is accessible from the module level namespace rather than as a method of the class. We've matched this user experience.

import cuml
from cuml.cluster import hdbscan
import hdbscan as cpu_hdbscan

print(cpu_hdbscan.all_points_membership_vectors)
print(cuml.cluster.hdbscan.all_points_membership_vectors, hdbscan.all_points_membership_vectors)
<function all_points_membership_vectors at 0x7fa895c611f0>
<cyfunction all_points_membership_vectors at 0x7fa897a35520> <cyfunction all_points_membership_vectors at 0x7fa897a35520>

I just ran the nightly yesterday, and it didn't seem to work I had to use your workaround from here. Am I missing something?

# this doesn't work
probs = cuml.cluster.hdbscan.all_points_membership_vectors(gpu_hdbscan_model)
# this does work
probs = cuml.cluster.all_points_membership_vectors(gpu_hdbscan_model)

[beckernick]

Thanks for commenting about this namespace issue. This will be resolved in #4895

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[hatemr]

Thanks for implementing this feature!

Has anyone had issues using all_points_membership_vectors with a large dataset? (for me, original embeddings are 235002 x 384). It causes my Python kernel to fail. Apologies if this isn't enough detail.

[cjnolet]

@hatemr can you provide the error you are getting when the Python kernel fails and describe your environment (type of GPU, etc...)?

[hatemr]

g4dn.8xlarge

[beckernick]

Would recommend we take further discussion of this to #4879 , which includes some potential ideas for ways to reduce it.

I will comment on that issue with this context.

[ravinani]

@hatemr I tried the BERTopic with GPU for almost 1M documents. My original embeddings are 1M x 384. I tried to get the probabilities of each topic for evary document using "all_points_membership_vectors" but the system crashes whenver I tried to perform the task

I am using Colab with GPU of 40GB. Can you please suggest the workaround to get the probabilities of each document for every topic

[hatemr]

@hatemr I tried the BERTopic with GPU for almost 1M documents. My original embeddings are 1M x 384. I tried to get the probabilities of each topic for evary document using "all_points_membership_vectors" but the system crashes whenver I tried to perform the task

I am using Colab with GPU of 40GB. Can you please suggest the workaround to get the probabilities of each document for every topic

@ravinani see answer here