Explore more clustering algs

[woodthom2]

Description

We are currently gravitating towards Kmeans as the default clustering alg. It's fast. Main disadvantage is it's non deterministic

Can we look into a couple of other algorithms and maybe offer them as options

• Umap
• HDBScan

We need to check for

• quality of the clusters that Harmony generates
• speed
• deterministic behaviour - do you get the same results for multiple calls to the alg

Rationale

We want Harmony to have the best possible clusters

[montygole]

Here are the results of some preliminary testing I've done with HDBSCAN. Please let me know if you would like any more details :).

tl;dr: I tested HDBSCAN's performance with three different approaches: UMAP, t-SNE, and no dimension reduction. UMAP produced the best results in terms of silhouette score. We should consider testing additional metrics for a more comprehensive evaluation.

I tested how HDBSCAN performs with dimension reduction techniques, specificlaly UMAP, t-SNE, and no reduction. I tested using the example data seen in the example walkthrough notebook. I outputted clusters on a scatter plot (seen below), and calculated their silhouette scores. Embedding dimensions are reduced to 2.

HDBSCAN without dimension reduction (Silhouette score: 0.25)

• -1 cluster is noise

HDBSCAN with UMAP dimension reduction (Silhouette score: 0.32)

HDBSCAN with t-SNE dimension reduction (Silhouette score: 0.3)

Each technique yields the same results when run multiple times (with the same random seed).

Conclusion

With respect to silhouette score, HDBSCAN with UMAP outperforms the other methods tested as well as K-means with 5 clusters (see the colab notebook for the K-means results). However, we can also test with more clustering performasnce metrics (a summary of more metrics can be found here)

[woodthom2]

Hi @montygole , thanks so much! We have just merged a PR from @AlexNikic who added kmeans and affinity propagation - it's now in main and you can see where we branch the code based on clustering algs here: https://github.com/harmonydata/harmony/blob/main/src/harmony/matching/matcher.py#L687

The enum listing the clustering algs is here: https://github.com/harmonydata/harmony/blob/main/src/harmony/schemas/enums/clustering_algorithms.py
We have hdbscan but we haven't done anything with it.

Do you want to try adding your suggestion to this as a PR? Feel free to contact me on Discord if you need a hand or some guidance, my name is Thomas Wood and the Discord is here: https://discord.gg/harmonydata

[montygole]

Hi @woodthom2 . Okay sounds good! I'll create a PR! Okay I'll say hi in the discord :)