Feature request: phik correlation metrics

[gokceneraslan]

It'd be great to add other correlations as distances, such as Spearman's rho or the new correlation coefficient called phi_k. I am aware that it's possible to use any distance metric using either metric='precomputed' or implementing a custom function, but it'd be more convenient to pass metric='spearmanrho' or 'phik' directly.

[sleighsoft]

You could contribute to scipy upstream and then everyone would be able to use these metrics, not only in UMAP. Under the hood of UMAP mostly from sklearn.metrics import pairwise_distances is used with some special cases for sparse matrices.

[gokceneraslan]

@sleighsoft thanks for the reply. Actually, scipy already has a Spearman's rho implementation but since it's not defined as a scipy distance it's not accessible from scikit-learn's pairwise_distances.

Even if it were available as a distance, it would be only useful for small datasets since, UMAP uses pairwise scikit-learn distances only if the dataset is size is small (n<4096) (and if kNN approximation is not forced by the user) (see this and that). In all other cases, it falls back to UMAP's own distances.py file where distances are implemented with numba's JIT support.

So, what I'm proposing is to implement spearman (and maybe phi_k too, if anyone is willing to) in distances.py as a "native distance" in UMAP (just like Pearson correlation and many other distance metrics that are implemented from scratch although they are available in scikit-learn/scipy) so that we can use it with both small matrices in a pairwise fashion and the large matrices via approximated kNN. I hope it's clear.

[sleighsoft]

Oh I see! Thanks for clarifying the issue.
It is probably a good thing that you pinged the scipy people.

Btw, I do not find pearson in the distances.py. Is it named correlation instead?

[sleighsoft]

@gokceneraslan I look at this again and I believe the best place for this would actually be the pynndescent project https://github.com/lmcinnes/pynndescent/blob/master/pynndescent/distances.py

It has the same number of metrics as UMAP and I assume it will be the default for UMAP.

[sleighsoft]

Added a PR.