cuML UMAP is great, but not capable of producing consistently reproduceable results :(

[drob-xx]

I've been using cuML UMAP on Colab and the speedup is incredible. However, this comes at the expense of reproducibility. Per the cuML doc -

Note: Unfortunately, achieving a high amount of parallelism during the optimization stage often comes at the expense of determinism, since many floating-point additions are being made in parallel without a deterministic ordering. This causes slightly different results across training sessions, even when the same seed is used for random number generation

So that:

from cuml.manifold import UMAP
embeddings = <place your embeddings here>
umap_params = {'n_components': 5, 
               'n_neighbors': 15,
               'min_dist': 0.0,
               'random_state': 42,
               'metric': 'cosine'
               }
umap1 = UMAP(**umap_params)
umap2 = UMAP(**umap_params)
umap1.fit(embeddings)
umap2.fit(embeddings)
np.all(umap1.embedding_ == umap2.embedding_) == False

While they talk about "slightly different results" "..allowing for reproducible results to 3 digits of precision" that precision is not enough (in my testing so far) to give even marginally reproduceable results when trying to match HDBSCAN params to a given UMAP reduction. Sad.

[drob-xx]

So there is a workaround - for a discussion see the cuML bug.

By setting init='random' as well as random_state the results will be the same over multiple runs. There are some other potential issues with running init='random' but it seems like a reasonable work-around if you need it for now. Also, the random_state value is supposed to be a np.uint64 - if it isn't the value passed in will be discarded a random uint64 generated instead :).

[drob-xx]

Adding another gotchya issue with cuML UMAP. If you rely on embedding_ to cache the embeddings then set hash_input=True so that embedding_ and the values returned by transform() will be the same assuming the same input vectors.