Custom losses, coherent embeddings

[znah]

Nice property of TSNE, that is not exploited in most of implementations, is that it can be treated as a combination of two orthogonal components: loss function and optimization algorithm. For example one may visualize set on temporally varying vectors with a sequence on coherent embeddings by adding a loss term that penalizes unnecessary movement of each vector between those embeddings. Is it possible to have provide such flexibility to use additional constrains with UMAP?

[lmcinnes]

It is a little hard to do that and still maintain efficiency; potentially it could be added as an extra code-path on the side that is slower but more flexible. That would be a more significant project however.

[znah]

Thank you for the prompt reply!
I'm actually looking forward seeing your write-up about UMAP algorithm, hoping to reimplement it in a flexible way. For example, the most naive implementation of tsne loss with TensorFlow boils down to something like this:

def tsne_kl_loss(points, P):
    n = tf.shape(points)[0]
    Q = 1.0 / (1.0 + pdist2(points))
    sQ = tf.reduce_sum(Q) - tf.cast(n, tf.float32)
    return tf.reduce_sum(P*tf.log(P/Q)) + tf.log(sQ)

Then one can combine this loss with others and use one of standard optimizers.
Do you think if this kind of approach can be adopted to UMAP?

[lmcinnes]

At that level, yes almost certainly; if you are willing to do N^2 work then you can certainly have custom loss -- I was generally seeking to avoid that. On that front you might be interested by smallvis which implements t-SNE, LargeVis and UMAP in a common framework which I suspect would easily be adaptable to custom loss functions. The cathc is that it only supports small datasets for exactly the reason cited above. As a way to experiment, however, it is quite powerful.

[znah]

Thank you for pointing me to smallvis! I think that's exactly what I needed.
N^2 can actually go surprisingly far with efficient GPU implementation. For example, here is a random youtube video showing n^2 nbody with 60k particles at 30fps.

Still, I like to think of algorithmic optimizations, like employing BH or something else as yet another, partially orthogonal component.

[vanhoan310]

How to obtain the matrix P from UMAP? Is it self.graph_ from fuzzy_simplicial_set function?

Thanks!

[lmcinnes]

That is the equivalent of it, yes.