Ensure PCA initialization has small variance

[pavlin-policar]

• As per Normalize PCA initialization to have small variance #10, it is vital for tSNE optimization that the initialization has small variance. Without this scaling, PCA initialization can potentially lead to really bad results.

• Remove pca option when embedding new points into an existing embedding. This makes no sense because likely the points in the original space would have moved significantly and adding new points would be meaningless.

[dkobak]

Did you check if it makes a difference?

[pavlin-policar]

I did - it makes a huge difference. Using non-scaled PCA initialization the results were very bizzare-looking, but scaling it down makes it nicely behaved. I also compared it against random initialization and it does seem like PCA does better. Random initialization seems like it would need more iterations to reach the same cluster separation, but this may be specific to the 5 tests I ran. Would definitely need to run more tests to verify this.

I will be preparing a couple of notebooks showing usage and I'll probably make one for this specifically. I don't think this is well known. Thanks for pointing it out!