# scikit-learn/scikit-learn

### Subversion checkout URL

You can clone with
or
.

# Updated K-means clustering for Nystroem#3126

Open
wants to merge 4 commits into from
+221 −70

### 6 participants

Because I wanted to try K-means clustering as the basis for Nystroem approximation and it appeared as though pull request #2591 might be stalled I created a slightly modified version. I also tried to address @amueller comment about the effectiveness of the method by including it in the plot_kernel_approximation example and @dougalsutherland comment concerning the possible singularity of the sub-sampled kernel matrix using the same approach as scipy does in pinv2.

Since it is my first commit to the project (hopefully the first of many) any feedback or suggestions you have would be appreciated.

 nateyoder Add k-means clustering to Nystroem kernel approximation method; and i… …mplement it in plot_kernel_approximation example to show difference e4aed09 nateyoder Deal with kernel matrix singularity in Nystroem kernel approximation 0b139b4

Coverage remained the same when pulling 0b139b4 on nateyoder:kmeans-nystroem into 48e2b13 on scikit-learn:master.

 nateyoder Fix error message formatting issue on Python 2.6 e7bec1e
changed the title from Implemented to Updated K-means clustering for Nystroem
Owner

Hi @nateyoder.
Thanks for tackling this. Could you maybe post the plot from the example?
Have you experimented with some datasets and seen an improvement?

Cheers,
Andy

doc/modules/kernel_approximation.rst
 @@ -35,9 +35,15 @@ Nystroem Method for Kernel Approximation The Nystroem method, as implemented in :class:Nystroem is a general method for low-rank approximations of kernels. It achieves this by essentially subsampling the data on which the kernel is evaluated. +The subsampling methodology used to generate the approximate kernel is specified by +the parameter basis_method which can either be random or clustered.
 Owner amueller added a note May 3, 2014 I would call it kmeans instead of clustered, to be more specific. Owner amueller added a note May 3, 2014 Maybe also basis_sampling or basis_selection? nateyoder added a note May 3, 2014 Great suggestions. They are incorporated in the new version. to join this conversation on GitHub. Already have an account? Sign in to comment
commented on the diff
examples/plot_kernel_approximation.py
 @@ -149,7 +167,7 @@ [kernel_svm_time, kernel_svm_time], '--', label='rbf svm') # vertical line for dataset dimensionality = 64 -accuracy.plot([64, 64], [0.7, 1], label="n_features") +accuracy.plot([64, 64], accuracy.get_ylim(), label="n_features")
 Owner amueller added a note May 3, 2014 nice :) to join this conversation on GitHub. Already have an account? Sign in to comment
 nateyoder change basis_metod to basis_sampling and clustered to kmeans 5f313f8

As far as performance it seems to help a bit, but not quite as much as I had hoped. I think the results would be bigger if the random selection method happened to select an outlier as part of the basis sampling set but didn't try different random seeds to make that occur.

Coverage remained the same when pulling 5f313f8 on nateyoder:kmeans-nystroem into 48e2b13 on scikit-learn:master.

closed this
deleted the nateyoder:kmeans-nystroem branch
restored the nateyoder:kmeans-nystroem branch

Sorry I accidentally deleted the branch and I think doing this closed the issue. Sorry!!

reopened this
Owner

Have you tried it on a different dataset? This above is digits, right? Maybe try MNIST? Or is there some other dataset where RBF works well?

Owner

I think this should help but I also think we should make sure that it actually does ;)

Owner

Have you tried it on a different dataset? This above is digits, right? Maybe try MNIST? Or is there some other dataset where RBF works well?

You could also try on Olivetti faces with RandomizedPCA preprocessing: http://scikit-learn.org/stable/auto_examples/applications/face_recognition.html

To try on a bigger dataset you can use LFW instead of Olivetti.

Sounds great guys thanks for the suggestions. I'll give them a shot this week and post the results.

Also I noticed my build failed but it failed because of errors in OrthogonalMatchingPursuitCV. Do you guys know if this an intermitant test or something I should look into?

Owner

The travis failure is unrelated, you can ignore it.

Sorry for the long layoff guys.

Finally got a chance to run amueller's MINST example with k-means and random. As the graph shows k-means does show some minor improvement but nothing big. However, since it seems to almost always be a little better in the examples I tried it seems like it might still be worth adding it?

I briefly tried on Olivetti but I think because of the limited amount of faces saw a lot of variance in the output and didn't really get anything useful other than k-means definitely isn't a silver bullet. I didn't have time to look into LFW.

Owner

It seems consistent from the little I have seen thus far - I will try to run some tests as well. Looks pretty nice!

Owner

At first these results seemed at odds to me with the MNIST line in Table 2 of Kumar, Mohri and Talwalkar, Sampling Methods for the Nyström Method, JMLR 2012. But actually, that table is showing the kernel reconstruction "accuracy" , where K_k is the optimal rank-k reconstruction (the truncated SVD), and \tilde{K}_k is the rank-k Nyström approximation. I guess the kernel isn't as well-approximated by the uniform reconstruction, but it's still good enough to do classification with. Might be good to make sure that's the case.

Also, it might be better to use kmeans++ initialization rather than random; did you try that?

Brief update. I ran MINST again to compare "better" clustering with k-means [KMeans++ initialization, max_iter=300, and n_init=10] vs k-means as suggested in the literature ['random' initialization, max_iter=5, n_init=1] vs random Nystroem. As shown below the much more time intensive clustering has almost no impact on the classification performance while significantly increasing the time needed to train the model.

I also did the same on LFW and the results are below. In this case k-means appears to little to no consistent improvement over random selection. If you are interested I used the parameters found in http://nbviewer.ipython.org/github/jakevdp/sklearn_scipy2013/blob/master/rendered_notebooks/05.1_application_to_face_recognition.ipynb other than doing my own RBF grid search to find the optimal RBF parameters.

I'll try to do the covertype test later this week if I get time and you guys think it is still needed.

Owner

Can you please rebase your branch on master and try with MinibatchKMeans? This might be master to converge while giving good enough centroids.

referenced this pull request
Open

### Custom indices for Nystroem approximation and other kernel methods #4982

Commits on May 2, 2014
1. nateyoder authored
…mplement it in plot_kernel_approximation example to show difference
2. nateyoder authored
3. nateyoder authored
Commits on May 3, 2014
1. nateyoder authored
 @@ -35,9 +35,15 @@ Nystroem Method for Kernel Approximation The Nystroem method, as implemented in :class:Nystroem is a general method for low-rank approximations of kernels. It achieves this by essentially subsampling the data on which the kernel is evaluated. +The subsampling methodology used to generate the approximate kernel is specified by +the parameter basis_sampling which can either be random or kmeans. +If the random method is specified randomly selected data will be utilized in +the approximation while the kmeans method uses the cluster centers found via +k-means clustering. Further details concerning the subsampling methods can be found +in [ZK2010]_. By default :class:Nystroem uses the rbf kernel, but it can use any kernel function or a precomputed kernel matrix. -The number of samples used - which is also the dimensionality of the features computed - +The number of bases used - which is also the dimensionality of the features computed - is given by the parameter n_components. @@ -197,3 +203,6 @@ or store training examples. .. [VVZ2010] "Generalized RBF feature maps for Efficient Detection" _ Vempati, S. and Vedaldi, A. and Zisserman, A. and Jawahar, CV - 2010 + .. [ZK2010] "Clustered Nystroem method for large scale manifold learning and dimension reduction" + _ + Zhang, K. and Kwok, J.T. - Neural Networks, IEEE Transactions on 21, no. 10 2010