IncrementalPCA.partial_fit gives an uninformative error message and is possibly broken

[erensezener]

IncrementalPCA.partial_fit raises the following error when the number of samples is smaller than n_components:

  File "___/sklearn/decomposition/incremental_pca.py", line 218, in partial_fit
    (self.components_.shape[0], self.n_components_))
ValueError: Number of input features has changed from 10 to 20 between calls to partial_fit! 
Try setting n_components to a fixed value.

This error can be generated via the following snippet as long as n_samples < n_components:

import numpy as np
from sklearn.decomposition import IncrementalPCA

n_samples, n_features = 10, 50
ipca = IncrementalPCA(n_components=20)

for i in range(5):
    ipca.partial_fit(np.random.rand(n_samples, n_features))

The error message is not informative because the number of input features is constant (i.e., 50)
and n_components is fixed anyways.

Also, I don't see any fundamental reason why this shouldn't work. In regular PCA, number of components after transformation can be larger than the number of samples.
Is this a limitation of the incremental PCA algorithm or the implementation?

[jakevdp]

Is this a limitation of the incremental PCA algorithm or the implementation?

Both, sort of. In standard PCA any component beyond the number of samples is ill-determined (they all have eigenvalue zero, and so are completely degenerate) and thus are useless in almost all contexts. That's probably why that corner case wasn't accounted for in IncrementalPCA.

That said, you're absolutely right that this error should be more informative. I'll put together a quick PR.

[jakevdp]

Looking at the code, I think we could make it support this case by using full_matrices=True rather than full_matrices=False in the SVD. But this would potentially be much more computationally costly without much (any?) gain in usefulness.

[jakevdp]

Actually, I now see it's more complicated than that. The svd_flip function makes some assumptions about U and V that don't hold with full_matrices=True. Though I believe it's possible to make this corner case work correctly, I think it might be cleaner to simply raise a more useful error. Any thoughts?

[nelson-liu]

Though I believe it's possible to make this corner case work correctly, I think it might be cleaner to simply raise a more useful error.

+1

[erensezener]

Both, sort of. In standard PCA any component beyond the number of samples is ill-determined (they all have eigenvalue zero, and so are completely degenerate) and thus are useless in almost all contexts. That's probably why that corner case wasn't accounted for in IncrementalPCA.

Isn't the goal of incremental PCA being able to process the samples in batches?
So n_samples > n_components > batch_size should work, right?
But with the current setting it must be batch_size > n_components.

[erensezener]

@jakevdp the variable names in the example I gave were not very accurate. But the following code should work in principle but it doesn't work if batch_size < n_components

import numpy as np
from sklearn.decomposition import IncrementalPCA

n_samples, n_features = 1000, 50
data = np.random.rand(n_samples, n_features)
batch_size = 10
number_of_batches = int(n_samples / batch_size) #1000/10 = 100
ipca = IncrementalPCA(n_components=20)

for i in range(number_of_batches):
    ipca.partial_fit(data[slice(i*batch_size,(i+1)*batch_size ), :])

[amueller]

I agree, it should be possible to do this with n_samples > n_components > batch_size. It's a bit tricky from an API perspective though in the first step. Also, you don't know n_samples in the first step...
If you can store the components, I think you should be able to increase batch_size, too. So maybe it's not worth covering this edge case, and we should just raise an error and suggest increasing the batch_size.

[wallygauze]

The dominant opinion I see here is that raising an error would be enough, but because of the 'Enhancement' and 'Moderate' labels that were added at the end, I have to ask whether enabling n_samples > n_components > batch_size is still what is wanted for this 0.19 release.
Ok for just an error?