FastICA throws warning with certain tolerance parameters

[danthedaniel]

Context of the issue

Sometimes calls to FastICA with evolve-able parameters yield the following warning:

UserWarning: FastICA did not converge. Consider increasing tolerance or the maximum number of iterations.

Process to reproduce the issue

The following code yielded the warning:

from tpot import TPOT
from sklearn.datasets import load_digits
from sklearn.cross_validation import train_test_split

digits = load_digits()
X_train, X_test, y_train, y_test = train_test_split(digits.data, digits.target, train_size=0.75, test_size=0.25)

tpot = TPOT(generations=2, population_size=10, verbosity=2)
tpot.fit(X_train, y_train)

Possible fix

Setting a minimum value for tol in FastICA, or requiring more iterations.

[rhiever]

We should explore a wide range of tol parameters for FastICA on e.g. the MNIST data set, and see where it throws this warning. It'd be a good idea to do the same sweep across a few different data sets and see if the findings hold true.

Have you seen this warning with any other operators?

[danthedaniel]

So it looks like by default tol has a value of 1e-04 in sklearn

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/decomposition/fastica_.py#L113

But quite frequently I'm finding that the limit that is compared against tol is something like ~ 0.3 (it needs to be smaller). I've seen it as high as 1.5, though.

What's really weird though is that this issue seems to be really common on my progress_bar branch, and not on my branch for working on this issue (which comes off of master). Another odd thing, I've seen that on the same run sometimes a tolerance of 0.0001 will run just fine, but a tolerance later in the GP of 0.01 causes FastICA to throw the UserWarning.

Upping max_iter all the way to 1000 has not helped.

[danthedaniel]

@amueller, you have any insight?

[amueller]

Sorry, what's the question? What the range for the tolerance should be?

[amueller]

(also GP here means genetic program, right? that's an evil way to abbreviate it ;)

[rhiever]

Right. How do we gauge the valid range for tol?

And yes, GP = Genetic Programming. It's meant that for decades. ;-)

On Wednesday, May 18, 2016, Andreas Mueller notifications@github.com
wrote:

(also GP here means genetic program, right? that's an evil way to
abbreviate it ;)

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#148 (comment)

Randal S. Olson, Ph.D.
Postdoctoral Researcher, Institute for Biomedical Informatics
University of Pennsylvania

E-mail: rso@randalolson.com | Twitter: @randal_olson
https://twitter.com/randal_olson
http://www.randalolson.com

[amueller]

Well depends on your community. Well tol and max_iter interact. And the algorithm not converging doesn't mean the value is not valid. Searching over it seems weird in the first place, though... Why do you do that?

[rhiever]

Should we just leave it at the default? What if it affects performance?

On Wednesday, May 18, 2016, Andreas Mueller notifications@github.com
wrote:

Well depends on your community. Well tol and max_iter interact. And the
algorithm not converging doesn't mean the value is not valid. Searching
over it seems weird in the first place, though... Why do you do that?

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#148 (comment)

Randal S. Olson, Ph.D.
Postdoctoral Researcher, Institute for Biomedical Informatics
University of Pennsylvania

E-mail: rso@randalolson.com | Twitter: @randal_olson
https://twitter.com/randal_olson
http://www.randalolson.com

[amueller]

it's a speed optimization though. Does tpot have a concept of accuracy time trade-off?
Higher tol will always be better, but take longer, and have diminishing returns...

[danthedaniel]

If FastICA does not converge, does that mean that its results are not useful?

I'm wondering if it'd be harmless to ignore those warnings. If it just results in mediocre results then I'd like to just have the GP weed out those not-so-great configurations.

[rhiever]

Should we just use the default then? Has that default been tuned?

On Wednesday, May 18, 2016, Andreas Mueller notifications@github.com
wrote:

it's a speed optimization though. Does tpot have a concept of accuracy
time trade-off?
Higher tol will always be better, but take longer, and have diminishing
returns...

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#148 (comment)

Randal S. Olson, Ph.D.
Postdoctoral Researcher, Institute for Biomedical Informatics
University of Pennsylvania

E-mail: rso@randalolson.com | Twitter: @randal_olson
https://twitter.com/randal_olson
http://www.randalolson.com

[danthedaniel]

The defaults have been giving me the same warning, though. It's particularly odd.

Edit:

Oh, and it occurs with Iris and MNIST, but much more frequently with MNIST.

[amueller]

The default probably hasn't been tuned. But if you don't consider time-accuracy trade-offs, tuning seems odd.

[bartleyn]

So I'm not too familiar with the model, but the convergence criteria is essentially that the change in each estimated component falls under the tolerance, right? If that's the case, then to double down on what's been previously said, we could have useful (or at least mediocre) results without convergence if a subset of the components were just being stubborn. I agree with @teaearlgraycold in ignoring the warning and letting the GP weed them out (assuming they're usually worse pipelines).

[amueller]

if you want to tune it, I'd also just ignore the warning.

[rhiever]

Thanks all for your input! We'll still allow TPOT to optimize the parameter for now, and ignore the warnings (per #149).