[MRG+2] GaussianNB(): new parameter var_smoothing

[Mottl]

This pull-request changes epsilon to 1/20 of max of variance to improve prediction strength.
MNIST handwritten digits recognition test:
score before: 0.563
score after: 0.798

[jnothman]

On what basis can we assume/know this will not degrade performance elsewhere?

[jnothman]

It's obviously changing other results in our tests

[agramfort]

I agree with @jnothman

if you want to do something like this I would add parameter like min_std so you can adjust it.

[Mottl]

I've added min_variance parameter to GaussianNB(), which is by default calculated as 1e-9 multiplied by the maximum variance across all dimensions. It behaves much like adding an epsilon to a variance as in the current code.

[amueller]

I was thinking about this a couple of weeks ago and couldn't find any references on it. I think it's a bit unclear on whether you want to have a minimum or an additive constant. I think a Bayesian prior would be an additive constant, right? I guess in practice it doesn't make a lot of difference. Having a reference would be good, though.

[amueller]

Actually a Bayesian prior would pull it towards a specific value, not be an additive constant.... That might be the most natural thing to implement imho?

[Mottl]

Adding ε (or setting a minimum variance) is necessary for the single purpose — make the calculation of Gaussian PDF computable as we have σ in the denominator:

It is not obvious for me why we should add ε to variances of all features instead of those that produce division by zero error.

So it is seems to me it is slightly better to use a minimum variance than to add ε to all variances, but the difference is negligible.

[agramfort]

why not using the same trick as before (cf below)

# put epsilon back each time?

[Mottl]

Because we don't use a constant epsilon anymore.

[Mottl]

And we can't subtract min_variance from self.sigma_ (that way we'd always get zero calculating previous sigma).

[Mottl]

So far I can't find any difference between adding epsilon and setting minimum variance when using Naive Bayes. But I found that adding epsilon is more stable when used in Non-naive Bayes classification:

Non-naive Bayes classification
==============================
MNIST handwritten digits
Train dataset: (1000, 784), test dataset: (1000, 784)
Score (epsilon=1e-09)       = 0.832
Score (min_variance=1e-09)  = 0.832
Score (epsilon=1e-08)       = 0.919
Score (min_variance=1e-08)  = 0.48
Score (epsilon=1e-07)       = 0.919
Score (min_variance=1e-07)  = 0.479
Score (epsilon=1e-06)       = 0.919
Score (min_variance=1e-06)  = 0.487
Score (epsilon=1e-05)       = 0.919
Score (min_variance=1e-05)  = 0.496
Score (epsilon=0.0001)      = 0.919
Score (min_variance=0.0001) = 0.588
Score (epsilon=0.001)       = 0.921
Score (min_variance=0.001)  = 0.668
Score (epsilon=0.01)        = 0.933
Score (min_variance=0.01)   = 0.729
Score (epsilon=0.1)         = 0.936
Score (min_variance=0.1)    = 0.488
Score (epsilon=1)           = 0.865
Score (min_variance=1)      = 0.928
Score (epsilon=10)          = 0.709
Score (min_variance=10)     = 0.87
Score (epsilon=100)         = 0.351
Score (min_variance=100)    = 0.204

[amueller]

what do you mean by non-naive bayes?

[amueller]

btw, this dataset is very non-gaussian and therefore a very odd example. I think I would usually assume that people use the StandardScaler before GaussianNB and that the data is at least somewhat gaussian.

[Mottl]

Non-naive bayes is that one uses full covariance matrix instead of just a variance vector in a multivariate gaussian pdf (see scipy.stats.multivariate_normal.logpdf() ).
I'm planning to add non-naive bayes class in a couple of days in another PR. That's why sharing the same parameter epsilon becomes somewhat better.

[Mottl]

What dataset do you suggest for testing Naive Bayes?

[agramfort]

non-naive Bayes full covariance is quadratic discriminant analysis (aka QDA) and is already in sklearn.

[agramfort]

epsilon is a too generic name. Please try to improve variable name and docstring. I not opposed to exposition this internal epsilon parameter.

[Mottl]

Is smoothing looks better than epsilon?

[agramfort]

min_var_scaling ?

[Mottl]

It is not a scaling parameter, but an additive one. And since 3de55d0 it is not a minimum of variance but an additive value to all variances.
var_smoothing?

[agramfort]

var_smoothing is fine with me.

[Mottl]

@agramfort
Done.

[agramfort]

+1 for MRG

[jnothman]

Looks great! Please add an entry to doc/whats_new, and we'll merge. Thanks!

[lesteve]

I added the PR link in whats_new this way it is easier to find out details about the change.

[lesteve]

This one should be merged when the CIs are green.

[Mottl]

Thank you all

[lesteve]

Merging, thanks a lot @Mottl!

[hongwen000]

I hope the purpose of adding this parameter will be documented. I am taking a ML course and working out a classification problem. My teacher asked me to explain how I tuned whatever selected models in the report, including the meaning of their parameters. I stuck there unable to find out what this var_smoothing really mean until I find this...