[MRG+1] avoid integer overflow by using floats for matthews_corrcoef

[sam-s]

Reference Issue

Fixes #9622

What does this implement/fix? Explain your changes.

avoid integer overflow by using floats

[jnothman]

Could you please add a test?

[jnothman]

Otherwise LGTM

[jnothman]

flake8 errors:

./sklearn/metrics/tests/test_classification.py:491:33: E231 missing whitespace after ','
        true_pos = conf_matrix[1,1]
                                ^
./sklearn/metrics/tests/test_classification.py:492:34: E231 missing whitespace after ','
        false_pos = conf_matrix[1,0]
                                 ^
./sklearn/metrics/tests/test_classification.py:493:34: E231 missing whitespace after ','
        false_neg = conf_matrix[0,1]

[jnothman]

flake8 errors:

./sklearn/metrics/tests/test_classification.py:491:33: E231 missing whitespace after ','
        true_pos = conf_matrix[1,1]
                                ^
./sklearn/metrics/tests/test_classification.py:492:34: E231 missing whitespace after ','
        false_pos = conf_matrix[1,0]
                                 ^
./sklearn/metrics/tests/test_classification.py:493:34: E231 missing whitespace after ','
        false_neg = conf_matrix[0,1]

[lesteve]

Maybe the code can be made simpler and the test as well

[sam-s]

what does "MRG+1" mean?

[sam-s]

what does "MRG+1" mean?

[NelleV]

@sam-s It means the pull requests has been approved by one core contributor.

[NelleV]

@sam-s It means the pull requests has been approved by one core contributor.

[jnothman]

@lesteve I don't entirely get why your solution works. Isn't the largest number n_samples_square, which remains an integer in your code?

[jnothman]

@lesteve I don't entirely get why your solution works. Isn't the largest number n_samples_square, which remains an integer in your code?

[lesteve]

@lesteve I don't entirely get why your solution works. Isn't the largest number n_samples_square, which remains an integer in your code?

Integer division behaves as we want for Python3 (i.e. true division) and we have from __future__ import division so that it behaves as we want as well for Python 2. Maybe more related to what you wanted to hear, the overflow was because we were multiplying two covariances (order n^2 * n^2 super roughly). For n^2 to overflow you need 3 billion samples.

[lesteve]

@lesteve I don't entirely get why your solution works. Isn't the largest number n_samples_square, which remains an integer in your code?

Integer division behaves as we want for Python3 (i.e. true division) and we have from __future__ import division so that it behaves as we want as well for Python 2. Maybe more related to what you wanted to hear, the overflow was because we were multiplying two covariances (order n^2 * n^2 super roughly). For n^2 to overflow you need 3 billion samples.

[sam-s]

in the current patch, n_samples is a float64.

[sam-s]

in the current patch, n_samples is a float64.

[lesteve]

given the lack of transparent bignums in numpy, your magic is dangerous.
I don't mind the other change ("manipulating quantities that are of order 1"), but casting to float is critical.

Although I agree it can be surprising at first I would not call it magic, it is just the behaviour of true division that operands get casted to floats before the division takes place.

I am not going to claim to be a floating point expert, but I would go with my version posted in #9693 (comment).

[lesteve]

given the lack of transparent bignums in numpy, your magic is dangerous.
I don't mind the other change ("manipulating quantities that are of order 1"), but casting to float is critical.

Although I agree it can be surprising at first I would not call it magic, it is just the behaviour of true division that operands get casted to floats before the division takes place.

I am not going to claim to be a floating point expert, but I would go with my version posted in #9693 (comment).

[sam-s]

* Loïc Estève ***@***.***> [2017-09-08 07:16:35 -0700]: > given the lack of transparent bignums in numpy, your magic is dangerous. I don't mind the other change ("manipulating quantities that are of order 1"), but casting to float is critical.

Yes, casting to float is the crux of the matter, and the current version does it via the `dtype=np.float64` argument.

Although I agree it can be surprising at first I would not call it magic, it is just the behaviour of true division that operands get casted to floats before the division takes place.

Yes, but we must ensure that they are correct before the cast. The current version does it.

I am not going to claim to be a floating point expert, but I would go with my version posted in #9693 (comment).

The only advantage of your version over the current one is that you compute `n_samples ** 2` once instead of twice. This is more than made up for by several extra divisions. At any rate, "manipulating quantities that are of order 1" does not matter for floats (by the mere _meaning_ of the term "floating point number").

…

-- Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504 http://steingoldpsychology.com http://www.childpsy.net http://iris.org.il http://mideasttruth.com http://www.dhimmitude.org http://no2bds.org At war time "salt of the earth" becomes "cannon fodder".

[sam-s]

* Loïc Estève ***@***.***> [2017-09-08 07:16:35 -0700]: > given the lack of transparent bignums in numpy, your magic is dangerous. I don't mind the other change ("manipulating quantities that are of order 1"), but casting to float is critical.

Yes, casting to float is the crux of the matter, and the current version does it via the `dtype=np.float64` argument.

Although I agree it can be surprising at first I would not call it magic, it is just the behaviour of true division that operands get casted to floats before the division takes place.

Yes, but we must ensure that they are correct before the cast. The current version does it.

I am not going to claim to be a floating point expert, but I would go with my version posted in #9693 (comment).

The only advantage of your version over the current one is that you compute `n_samples ** 2` once instead of twice. This is more than made up for by several extra divisions. At any rate, "manipulating quantities that are of order 1" does not matter for floats (by the mere _meaning_ of the term "floating point number").

…

-- Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504 http://steingoldpsychology.com http://www.childpsy.net http://iris.org.il http://mideasttruth.com http://www.dhimmitude.org http://no2bds.org At war time "salt of the earth" becomes "cannon fodder".

[amueller]

At any rate, "manipulating quantities that are of order 1" does not
matter for floats (by the mere meaning of the term "floating point
number").

The exponent can also overflow, though I feel that is very unlikely to happen.

I'm more concerned about C.sum() to overflow (though also unlikely?), which is avoided by casting to float.

[amueller]

At any rate, "manipulating quantities that are of order 1" does not
matter for floats (by the mere meaning of the term "floating point
number").

The exponent can also overflow, though I feel that is very unlikely to happen.

I'm more concerned about C.sum() to overflow (though also unlikely?), which is avoided by casting to float.

[amueller]

Either solution seems fine for me.

[amueller]

Either solution seems fine for me.

[sam-s]

* Andreas Mueller ***@***.***> [2017-09-08 15:51:57 +0000]: > At any rate, "manipulating quantities that are of order 1" does not > matter for floats (by the mere _meaning_ of the term "floating point > number"). The exponent can also overflow, though I feel that is very unlikely to happen.

The only way it could happen is if sample_weight.sum() overflows, as in sample_weight = np.repeat([1e305], 10000) We can deal with this issue by scaling: sample_weight /= abs(sample_weight.max()) + abs(sample_weight.min()) but it is unclear if this is worth the effort. Thanks.

…

-- Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504 http://steingoldpsychology.com http://www.childpsy.net http://honestreporting.com http://islamexposedonline.com http://www.memritv.org http://camera.org Daddy, what does "format disk c: complete" mean?

[sam-s]

* Andreas Mueller ***@***.***> [2017-09-08 15:51:57 +0000]: > At any rate, "manipulating quantities that are of order 1" does not > matter for floats (by the mere _meaning_ of the term "floating point > number"). The exponent can also overflow, though I feel that is very unlikely to happen.

The only way it could happen is if sample_weight.sum() overflows, as in sample_weight = np.repeat([1e305], 10000) We can deal with this issue by scaling: sample_weight /= abs(sample_weight.max()) + abs(sample_weight.min()) but it is unclear if this is worth the effort. Thanks.

…

-- Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504 http://steingoldpsychology.com http://www.childpsy.net http://honestreporting.com http://islamexposedonline.com http://www.memritv.org http://camera.org Daddy, what does "format disk c: complete" mean?

[jnothman]

I'm happy to merge whichever solution

[sam-s]

On Sat, Sep 9, 2017 at 7:50 AM, Joel Nothman ***@***.***> wrote: > @jnothman commented on this pull request. > I'm happy to merge whichever solution

Thank you, the current version works.

[sam-s]

On Sat, Sep 9, 2017 at 7:50 AM, Joel Nothman ***@***.***> wrote: > @jnothman commented on this pull request. > I'm happy to merge whichever solution

Thank you, the current version works.

[jnothman]

We still need a second lgtm

[jnothman]

We still need a second lgtm

[amueller]

thanks

[amueller]

thanks