Implementation of r2_score function #1896

mottodora · 2016-11-18T08:49:16Z

I implemented the r2_score function that calculates R^2(coefficient of determination) regression score function.
As of now, the code is compatible with scikit-learn, only when the "sample_weight" argument is set to "none".
Is this argument necessary for chainer? If necessary, I will implement the changes.
Additionally, I would like to know whether the type of this argument is Variable or ndarray?

delta2323 · 2016-11-18T12:26:46Z

Please fix violation of the coding guideline by referencing travis result.

delta2323

First review.

delta2323 · 2016-11-18T12:34:10Z

chainer/functions/evaluation/r2_score.py

+        )
+
+        type_check.expect(
+            pred_type.ndim >= true_type.ndim,


We do not need this condition because pred_type.shape == true_type.shape implies the equality of ndim's

I deleted this line.

delta2323 · 2016-11-18T12:37:08Z

chainer/functions/evaluation/r2_score.py

+        if self.multioutput == 'uniform_average':
+            return xp.asarray((1 - SS_res / SS_tot).mean(), dtype=pred.dtype),
+        elif self.multioutput == 'raw_values':
+            return xp.asarray((1 - SS_res / SS_tot), dtype=pred.dtype),


scikit learn seems to set r2 value to 0 when SS_tot is 0.

I fixed this problem in 73d3f3b. Please check it.

delta2323 · 2016-11-18T12:38:13Z

chainer/functions/evaluation/r2_score.py

+    """Computes R^2(coefficient of determination) regression score function.
+
+    Args:
+        pred(Variable): Variable holding a vector or matrix of estimated \


We do not need backslashes in breaking the description of an argument.

I deleted backslash.

delta2323 · 2016-11-20T09:01:16Z

chainer/functions/evaluation/r2_score.py

+
+    """
+    return R2_score(sample_weight=sample_weight, multioutput=multioutput)\
+            (pred, true)


You can break the line in the middle of parentheses as follows:

return R2_score(sample_weight=sample_weight, multioutput=multioutput)(pred, true)

I fixed this line.

delta2323

Second review comments

delta2323 · 2016-11-22T11:11:34Z

chainer/functions/evaluation/r2_score.py



 def r2_score(pred, true, sample_weight=None, multioutput='uniform_average'):
    """Computes R^2(coefficient of determination) regression score function.

    Args:
-        pred(Variable): Variable holding a vector or matrix of estimated \
+        pred(Variable): Variable holding a vector or matrix of estimated
                target values.
        true(Variable): Variable holding a vector or matrix of correct target \


Please remove backslash here, too.

delta2323 · 2016-11-22T11:21:26Z

chainer/functions/evaluation/r2_score.py

+                return xp.asarray((1 - SS_res / SS_tot).mean(),
+                                  dtype=pred.dtype),
+        elif self.multioutput == 'raw_values':
+            if xp.any(SS_tot == 0):


If the reason for this branching is to avoid zero division by SS_tot, we cannot avoid it because SS_res / SS_tot is evaluated anyway in case SS_tot == 0. Is there other reason for this branching? If not, I recommend to remove the branching. By removing it, we can expect faster forward computation.

scikit learn returns 0 when SS_tot is 0. If not brancing, this function will return NaN.

I thought something like this:

ret = xp.where(SS_tot != 0, 1 - SS_res / SS_tot, 0.0).astype(pred.dtype) if self.multioutput == 'uniform_average': return ret.mean() else: return ret

Does it make sense to you?

Yes. I'm going to implement like this.

delta2323 · 2016-11-22T11:23:51Z

chainer/functions/evaluation/r2_score.py

+                    .astype(pred.dtype),
+            else:
+                return xp.asarray((1 - SS_res / SS_tot), dtype=pred.dtype),
+


We can merge uniform_average case and raw_values case as follows

ret = xp.asarray((1 - SS_res / SS_tot), dtype=pred.dtype) if self.multioutput == 'uniform_average' return ret.mean() else: return ret

delta2323 · 2016-11-22T11:25:09Z

chainer/functions/evaluation/r2_score.py

+
+    Args:
+        pred(Variable): Variable holding a vector or matrix of estimated
+                target values.


pred can be not only a vector or a matrix but a tensor of any dimension.

delta2323 · 2016-11-22T11:25:30Z

chainer/functions/evaluation/r2_score.py

+    Args:
+        pred(Variable): Variable holding a vector or matrix of estimated
+                target values.
+        true(Variable): Variable holding a vector or matrix of correct target \


true can be not only a vector or a matrix but a tensor of any dimension.

delta2323 · 2016-11-22T11:27:29Z

chainer/functions/evaluation/r2_score.py

+                target values.
+        true(Variable): Variable holding a vector or matrix of correct target \
+                values.
+        sample_weight: None.


How about writing the description in more detail like this:

This argument is for compatibility with scikit-learn's implementation of r2_score. Current implementation admits None only.

delta2323 · 2016-11-22T11:27:47Z

chainer/functions/evaluation/r2_score.py

+                values.
+        sample_weight: None.
+        multioutput(string): ['uniform_average', 'raw_values']. if
+                'uniform_average', this function return an average of R^2


delta2323 · 2016-11-22T11:30:27Z

chainer/functions/evaluation/r2_score.py

+                'multioutput' is 'uniform_average' or a vector of R^2
+                scores if 'multioutput' is 'raw_values'.
+
+    .. note:: This function is non-differentiable.


As forward computation of this function consists of differentiable operations, we can back propagate an error if we implement backward method appropriately. Do you think it is beneficial to implement backward?

I think that there is no difference between mean squared error and r_squared as loss function.

delta2323 · 2016-11-22T11:31:30Z

tests/chainer_tests/functions_tests/evaluation_tests/test_r2_score.py

+        expected = r2_score(self.x, self.t, sample_weight=None,
+                            multioutput=self.multioutput)
+        testing.assert_allclose(
+            expected, cuda.to_cpu(y.data), **self.check_forward_options)


You need not transfer y.data to CPU because testing.assert_allclose does it.

delta2323 · 2016-11-29T01:05:22Z

Could you merge the latest master branch?

delta2323 · 2016-12-10T12:26:06Z

LGTM

delta2323 · 2016-12-10T12:26:35Z

Thank you for your contribution!

mottodora and others added 3 commits November 16, 2016 17:48

implement r2_score evaluation function

0981ef7

write documents

ce4eab4

fix documents

d03571b

delta2323 self-assigned this Nov 18, 2016

delta2323 requested changes Nov 18, 2016

View reviewed changes

delta2323 reviewed Nov 20, 2016

View reviewed changes

mottodora added 5 commits November 22, 2016 14:19

fix flake8

903ba1d

set r2 value to 0 when SS_tot is 0.

73d3f3b

fix document

4e7732d

merge conflict

af48448

fix flake8

c308564

delta2323 requested changes Nov 22, 2016

View reviewed changes

fix codes

58797a5

delta2323 added the cat:feature Implementation that introduces new interfaces. label Nov 28, 2016

mottodora and others added 5 commits November 29, 2016 10:57

merge master

bd95c03

fixed conflict

c681dd8

flake8

55b026e

fix flake8

928254c

Merge remote-tracking branch 'upstream/master' into r_squared

deedc16

delta2323 approved these changes Dec 10, 2016

View reviewed changes

delta2323 merged commit fc688be into chainer:master Dec 10, 2016

delta2323 added this to the v1.19.0 milestone Dec 10, 2016

mottodora deleted the r_squared branch December 10, 2016 12:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation of r2_score function #1896

Implementation of r2_score function #1896

mottodora commented Nov 18, 2016

delta2323 commented Nov 18, 2016

delta2323 left a comment

delta2323 Nov 18, 2016

mottodora Nov 22, 2016

delta2323 Nov 18, 2016

mottodora Nov 22, 2016

delta2323 Nov 18, 2016

mottodora Nov 22, 2016

delta2323 Nov 20, 2016

mottodora Nov 22, 2016

delta2323 left a comment

delta2323 Nov 22, 2016

delta2323 Nov 22, 2016 •

edited

mottodora Nov 22, 2016

delta2323 Nov 23, 2016

mottodora Nov 24, 2016

delta2323 Nov 22, 2016 •

edited

delta2323 Nov 22, 2016

delta2323 Nov 22, 2016

delta2323 Nov 22, 2016 •

edited

delta2323 Nov 22, 2016

delta2323 Nov 22, 2016

mottodora Nov 22, 2016

delta2323 Nov 22, 2016

delta2323 commented Nov 29, 2016

delta2323 commented Dec 10, 2016

delta2323 commented Dec 10, 2016

Implementation of r2_score function #1896

Implementation of r2_score function #1896

Conversation

mottodora commented Nov 18, 2016

delta2323 commented Nov 18, 2016

delta2323 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

delta2323 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

delta2323 Nov 22, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

delta2323 Nov 22, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

delta2323 Nov 22, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

delta2323 commented Nov 29, 2016

delta2323 commented Dec 10, 2016

delta2323 commented Dec 10, 2016

delta2323 Nov 22, 2016 •

edited

delta2323 Nov 22, 2016 •

edited

delta2323 Nov 22, 2016 •

edited