Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log loss metric #318

Merged
merged 5 commits into from Jul 27, 2018

Conversation

Projects
None yet
1 participant
@TomAugspurger
Copy link
Member

TomAugspurger commented Jul 27, 2018

No description provided.

TomAugspurger added some commits Jul 27, 2018

@TomAugspurger

This comment has been minimized.

Copy link
Member Author

TomAugspurger commented Jul 27, 2018

The following gets sample_weight "working", in the sense that the computation completes:

diff --git a/dask_ml/metrics/classification.py b/dask_ml/metrics/classification.py
index 7766370..3025501 100644
--- a/dask_ml/metrics/classification.py
+++ b/dask_ml/metrics/classification.py
@@ -94,11 +94,13 @@ def accuracy_score(y_true, y_pred, normalize=True, sample_weight=None, compute=T
     return score
 
 
-def _log_loss_inner(x, y, **kwargs):
+def _log_loss_inner(x, y, sample_weight, **kwargs):
     # da.map_blocks wasn't able to concatenate together the results
     # when we reduce down to a scalar per block. So we make an
     # array with 1 element.
-    return np.array([sklearn.metrics.log_loss(x, y, **kwargs)])
+    if sample_weight is not None:
+        sample_weight = sample_weight.ravel()
+    return np.array([sklearn.metrics.log_loss(x, y, sample_weight=sample_weight, **kwargs)])
 
 
 def log_loss(
@@ -107,19 +109,20 @@ def log_loss(
     if y_true.ndim == 1:
         y_true = y_true.reshape(-1, 1)
     if sample_weight is not None:
-        raise ValueError("sample_weight is not supported.")
+        # raise ValueError("sample_weight is not supported.")
+        sample_weight = sample_weight.reshape(-1, 1)
     assert y_pred.ndim == 2
 
     result = da.map_blocks(
         _log_loss_inner,
         y_true,
         y_pred,
+        sample_weight,
         chunks=(1,),
         drop_axis=1,
         dtype="f8",
         eps=eps,
         normalize=normalize,
-        sample_weight=sample_weight,
         labels=labels,
     )
     if normalize:
diff --git a/tests/metrics/test_metrics.py b/tests/metrics/test_metrics.py
index 795fe7a..0876414 100644
--- a/tests/metrics/test_metrics.py
+++ b/tests/metrics/test_metrics.py
@@ -86,7 +86,7 @@ def test_pairwise_kernels(kernel):
     "sample_weight",
     [
         pytest.param(
-            True, marks=pytest.mark.xfail(reason="sample_weight not implemented")
+            True, # marks=pytest.mark.xfail(reason="sample_weight not implemented")
         ),
         False,
     ],

Unfortunately, this means the problem is no longer linear, so we can just take a simple sum / mean of the block-wise scores.

Although... I wonder if we could do a final weighted sum of the blockwise scores, weighted by each block's weight?

@TomAugspurger

This comment has been minimized.

Copy link
Member Author

TomAugspurger commented Jul 27, 2018

OK, I didn't do the math on sample_weight, but the tests pass :)

@TomAugspurger TomAugspurger merged commit 3834756 into dask:master Jul 27, 2018

5 checks passed

ci/circleci: py27 Your tests passed on CircleCI!
Details
ci/circleci: py36 Your tests passed on CircleCI!
Details
ci/circleci: sklearn_dev Your tests passed on CircleCI!
Details
codecov/patch 100% of diff hit (target 92.51%)
Details
codecov/project 92.57% (+0.05%) compared to 6c6eeb4
Details

@TomAugspurger TomAugspurger deleted the TomAugspurger:log-loss branch Jul 27, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.