[HIVEMALL-78] Implement AUC UDAF for binary classification #52

takuti · 2017-02-28T06:22:59Z

What changes were proposed in this pull request?

In addition to current auc(array, array) for ranking (myui/hivemall#326), this patch supports auc(double, double) for binary classification.

What type of PR is it?

Feature

What is the Jira issue?

https://issues.apache.org/jira/browse/HIVEMALL-78

How was this patch tested?

Created unit test for the UDAF, and passed:

$ mvn -Dtest=hivemall.evaluation.AUCUDAFTest test

Moreover, I have launched manual tests by the following queries:

with data as (
  select 0.5 as prob, 0 as label
  union all
  select 0.3 as prob, 1 as label
  union all
  select 0.2 as prob, 0 as label
  union all
  select 0.8 as prob, 1 as label
  union all
  select 0.7 as prob, 1 as label
), data_ordered as (
  select prob, label
  from data
  order by prob desc
)
select auc(prob, label)
from (
  select prob, label
  from data_ordered
  distribute by floor(prob / 0.2)
) t;

with data as (
  select 0.5 as prob, 0 as label
  union all
  select 0.3 as prob, 1 as label
  union all
  select 0.2 as prob, 0 as label
  union all
  select 0.8 as prob, 1 as label
  union all
  select 0.7 as prob, 1 as label
), data_ordered as (
  select prob, label
  from data
  order by prob desc
)
select auc(prob, label)
from data_ordered;

Both showed AUC=0.83333. This result is same as scikit-learn's roc_auc_score():

>>> roc_auc_score([0,1,0,1,1],[0.5,0.3,0.2,0.8,0.7])
0.83333333333333326

How to use this feature?

See above queries. Input data needs to be ordered by scores in a descending order.

myui

@takuti Comments attached.

myui · 2017-02-28T06:51:54Z

core/src/test/java/hivemall/evaluation/AUCUDAFTest.java

+            Assert.assertEquals(0.83333, agg.get(), 1e-5);
+        }
+    }
+}


add a blank line

myui · 2017-02-28T07:01:01Z

core/src/main/java/hivemall/evaluation/AUCUDAF.java

+            ClassificationAUCAggregationBuffer myAggr = (ClassificationAUCAggregationBuffer) agg;
+
+            double score = HiveUtils.getDouble(parameters[0], scoreOI);
+            double label = HiveUtils.getDouble(parameters[1], labelOI);


if(parameters[0] == null) { continue; } if(parameters[1]) == null) { // separate ^ for debugging (e.g., attaching breakpoints) continue; } int label = PrimitiveObjectInspectorUtils.getInt(parameters[1], labelOI); if(label == -1) { label = 0; } else if (label != 0 && label != 1) { throw new UDFArgumentException("label MUST be 0/1 or -1/1: " + label); }

https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java#L634

continue; -> return;

myui · 2017-02-28T07:03:33Z

core/src/main/java/hivemall/evaluation/AUCUDAF.java

+            // initialize input
+            if (mode == Mode.PARTIAL1 || mode == Mode.COMPLETE) {// from original data
+                this.scoreOI = (PrimitiveObjectInspector) parameters[0];
+                this.labelOI = (PrimitiveObjectInspector) parameters[1];


this.scoreOI = HiveUtils.asDoubleCompatibleOI(parameters[0]); this.labelOI = HiveUtils.asIntegerOI(parameters[1]);

myui · 2017-02-28T07:13:51Z

core/src/main/java/hivemall/evaluation/AUCUDAF.java

-        if (!HiveUtils.isPrimitiveTypeInfo(arg1type.getListElementTypeInfo())) {
-            throw new UDFArgumentTypeException(0,
-                "The first argument `array rankItems` is invalid form: " + typeInfo[0]);
+        if (HiveUtils.isNumberTypeInfo(typeInfo[0]) && HiveUtils.isNumberTypeInfo(typeInfo[1])) {


&& HiveUtils.isIntegerTypeInfo(typeInfo[1])

myui

Thanks. @takuti Added comments again.

Better to validate score value.

myui · 2017-02-28T08:45:25Z

core/src/main/java/hivemall/evaluation/AUCUDAF.java

+            return res / (tp * fp); // scale
+        }
+
+        void iterate(@Nonnull double score, @Nonnull int label) {


Remove @Nonnull for primitives (never become null)

myui · 2017-02-28T08:49:21Z

docs/gitbook/eval/auc.md

+
+# Area Under the ROC Curve
+
+[ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) and Area Under the ROC Curve (AUC) are widely-used metric for binary (i.e., positive or negative) classification problems such as [Logistic Regression](../binaryclass/a9a_lr.md).


fix the link to ../binaryclass/a9a_lr.html

myui · 2017-02-28T08:53:28Z

docs/gitbook/eval/auc.md

+  distribute by floor(prob / 0.2)
+) t;
+```
+


Add a note explaining what floor(prob / 0.2) is meaning. Distribute AUC computation into 5 bins.

myui · 2017-02-28T08:55:20Z

core/src/main/java/hivemall/evaluation/AUCUDAF.java

+                return;
+            }
+
+            double score = HiveUtils.getDouble(parameters[0], scoreOI);


if(score < 0.0d || score > 1.0d) {
throw new UDFArgumentException("score value MUST be in range [0,1]: " + score);
}

myui · 2017-02-28T10:17:25Z

LGTM. Merged. Thank you so much for a big contribution!

coveralls · 2017-02-28T15:05:37Z

Coverage increased (+0.4%) to 36.739% when pulling 681e7fb on takuti:auc into 19d472b on apache:master.

takuti added 2 commits February 28, 2017 15:08

Implement AUCUDAF

e60ff23

Add unit test of AUCUDAF for classification

4756f46

myui reviewed Feb 28, 2017

View reviewed changes

takuti added 4 commits February 28, 2017 17:26

Document AUC UDAF for binary classification

e42cbf3

Add a blank line

0db472a

Make second argument integer

d84f2d2

Handle parameters OIs appropriately

221a8e1

myui reviewed Feb 28, 2017

View reviewed changes

takuti added 5 commits February 28, 2017 18:04

Update document for AUC

43b9e04

Remove @nonnull for primitives which are not null

5171d85

Validate parameter score

bdd04c1

Throw exception if TP/FP=0

1d9d71b

label is integer

681e7fb

asfgit closed this in 97bc912 Feb 28, 2017

takuti deleted the auc branch March 1, 2017 05:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HIVEMALL-78] Implement AUC UDAF for binary classification #52

[HIVEMALL-78] Implement AUC UDAF for binary classification #52

takuti commented Feb 28, 2017

myui left a comment

myui Feb 28, 2017

myui Feb 28, 2017

takuti Feb 28, 2017

myui Feb 28, 2017

myui Feb 28, 2017

myui left a comment

myui Feb 28, 2017

myui Feb 28, 2017

myui Feb 28, 2017

myui Feb 28, 2017

myui commented Feb 28, 2017

coveralls commented Feb 28, 2017 •

edited


		# Area Under the ROC Curve

		[ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) and Area Under the ROC Curve (AUC) are widely-used metric for binary (i.e., positive or negative) classification problems such as [Logistic Regression](../binaryclass/a9a_lr.md).

[HIVEMALL-78] Implement AUC UDAF for binary classification #52

[HIVEMALL-78] Implement AUC UDAF for binary classification #52

Conversation

takuti commented Feb 28, 2017

What changes were proposed in this pull request?

What type of PR is it?

What is the Jira issue?

How was this patch tested?

How to use this feature?

myui left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

myui left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

myui commented Feb 28, 2017

coveralls commented Feb 28, 2017 • edited

coveralls commented Feb 28, 2017 •

edited