[HIVEMALL-90] Refine incomplete AUC UDAF implementation #63
Conversation
long fp = PrimitiveObjectInspectorFactory.writableLongObjectInspector.get(fpObj); | ||
long tp = PrimitiveObjectInspectorFactory.writableLongObjectInspector.get(tpObj); | ||
long fpPrev = PrimitiveObjectInspectorFactory.writableLongObjectInspector.get(fpPrevObj); | ||
long tpPrev = PrimitiveObjectInspectorFactory.writableLongObjectInspector.get(tpPrevObj); | ||
|
||
Map<Double, Double> areaPartialMap = (Map<Double, Double>) ObjectInspectorFactory.getStandardMapObjectInspector( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ObjectInspectorFactory.getStandardMapObjectInspector(
PrimitiveObjectInspectorFactory.writableDoubleObjectInspector,
PrimitiveObjectInspectorFactory.writableLongObjectInspector
)
Invalid casting Map<DoubleWritable, LongWritable>
to Map<Double, Double>
@@ -35,7 +39,9 @@ | |||
import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator; | |||
import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.AbstractAggregationBuffer; | |||
import org.apache.hadoop.hive.serde2.io.DoubleWritable; | |||
import org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryMap; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unused import
PrimitiveObjectInspectorFactory.writableLongObjectInspector).getMap( | ||
HiveUtils.castLazyBinaryObject(areaPartialMapObj)); | ||
|
||
Map<Double, Long> fpPartialMap = (Map<Double, Long>) ObjectInspectorFactory.getStandardMapObjectInspector( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TerminatePartial returns Writable objects but receiving Java objects.
partialResult[2] = new LongWritable(myAggr.fp); | ||
partialResult[3] = new LongWritable(myAggr.tp); | ||
partialResult[4] = new LongWritable(myAggr.fpPrev); | ||
partialResult[5] = new LongWritable(myAggr.tpPrev); | ||
partialResult[6] = myAggr.areaPartialMap; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
revise OI. javaDoubleObjectInspector
instead of writableDoubleObjectInspector
?
@takuti terminatePartial/merge OIs are invalid ones. |
@myui fixed. plz check them. |
@takuti LGTM. Merged to small modifications. Thanks! |
What changes were proposed in this pull request?
Since AUC UDAF (classification) did not work correctly for some specific merge orders, this PR fixes the issue by modifying the UDAF's
merge()
andterminate()
implementation.Moreover, unit tests are refined accordingly, and a utility method is created in HiveUtils.
What type of PR is it?
Bug Fix
What is the Jira issue?
https://issues.apache.org/jira/browse/HIVEMALL-90
How was this patch tested?
How to use this feature?
Nothing has been changed from current AUC UDAF.