Re-implement PR-AUC. #7297
Re-implement PR-AUC. #7297
Conversation
|
LTR support is added. No breaking change. |
Codecov Report
@@ Coverage Diff @@
## master #7297 +/- ##
=======================================
Coverage 83.62% 83.62%
=======================================
Files 13 13
Lines 3884 3884
=======================================
Hits 3248 3248
Misses 636 636 Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Am I correct that you are trying to extract the scan and reduce phase of the AUC computation, but pass lambda functions to achieve the different variations in AUC (ROC-AUC vs PR-AUC, multiclass and ranking)?
This PR probably needs some performance testing on CPU and GPU.
| if (!cache) { | ||
| cache.reset(new DeviceAUCCache); | ||
| } | ||
| cache->Init(predts, is_multi, device); |
There was a problem hiding this comment.
Any reason not to use its constructor?
There was a problem hiding this comment.
init can handle a changed input matrix.
Yes.
Will run some simple benchmarks. |
|
@RAMitchell I attached some benchmark results to the PR description. |
* Support binary/multi-class classification, ranking. * Add documents. * Handle missing data.
|
Pushed a commit that prevents integer overflow inside cub for ROC-AUC. |
The new implementation supports binary/multi-class classification and learning to rank to binary relevance. Also, it handles empty datasets and irregular datasets by returning NaN instead of making an exception. Lastly, the GPU implementation has the same functionality as the CPU one instead of being ranking only.
n_samples = 1e7
runs = 16
n_classes = 8 (multi-class only)
Close #6561 .
Close #6272 .
Close #6551 .
Close #6692 .