fix: remove verbose CMS output and convert scoredLabelsCol to DoubleType #1361
Conversation
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Codecov Report
@@ Coverage Diff @@
## master #1361 +/- ##
==========================================
+ Coverage 84.78% 84.82% +0.03%
==========================================
Files 287 287
Lines 14231 14230 -1
Branches 732 732
==========================================
+ Hits 12066 12070 +4
+ Misses 2165 2160 -5
Continue to review full report at Codecov.
|
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
nice, seems it was added as part of this PR by accident: |
| @@ -281,7 +281,7 @@ class ComputeModelStatistics(override val uid: String) extends Transformer | |||
| // Calculate confusion matrix and output it as DataFrame | |||
| // TODO: We call cache in order to avoid a bug with catalyst where CMS seems to get stuck in a loop | |||
| // For future spark upgrade past 2.2.0, we should try to see if the cache() call can be removed | |||
| dataset.select(col(scoredLabelsColumnName), col(labelColumnName)) | |||
| dataset.select(col(scoredLabelsColumnName).cast(DoubleType), col(labelColumnName)) | |||
| .cache() | |||
There was a problem hiding this comment.
might be interesting to see if this cache call can indeed finally be removed, I remember this was a pretty nasty bug
There was a problem hiding this comment.
Since I don't know anything about this bug, I'm not sure how to validate it. Do you have any ideas what this bug is?
No description provided.