-
Notifications
You must be signed in to change notification settings - Fork 832
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: remove verbose CMS output and convert scoredLabelsCol to DoubleType #1361
Conversation
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
Codecov Report
@@ Coverage Diff @@
## master #1361 +/- ##
==========================================
+ Coverage 84.78% 84.82% +0.03%
==========================================
Files 287 287
Lines 14231 14230 -1
Branches 732 732
==========================================
+ Hits 12066 12070 +4
+ Misses 2165 2160 -5
Continue to review full report at Codecov.
|
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
nice, seems it was added as part of this PR by accident: |
@@ -281,7 +281,7 @@ class ComputeModelStatistics(override val uid: String) extends Transformer | |||
// Calculate confusion matrix and output it as DataFrame | |||
// TODO: We call cache in order to avoid a bug with catalyst where CMS seems to get stuck in a loop | |||
// For future spark upgrade past 2.2.0, we should try to see if the cache() call can be removed | |||
dataset.select(col(scoredLabelsColumnName), col(labelColumnName)) | |||
dataset.select(col(scoredLabelsColumnName).cast(DoubleType), col(labelColumnName)) | |||
.cache() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
might be interesting to see if this cache call can indeed finally be removed, I remember this was a pretty nasty bug
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since I don't know anything about this bug, I'm not sure how to validate it. Do you have any ideas what this bug is?
No description provided.