Commit
## What changes were proposed in this pull request? add `setWeightCol` method for OneVsRest. `weightCol` is ignored if classifier doesn't inherit HasWeightCol trait. ## How was this patch tested? + [x] add an unit test. Author: Yan Facai (颜发才) <facai.yan@gmail.com> Closes #18554 from facaiy/BUG/oneVsRest_missing_weightCol. (cherry picked from commit a5a3189) Signed-off-by: Yanbo Liang <ybliang8@gmail.com>
- Loading branch information
There are no files selected for viewing
6 comments
on commit 8520d7c
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@yanboliang @facaiy I think this makes branch 2.1 fail:
[info] - SPARK-21306: OneVsRest should support setWeightCol *** FAILED *** (180 milliseconds)
[info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 382.0 failed 1 times, most recent failure: Lost task 0.0 in stage 382.0 (TID 759, localhost, executor driver): scala.MatchError: [0.0,1,[6.7885086340489185,3.4576551565453197,2.0812768587303507,0.3552148531053505]] (of class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema)
[info] at org.apache.spark.ml.classification.LogisticRegression$$anonfun$12.apply(LogisticRegression.scala:330)
[info] at org.apache.spark.ml.classification.LogisticRegression$$anonfun$12.apply(LogisticRegression.scala:330)
[info] at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
[info] at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:216)
...
Branch 2.0 seems to not compile?
[error] /home/jenkins/workspace/spark-branch-2.0-test-maven-hadoop-2.2/mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala:147: not found: value lit
[error] val dataset2 = dataset.withColumn("weight", lit(1))
[error] ^
[error] one error found
[error] Compile failed at Jul 27, 2017 7:34:14 PM [20.414s]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll check it this weekend, @srowen.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@srowen Thanks for catching this. I will check it as well, maybe we can't directly merge it into branch before 2.1.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No problem. I'd like to work on it this weekend. However, the SparkException on 2.1 is quite amazing for me, so any help / suggestion will be appreciated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@yanboliang @srowen Sigh. The exception is resolved by
lit(1.0)
on branch-2.1.