feat: add interface function for updating learning_rate per each iteration in LightGBMDelegate #849
Conversation
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Codecov Report
@@ Coverage Diff @@
## master #849 +/- ##
=======================================
Coverage 85.16% 85.17%
=======================================
Files 186 187 +1
Lines 8603 8612 +9
Branches 508 521 +13
=======================================
+ Hits 7327 7335 +8
- Misses 1276 1277 +1
Continue to review full report at Codecov.
|
|
@imatiach-msft Can I see detailed failure logs in unittest? |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: org.scalatest.Assertions$AssertionsHelper Serialization stack: - object not serializable (class: org.scalatest.Assertions$AssertionsHelper, value: org.scalatest.Assertions$AssertionsHelper@15d041ed) - field (class: org.scalatest.FunSuite, name: assertionsHelper, type: class org.scalatest.Assertions$AssertionsHelper) - object (class com.microsoft.ml.spark.lightgbm.split1.VerifyLightGBMClassifier, VerifyLightGBMClassifier) - field (class: com.microsoft.ml.spark.lightgbm.split1.VerifyLightGBMClassifier$$anonfun$21, name: org.apache.spark.SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: org.scalatest.Assertions$AssertionsHelper |
|
[info] - Verify LightGBM Classifier updating learning_rate on training by using LightGBMDelegate *** FAILED *** [info] - object (class scala.collection.immutable.List$SerializationProxy, scala.collection.immutable.List$SerializationProxy@3d4ebb56) [info] - writeReplace data (class: scala.collection.immutable.List$SerializationProxy) [info] - object (class scala.collection.immutable.$colon$colon, List(org.apache.spark.OneToOneDependency@230f8966)) [info] - field (class: org.apache.spark.rdd.RDD, name: org$apache$spark$rdd$RDD$$dependencies_, type: interface scala.collection.Seq) [info] - object (class org.apache.spark.rdd.MapPartitionsRDD, MapPartitionsRDD[4766] at reduce at LightGBMBase.scala:160) [info] - field (class: scala.Tuple2, name: _1, type: class java.lang.Object) [info] - object (class scala.Tuple2, (MapPartitionsRDD[4766] at reduce at LightGBMBase.scala:160,)) [info] at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40) [info] at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46) [info] at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100) [info] at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1155) [info] at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:1069) [info] at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:1013) [info] at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2067) [info] at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059) [info] at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048) [info] at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) [info] ... |
|
@ocworld it looks like there was a serialization failure in the test case, will take a look and try to run locally when I get a chance |
| while (!isFinished && iters < trainParams.numIterations) { | ||
|
|
||
| if (delegate.isDefined) { | ||
| delegate.get.beforeTrainIteration(partitionId, iters, log, trainParams, boosterPtr, hasValid) | ||
| val newLearningRate = delegate.get.getLearningRate(partitionId, iters, log, trainParams, learningRate) | ||
| if (newLearningRate != learningRate) { | ||
| log.info(s"LightGBM worker calling LGBM_BoosterResetParameter to reset learningRate" + |
| @@ -360,6 +362,47 @@ class VerifyLightGBMClassifier extends Benchmarks with EstimatorFuzzing[LightGBM | |||
| assert(metric > 0.8) | |||
| } | |||
|
|
|||
| test("Verify LightGBM Classifier updating learning_rate on training by using LightGBMDelegate") { | |||
|
|
|||
| class TrainDelegate extends LightGBMDelegate { | |||
There was a problem hiding this comment.
maybe the test is failing because this class is not serializable?
|
maybe add this to the class: https://alvinalexander.com/scala/how-to-use-serialization-in-scala-serializable-trait/ give it a @serialversionuid(100L) and "extends Serializable" |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@imatiach-msft Thanks to your comments and logs. So, Two things are fixed.
|
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@imatiach-msft Can I see error logs on Azure.mmlspark (UnitTests cognitive) and Azure.mmlspark (E2E)? |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@ocworld it looks like some new test infrastructure issue, let me see if @mhamilton723 might take a look since he is more familiar with the cognitive tests |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…ation in LightGBMDelegate (microsoft#849) * feat: add update learning_rate by using LightGBMDelegate * feat: add update learning_rate by using LightGBMDelegate * feat: add update learning_rate by using LightGBMDelegate * feat: add update learning_rate by using LightGBMDelegate * fix minor * fix serialization error * fix serialization error * change LightGBMDelegate to trait for scala style * change LightGBMDelegate to trait for scala style * change LightGBMDelegate to trait for scala style
Interface function is added for updating learning_rate per each iteration in LightGBMDelegate.
It is need if researcher want to decreasing or increasing learning rate per each boost round.