You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It may seem harmless at first glance as we are providing grad and hess during iteration. But it actually makes the output incorrect by having the is_constant_hessian to true all the time (which is the case for unweighted regression), and cause the calculated sum_hessian split incorrect, which eventually leads to wrong model output.
The fix I propose here is simply to pass objective=custom rather than no objective in param when provided fobj.
I'm pretty new to Scala and Java as this is the first Scala project I've seen. I've tested things in sbt console. But not sure how to setup the environment properly as something fails when I run sbt test on master without any change. Would be really appreciate if anyone can point me to any updated doc I can reference. (the runme link in Contribute.md leads to 404).
Could also include code similar with the code below (which I used to debug the mismatch) as a test case if needed. You may also use the same code to verify the bug.
import com.microsoft.azure.synapse.ml.lightgbm.dataset.LightGBMDataset
import com.microsoft.azure.synapse.ml.lightgbm.params.FObjTrait
import com.microsoft.azure.synapse.ml.lightgbm._
import org.apache.spark.ml.feature.VectorAssembler
import scala.math.exp
import scala.util.Random
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().config("spark.master", "local[*]").getOrCreate()
import spark.implicits._
val rng = new Random(1206)
val x:Seq[Double] = for (i <- 0 to 99) yield (rng.nextDouble)
val y = for (i <- 0 to 99) yield (if (rng.nextDouble > x(i)) 1 else 0)
val df = (x.zip(y).map {case (x,y) => (x,y,0)}).toDF("feature", "label", "init_score")
val converted_df = new VectorAssembler().setInputCols(Array("feature")).setOutputCol("features").transform(df)
converted_df.show()
converted_df.registerTempTable("temp_table")
def baseModel: LightGBMClassifier = {
new LightGBMClassifier()
.setFeaturesCol("features")
.setNumLeaves(2)
.setLambdaL2(0)
.setLearningRate(0.1)
.setMinGainToSplit(0)
.setLabelCol("label")
.setNumTasks(1)
.setMaxDepth(1)
.setVerbosity(1)
.setNumThreads(1)
.setMinDataInLeaf(20)
.setInitScoreCol("init_score")
}
class CrossEntropy extends FObjTrait {
override def getGradient(predictions: Array[Array[Double]],
trainingData: LightGBMDataset): (Array[Float], Array[Float]) = {
// Get the labels
val labels = trainingData.getLabel()
// Compute gradient and hessian
val probabilities = predictions.map(rowPrediction =>
rowPrediction.map(prediction => 1.0 / (1.0 + exp(-prediction))))
val grad = probabilities.zip(labels).map {
case (prob: Array[Double], label: Float) => (prob(0)-label).toFloat
}
val hess = probabilities.map(probabilityArray => (probabilityArray(0) * (1 - probabilityArray(0))).toFloat)
(grad, hess)
}
}
baseModel.setNumIterations(1).setObjective("binary").fit(converted_df).transform(converted_df).show(10, false)
baseModel.setNumIterations(1).setFObj(new CrossEntropy()).fit(converted_df).transform(converted_df).show(10, false)
val native_model = baseModel.setNumIterations(3).setObjective("binary").fit(converted_df)
val native_res = native_model.transform(converted_df)
native_res.show(10, false)
// For binary classification: cross entropy is equivalent with binary.
val fobj_model = baseModel.setNumIterations(3).setFObj(new CrossEntropy()).fit(converted_df)
val fobj_res = fobj_model.transform(converted_df)
fobj_res.show(10, false)
native_res.select("rawPrediction").except(fobj_res.select("rawPrediction")).count == 0
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently there will be no objective param if a fobj is provided.
That caused the LightGBM mistakenly think it's regression during initial setup as that's the [default value] for it(https://github.com/microsoft/LightGBM/blob/master/include/LightGBM/config.h#L139).
It may seem harmless at first glance as we are providing grad and hess during iteration. But it actually makes the output incorrect by having the
is_constant_hessianto true all the time (which is the case for unweighted regression), and cause the calculated sum_hessian split incorrect, which eventually leads to wrong model output.The fix I propose here is simply to pass
objective=customrather than no objective in param when provided fobj.I'm pretty new to Scala and Java as this is the first Scala project I've seen. I've tested things in sbt console. But not sure how to setup the environment properly as something fails when I run sbt test on master without any change. Would be really appreciate if anyone can point me to any updated doc I can reference. (the runme link in Contribute.md leads to 404).
Could also include code similar with the code below (which I used to debug the mismatch) as a test case if needed. You may also use the same code to verify the bug.