fix: correct LightGBM improvement tolerance semantics

[ranadeepsingh]

Related Issues/PRs

Fixes #2565

What changes are proposed in this pull request?

Correct LightGBM early-stopping behavior when improvementTolerance is positive for lower-is-better metrics such as RMSE and MAE.

The previous comparison treated small regressions as improvements and reset the early-stopping counter. This change makes the comparison symmetric with higher-is-better metrics: the metric must improve by more than improvementTolerance before the counter resets.

The patch also:

• preserves LightGBM's earlyStoppingRound = 0 behavior, which disables early stopping;
• rejects negative stopping rounds, negative tolerances, and NaN tolerances;
• clarifies both parameter descriptions.

Focused tests cover 26 representative lower-is-better and higher-is-better metric names across zero, small, medium, and large tolerances. They also cover exact tolerance boundaries, insufficient improvements, regressions, default values, disabled stopping, positive stopping-round boundaries, and invalid inputs.

How is this patch tested?

• I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Commands:

lightgbm/Test/compile
lightgbm/testOnly com.microsoft.azure.synapse.ml.lightgbm.split1.TrainUtilsSuite
lightgbm/scalastyle
lightgbm/Test/scalastyle

The existing native LightGBM integration test was also attempted locally, but the packaged lib_lightgbm.dylib could not be loaded on macOS.

Azure Pipelines build 228836638 passed all six LightGBM Scala unit-test jobs and the LightGBM R test job. The overall build failed for unrelated infrastructure/dependency issues affecting multiple components: Maven Central returned HTTP 429 during setup, and Python test collection failed on an mlflow/google.protobuf incompatibility before LightGBM Python tests executed.

Failed-job retry update: UnitTests recommendation passed on attempt 2, confirming the Maven HTTP 429 was transient. The six shared Python jobs still failed during collection on the same MLflow/protobuf incompatibility, and the Databricks GPU cluster again timed out while resizing. These repository-wide baseline failures are handled separately by #2580 and #2579. PR #2580 has now passed full Azure build 228877170, including the LightGBM Python tests; PR #2579 independently passed its Databricks GPU job. This PR remains dependency-free and focused on #2565.

Does this PR change any dependencies?

• No. You can skip this section.
• Yes. Make sure the dependencies are resolved correctly, and list changes here.

Does this PR add a new feature? If so, have you added samples on website?

• No. You can skip this section.
• Yes. Make sure you have added samples following below steps.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey @ranadeepsingh 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Copilot]

Pull request overview

Fixes LightGBM early-stopping “improvementTolerance” handling for lower-is-better metrics (e.g., RMSE/MAE) so that the early-stopping counter only resets when the metric improves by more than the configured tolerance, matching the semantics already used for higher-is-better metrics.

Changes:

• Introduces a shared TrainUtils.isImprovement helper to apply symmetric improvement logic for higher- vs lower-is-better metrics.
• Updates validation-evaluation early-stopping logic to use the new improvement predicate.
• Adds focused unit tests and clarifies the improvementTolerance parameter description.

Show a summary per file

File	Description
lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/TrainUtils.scala	Adds isImprovement and updates early-stopping comparison to require improvement exceeding tolerance for both metric directions.
lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split1/TrainUtilsSuite.scala	Adds unit tests covering lower-/higher-is-better metrics, boundary behavior, and zero-tolerance defaults.
lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/params/LightGBMParams.scala	Clarifies improvementTolerance parameter documentation to reflect the corrected behavior.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Low

[ranadeepsingh]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.62%. Comparing base (9e738e0) to head (20630db).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2578      +/-   ##
==========================================
- Coverage   84.79%   84.62%   -0.17%     
==========================================
  Files         334      334              
  Lines       17801    17806       +5     
  Branches     1619     1623       +4     
==========================================
- Hits        15094    15068      -26     
- Misses       2707     2738      +31     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ranadeepsingh]

Expanded validation after reviewing parameter interactions:

• All six Azure Pipelines LightGBM Scala unit-test jobs passed.
• LightGBM R tests passed.
• Local LightGBM test compilation, focused matrix tests, and Scala style checks passed.
• The full build failed for unrelated shared-pipeline issues: Maven Central HTTP 429s and an mlflow/google.protobuf import incompatibility that prevented multiple Python suites (including LightGBM) from collecting tests.

Build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=228836638

[ranadeepsingh]

Retried only the failed jobs in Azure build 228836638. The recommendation job recovered, while the shared Python collection failures and GPU resize timeout reproduced.

The baseline fixes are now isolated and validated:

• ci: pin MLflow for protobuf compatibility AB#5480878 #2580 pins MLflow; full Azure build 228877170 passed, including PythonTests lightgbm and every other Python job.
• ci: guard Databricks T4 GPU pool migration AB#5478524 #2579 hardens the Databricks GPU pool contract; its GPU E2E job passed.

No unrelated CI dependency or infrastructure changes were added to this LightGBM bug-fix PR.

[ranadeepsingh]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).