Pairwise model have different performance with python api with the same configurations

[jinmfeng001]

Describe the bug
We trained a pairwise model using LightGBMRanker with below settings, the model performance is poor, it event didn't split the records correctly(about 98% records in a single leaf node). But when we train the model with python lib with the same settings, it works fine.

new LightGBMRanker()
        .setObjective("lambdarank")
        .setMetric("ndcg")
        .setGroupCol("QueryID")
        .setFeaturesCol("features")
        .setLabelCol("target")
        .setPredictionCol("predictedScore")
        .setMinDataInLeaf(500)
        .setBaggingFraction(0.5)
        .setMaxDepth(8)
        .setNumIterations(100)
        .setLearningRate(0.3)
        .setNumTasks(1)
        .setIsProvideTrainingMetric(true)

tree infos:

Tree=0
num_leaves=31
num_cat=0
split_feature=4 22 29 4 25 18 26 2 2 3 25 23 22 25 17 25 20 19 29 29 2 19 18 18 6 1 26 29 23 26
split_gain=5.70835 6.27817 8.08507 5.59035 7.62217 8.23141 5.67411 5.48662 7.59295 5.81305 5.41236 10.088 6.70688 13.7927 7.29927 6.12288 7.77411 5.9368 5.23978 6.28279 4.81717 4.77172 4.6542 6.10098 6.72341 4.70862 4.68191 9.06013 5.98668 5.88347
threshold=0.6135888695716859 -7.2360394001007071 0.33265334367752081 0.31386297941207891 0.73309999704360973 38296892.000000007 -0.052871471270918839 0.25866188108921057 0.22967033088207248 0.99999997019767772 0.42962500452995306 0.74643373489379894 -7.6787121295928946 0.12470000237226488 1.5790685415267947 0.53870850801467907 -5.0241968631744376 0.51499998569488536 0.75568181276321422 0.68257576227188121 0.27386364340782171 0.28499999642372137 182613240.00000003 94961176.000000015 293.50000000000006 6.5000000000000009 2.1313939094543461 0.79616478085517894 0.88841971755027782 -0.2136563956737518
decision_type=2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
left_child=1 2 -1 4 18 6 -6 8 -3 -9 12 -12 14 -14 17 -13 -17 -5 19 21 -20 -4 23 26 -25 -26 28 -28 29 -2
right_child=22 7 3 10 5 -7 -8 9 -10 -11 11 15 13 -15 -16 16 -18 -19 20 -21 -22 -23 -24 24 25 -27 27 -29 -30 -31
leaf_value=0.079347574053907063 0.0043328297349718749 1.9967691120491542e-05 -0.028972150932266603 0.01953118682384956 -0.069643881458470239 -0.13143316934759106 0.015339967128005793 -0.069572184034506893 0.074866666517422464 -0.003122346569939663 0.14329669717441659 0.10787757290536579 0.14999806278681718 -0.041112160595744685 0.064277012565171238 0.062065755476571005 -0.046508163546626582 -0.088076156595065333 0.0065386729383173818 0.12307585830469921 -0.075571868842268436 0.058466584738766078 -0.082578632649927264 -0.045524579382198359 0.027392266606243731 0.12536116415453316 0.021253962405801876 -0.12961177307994712 0.0062070550275559744 -0.033830592539434073
leaf_weight=162.54927922211937 516.11397882670281 392277.32008506852 96.814097145048436 71.822141570854001 112.26719534327276 68.250760232331231 191.00827324338024 209.40532596869161 122.0233643213287 272.88524081069045 91.51321722031571 69.89176998147741 72.794390122260666 63.75518068112433 67.735530590405688 116.72653960809112 120.75351655390114 129.0621189288795 1008.0694646562042 72.371563006425276 68.685221546096727 133.80073021931457 83.046475585084409 71.589700516778976 124.53649350069463 68.405183962342562 61.457351172342896 85.90026979096001 1039.2930430820707 1230.030136897738
leaf_count=1283 4285 3355997 814 587 995 608 1570 1833 925 2241 697 520 543 536 533 939 1062 1107 8358 565 595 1081 738 592 1016 531 478 783 8595 10553
internal_value=0 0.000103341 0.0144723 0.0103455 0.000837588 -0.0373031 -0.0161195 3.93891e-06 4.32426e-05 -0.0319741 0.0310541 0.0558607 0.00663233 0.0607684 -0.0208871 0.0298285 0.00685825 -0.0496033 0.0111078 0.0459597 0.00130091 0.021759 -0.012463 -0.0106418 0.0329927 0.062126 -0.0145775 -0.0666913 -0.0118206 -0.0225505
internal_weight=0 395600 2717.87 2555.32 1751.27 371.526 303.275 392882 392399 482.291 804.054 398.885 405.169 136.55 268.62 307.372 237.48 200.884 1379.74 302.986 1076.75 230.615 3280.37 3197.33 264.531 192.942 2932.79 147.358 2785.44 1746.14
internal_count=3410960 3383389 22393 21110 14586 3173 2565 3360996 3356922 4074 6524 3218 3306 1079 2227 2521 2001 1694 11413 2460 8953 1895 27571 26833 2139 1547 24694 1261 23433 14838
is_linear=0
shrinkage=0.3

...

Tree=99
num_leaves=12
num_cat=0
split_feature=15 21 8 20 18 18 16 22 13 22 30
split_gain=3.1816 3.35391 2.5403 1.89021 6.8107 5.20073 4.34468 2.70004 2.21346 8.93018 2.05589
threshold=322.61499023437506 476.50000000000006 20.000000000000004 -6.122980833053588 9018408.0000000019 823515.00000000012 -9.8687191009521467 -11.540429592132567 60296.000000000007 -7.0380601882934561 2.5000000000000004
decision_type=2 2 2 2 2 2 2 2 2 2 2
left_child=1 2 3 4 5 -1 -5 -8 10 -10 -9
right_child=-2 -3 -4 6 -6 -7 7 8 9 -11 -12
leaf_value=0.0093251369358115354 -0.061213449598266335 0.061138198447487371 -0.060388819936791155 0.067141073209400715 -0.021630632009306691 0.093367529913281247 -0.056010133285471046 3.7802852927126579e-05 0.026310694199394531 -0.088008121994765473 -0.012004266329944628
leaf_weight=134.11789747304283 76.403518556151539 80.768956777639687 62.684197120834142 86.688135979464278 208.38180961180478 130.99452743958682 77.521699623204768 408543.57700718124 128.40825170121389 118.02491908031516 1279.9690699464409
leaf_count=498 352 364 226 359 897 522 326 3401392 538 513 4973
internal_value=0 1.13794e-05 -6.39861e-07 8.57688e-06 0.0189524 0.0508513 -1.32882e-05 -2.74819e-05 -1.68986e-05 -0.0284403 1.92822e-07
internal_weight=0 410851 410770 410708 473.494 265.112 410234 410148 410070 246.433 409824
internal_count=3410960 3410608 3410244 3410018 1917 1020 3408101 3407742 3407416 1051 3406365
is_linear=0
shrinkage=0.3

To Reproduce
The parameters from the model dump file are:

parameters:
[boosting: gbdt]
[objective: lambdarank]
[metric: ndcg]
[tree_learner: serial]
[device_type: cpu]
[linear_tree: 0]
[data: ]
[valid: ]
[num_iterations: 100]
[learning_rate: 0.3]
[num_leaves: 31]
[num_threads: 1]
[deterministic: 0]
[force_col_wise: 0]
[force_row_wise: 0]
[histogram_pool_size: -1]
[max_depth: 8]
[min_data_in_leaf: 500]
[min_sum_hessian_in_leaf: 0.001]
[bagging_fraction: 0.5]
[pos_bagging_fraction: 1]
[neg_bagging_fraction: 1]
[bagging_freq: 0]
[bagging_seed: 3]
[feature_fraction: 1]
[feature_fraction_bynode: 1]
[feature_fraction_seed: 2]
[extra_trees: 0]
[extra_seed: 6]
[early_stopping_round: 0]
[first_metric_only: 0]
[max_delta_step: 0]
[lambda_l1: 0]
[lambda_l2: 0]
[linear_lambda: 0]
[min_gain_to_split: 0]
[drop_rate: 0.1]
[max_drop: 50]
[skip_drop: 0.5]
[xgboost_dart_mode: 0]
[uniform_drop: 0]
[drop_seed: 4]
[top_rate: 0.2]
[other_rate: 0.1]
[min_data_per_group: 100]
[max_cat_threshold: 32]
[cat_l2: 10]
[cat_smooth: 10]
[max_cat_to_onehot: 4]
[top_k: 20]
[monotone_constraints: ]
[monotone_constraints_method: basic]
[monotone_penalty: 0]
[feature_contri: ]
[forcedsplits_filename: ]
[refit_decay_rate: 0.9]
[cegb_tradeoff: 1]
[cegb_penalty_split: 0]
[cegb_penalty_feature_lazy: ]
[cegb_penalty_feature_coupled: ]
[path_smooth: 0]
[interaction_constraints: ]
[verbosity: -1]
[saved_feature_importance_type: 0]
[max_bin: 255]
[max_bin_by_feature: ]
[min_data_in_bin: 3]
[bin_construct_sample_cnt: 200000]
[data_random_seed: 1]
[is_enable_sparse: 1]
[enable_bundle: 1]
[use_missing: 1]
[zero_as_missing: 0]
[feature_pre_filter: 1]
[pre_partition: 1]
[two_round: 0]
[header: 0]
[label_column: ]
[weight_column: ]
[group_column: ]
[ignore_column: ]
[categorical_feature: ]
[forcedbins_filename: ]
[objective_seed: 5]
[num_class: 1]
[is_unbalance: 0]
[scale_pos_weight: 1]
[sigmoid: 1]
[boost_from_average: 1]
[reg_sqrt: 0]
[alpha: 0.9]
[fair_c: 1]
[poisson_max_delta_step: 0.7]
[tweedie_variance_power: 1.5]
[lambdarank_truncation_level: 30]
[lambdarank_norm: 1]
[label_gain: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]
[eval_at: ]
[multi_error_top_k: 1]
[auc_mu_weights: ]
[num_machines: 1]
[local_listen_port: 12400]
[time_out: 120]
[machine_list_filename: ]
[machines: ]
[gpu_platform_id: -1]
[gpu_device_id: -1]
[gpu_use_dp: 0]
[num_gpu: 1]

Expected behavior
A clear and concise description of what you expected to happen.

Info (please complete the following information):

• SynapseML Version: v0.9.5
• Spark Version: 3.2.0
• Spark Platform: apache

** Stacktrace**

Please post the stacktrace here if applicable

Additional context
Add any other context about the problem here.

AB#1767712

[imatiach-msft]

@jinmfeng001 could you provide some sample dataset (maybe some dummy ranking data that looks similar to your actual data and which can be run on synapseml and python versions of lightgbm ranker) that we could reproduce the issue on? This could be happening for a variety of reasons (maybe some different parameters? or possibly even some bug?), but without a way to reproduce the issue it's hard to diagnose. Do you notice any difference if you set the parameter setUseSingleDatasetMode(false)? That was the biggest change in the recent past, but I'm not sure if it's related to your issue.

[imatiach-msft]

@jinmfeng001 I think this may actually be related to the issue:
#1478
can you please try the new build:

Maven Coordinates
com.microsoft.azure:synapseml_2.12:0.9.5-92-76c32ccf-SNAPSHOT

Maven Resolver
https://mmlspark.azureedge.net/maven

to see if it may fix your issue? Thank you!

[jinmfeng001]

@imatiach-msft Thanks for your update, I tried com.microsoft.azure:synapseml_2.12:0.9.5-92-76c32ccf-SNAPSHOT, and the issue was resolved.

thanks for your help.