[bug] Creating multiple boosters from file results in broken scores.

[AlbertoEAF]

Using the master branch, if I create 2 boosters from file using the C API, I get broken scores from all but the first created booster:

BoosterHandle bh1, bh2;  

LGBM_BoosterCreateFromModelfile("./LightGBM_model.txt", &num_iterations, &bh1); // comment this & you get good results
LGBM_BoosterCreateFromModelfile("./LightGBM_model.txt", &num_iterations, &bh2);

// Score is only correct if I predict with bh1, or predict with bh2 but don't initialize bh1 before initializing bh2:
predict(bh2, values             , NUM_FEATURES, num_iterations, &out_len, &ref_scores[0]); // score changes depending on bh2 being the first Booster created or not.

Meaning, only the first booster created works properly.

I'm on Ubuntu 18.04, x86_64.

Thank you

[shiyu1994]

@AlbertoEAF It seems that the predict function is not from the C API ? I failed to reproduce the problem with LGBM_BoosterPredictForFile. Loading two models with LGBM_BoosterCreateFromModelfile and then predict with LGBM_BoosterPredictForFile, the two boosters produce consistent results. So could you provide more details about predict function ? Or provide a complete reproducible example ? Thanks.

[AlbertoEAF]

Yes @shiyu1994 it's from this test code: https://gist.github.com/AlbertoEAF/5972db15a27c294bab65b97e1bc4c315 :)

If you modify that gist like the snippet above this happens.

[shiyu1994]

Sorry @AlbertoEAF , I still cannot reproduce the error. At least, for the attached LightGBM_model.txt file and the following code snippet, using the first and the second booster handle produce exactly the same result.

#include <iostream>
#include <stdio.h>
#include <math.h>
#include <vector>
#include <thread>
#include <ctime>
#include <cstring>
#include "LightGBM/c_api.h"

using namespace std;

#define FMT_HEADER_ONLY
#include "LightGBM/../../external_libs/fmt/include/fmt/format.h"

inline void predict(BoosterHandle handle,
            const void* data,            
            int32_t ncol,                                    
            int num_iterations,            
            int64_t* out_len,
            double* out_result) {
  if (0 != LGBM_BoosterPredictForMatSingleRow(
    handle, 
    data, 
    C_API_DTYPE_FLOAT64,
    ncol, 
    1, // is_row_major
    C_API_PREDICT_NORMAL, 
    0, // start_iteration
    num_iterations, 
    "", 
    out_len, 
    out_result)) {
      throw std::exception();
  }
}

int main(int /*argc*/, char** /*argv*/) {
  BoosterHandle boosterHandle1, boosterHandle2;
  int num_iterations;
  LGBM_BoosterCreateFromModelfile("./LightGBM_model.txt", &num_iterations, &boosterHandle1);
  LGBM_BoosterCreateFromModelfile("./LightGBM_model.txt", &num_iterations, &boosterHandle2);
  cout << "Model iterations " << num_iterations<< "\n";

  double values[] = { 
      0.0, 45.44200,-30.74976,31.78587,4.63569,-15.14894,0.23370,-11.97968,-9.59708,6.48111,-8.89073,4.02405,-2.28873,17.90204,1377.12225,
      1762.04257,947.34430,562.28140,524.26811,361.25410,514.53507,247.70126,399.78880,205.36521,211.86870,44.88971,-299.63923,-227.64169,
      7.34210,-85.80057,16.81211,-41.18581,-12.00858,-41.92117,-26.52312,-12.19765,89.61104,-266.07834,-230.31223,-168.29609,38.40371,31.84016,
      28.98071,-98.73447,26.46565,-30.23272,-1.34637,253.36945,61.73424,16.15765,185.24868,75.57244,55.17367,-41.08869,15.86438,16.03692,-142.57968,
      59.74796,-151.46761,-12.70612,-104.99242,23.10514,47.00179,-13.59894,-79.19045,-27.63745,38.94605,-55.40585,54.73527,15.10521,-3.79596,390.47720,
      17.64018,-68.64827,-62.43611,-31.96095,33.90780,-181.37609,139.84078,-129.48841,76.23826,-8.84046,-0.15439,137.44210,77.54739,-4.22875,-61.92657,
      -33.52722,-3.86253,36.42400,7.17309,
      0.0, 45.44200,-30.74976,31.78587,4.63569,-15.14894,0.23370,-11.97968,-9.59708,6.48111,-8.89073,4.02405,-2.28873,17.90204,1377.12225,
      1762.04257,947.34430,562.28140,524.26811,361.25410,514.53507,247.70126,399.78880,205.36521,211.86870,44.88971,-299.63923,-227.64169,
      7.34210,-85.80057,16.81211,-41.18581,-12.00858,-41.92117,-26.52312,-12.19765,89.61104,-266.07834,-230.31223,-168.29609,38.40371,31.84016,
      28.98071,-98.73447,26.46565,-30.23272,-1.34637,253.36945,61.73424,16.15765,185.24868,75.57244,55.17367,-41.08869,15.86438,16.03692,-142.57968,
      59.74796,-151.46761,-12.70612,-104.99242,23.10514,47.00179,-13.59894,-79.19045,-27.63745,38.94605,-55.40585,54.73527,15.10521,-3.79596,390.47720,
      17.64018,-68.64827,-62.43611,-31.96095,33.90780,-181.37609,139.84078,-129.48841,76.23826,-8.84046,-0.15439,137.44210,77.54739,-4.22875,-61.92657,
      -33.52722,-3.86253,36.42400,7.17309
  };
  const size_t NROWS=2;
  const int NUM_FEATURES = 91;
  double ref_scores[NUM_FEATURES * NROWS];

  int64_t dummy_out_len;

  predict(boosterHandle1, values, NUM_FEATURES, num_iterations, &dummy_out_len, &ref_scores[0]);
  fmt::print("Ref scores: {:.6g}\n", ref_scores[0]);
  predict(boosterHandle2, values, NUM_FEATURES, num_iterations, &dummy_out_len, &ref_scores[0]);
  fmt::print("Ref scores: {:.6g}\n", ref_scores[0]);
}

Could you please provide the LightGBM_model.txt file which produces the error? Thanks!
LightGBM_model.txt

[shiyu1994]

@AlbertoEAF Could you please recheck this issue and provide a reproducible case? We need to fix this before next release. Thank you! :)

[guolinke]

gentle ping @AlbertoEAF , for this issue blocks the release of 3.2.0.

[AlbertoEAF]

Hi there :),

Unfortunately I cannot provide that model file as it is sensitive, neither do I have much time on my hands to try to produce a simpler example this week. I can try it in the weekend though since it's urgent.

[AlbertoEAF]

Hello @shiyu1994 and @guolinke I managed to reproduce it. I created a new branch https://github.com/feedzai/LightGBM/tree/test-double-model-load which you can use (has the model and the profile code).

To test:

# Compile
mkdir build && cd build && cmake -DBUILD_PROFILING_TESTS=ON

# Go to main repo folder & run
cd ..
./lightgbm_profile_single_row_predict

you should get the following outputs, when they should be the same for both boosterHandles 1 & 2:

and if you comment the creation of boosterHandle1 and its predictions, boosterHandle2 starts spitting the results of boosterHandle1 in the screenshot above.

[shiyu1994]

@AlbertoEAF Thanks for your effort. It seems that the file profiling/profile_single_row_predict.cpp is missing from the branch. Can you please upload it ?

[AlbertoEAF]

Done! Sorry for that @shiyu1994 :)

[shiyu1994]

@AlbertoEAF I think I found the problem. Is your model file produced by an earlier version of LightGBM (earlier than the merge of linear_tree in #3299)? If so, the field is_linear= is missing from the model file. And when loaded by the latest version of LightGBM, the is_linear_ member of tree will be undefined, which causes undefined prediction behavior.

To fix this, let's set the is_linear_ filed to false when it is absent from the model file, for backward compatibility.

I'll open a PR for this.

[AlbertoEAF]

Wow, great catch @shiyu1994

Yes the model is older than that.

Thank you :)

[cyfdecyf]

I've also been biten by this bug when loading a model file without is_linear= set with the latest master code.

Since there's version= in the model file, I'd like to suggest adding a minior version number which increases whenever the model file format changes. When loading model file and found version mismatch, print a warning message to users to indicate it's possible to get wrong result.

[shiyu1994]

@cyfdecyf Thanks for the information. Does the error still happens after #4056 is merged ? I thought it should be already solved.

[cyfdecyf]

@shiyu1994 The error is fixed now. I'm planning to send a PR this morning and found that you've already fixed the problem.

I noticed predict result would change if I specify max_bin or label_column during prediction which is not expected. It took me several days to realize this is a memory related bug. I've tried both AddressSanitize and Valgrind to catch memory related errors on Linux but with no luck.

• For AddressSanitize, it's weird prediction result becomes all 0. But I remember it can't find uninitialized read so won't be helpful in this case.
• For valgrind, it didn't catch the uninitialized read either. Maybe I'm using it in the wrong way.

[shiyu1994]

@cyfdecyf How did you specify max_bin or label_column when prediction? Is the inconsistency of prediction results related to this bug?

[cyfdecyf]

@shiyu1994 The inconsistency is definitely related to this bug. I tried the following cases for my prediction:

1. Specify no max_bin, max_bin=63, max_bin=255 in config, the latter two has same results. It only matters whether you specify max_bin or not.
2. Specify label_column on cmdline, in config, on cmdline and also in config, the result differs.

[shiyu1994]

@cyfdecyf Could you provide a reproducible example? Thanks.

[cyfdecyf]

@shiyu1994 Spent sometime to construct a simple example. I'm not able to reproduce the exact symptoms I encounted with this simple example, only num_threads will impact the prediction result.

Here's the steps:

Build 2 lightgbm binaries with commit fcfd413 and its previous commit d90a16d

• The bug is introduced in commit fcfd413
• Name the 2 binaries as lightgbm.fcfd4132 and lightgbm.d90a16d5
• I'm compiling the binary on Ubuntu 18.04 with g++ 7.5.0

Using the regression example:

cd examples/regression
ln -s ../../lightgbm.fcfd413 lightgbm.buggy
ln -s ../../lightgbm.d90a16d lightgbm.old

./lightgbm.old config=train.conf
./lightgbm.old config=predict.conf num_threads=1 deterministic=true output_result=old-1.txt
./lightgbm.old config=predict.conf num_threads=8 deterministic=true output_result=old-8.txt
./lightgbm.old config=predict.conf num_threads=16 deterministic=true output_result=old-16.txt
./lightgbm.buggy config=predict.conf num_threads=1 deterministic=true output_result=buggy-1.txt
./lightgbm.buggy config=predict.conf num_threads=8 deterministic=true output_result=buggy-8.txt
./lightgbm.buggy config=predict.conf num_threads=16 deterministic=true output_result=buggy-16.txt
md5sum old-*.txt buggy-*.txt

Here's the md5sum output on my server:

08888ca63a5500113856f2d6759aba50  old-1.txt
08888ca63a5500113856f2d6759aba50  old-8.txt
08888ca63a5500113856f2d6759aba50  old-16.txt
4e51234584f2a4c19a587cf298e8d792  buggy-1.txt
843753a4c9589969619c3c8e45f42010  buggy-8.txt
51e5da0cafaaf50ac763fcf71c28ee87  buggy-16.txt

If I change the train step to ./lightgbm.buggy config=train.conf (with the model file contianing is_linear=0), then all prediction results will be the same.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.