Unknown objective function during predict, only in 1.3.0 RC (1.2.0 ok)

[pseudotensor]

We have a custom loss test, and when upgrading to 1.3.0 it started failing. It works perfectly fine in 1.2.0. The problem happens when doing:

1. fit in subprocess
2. restore pickled state in separate subprocess and predict

The predict hits:

/home/jon/h2oai.fullcondatest/xgboost/src/objective/objective.cc:26: Unknown objective function: `<function custom_objective_fair_linear at 0x7f711166c048>`
Objective candidate: survival:aft
Objective candidate: binary:hinge
Objective candidate: multi:softmax
Objective candidate: multi:softprob
Objective candidate: rank:pairwise
Objective candidate: rank:ndcg
Objective candidate: rank:map
Objective candidate: reg:squarederror
Objective candidate: reg:squaredlogerror
Objective candidate: reg:logistic
Objective candidate: reg:pseudohubererror
Objective candidate: binary:logistic
Objective candidate: binary:logitraw
Objective candidate: reg:linear
Objective candidate: count:poisson
Objective candidate: survival:cox
Objective candidate: reg:gamma
Objective candidate: reg:tweedie

Stack trace:
  [bt] (0) /home/jon/minicondadai/lib/python3.6/site-packages/xgboost/lib/libxgboost.so(xgboost::ObjFunction::Create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, xgboost::GenericParameter const*)+0x9e6) [0x7f6fce418846]
  [bt] (1) /home/jon/minicondadai/lib/python3.6/site-packages/xgboost/lib/libxgboost.so(xgboost::LearnerConfiguration::ConfigureObjective(xgboost::LearnerTrainParam const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*)+0x327) [0x7f6fce3c4bf7]
  [bt] (2) /home/jon/minicondadai/lib/python3.6/site-packages/xgboost/lib/libxgboost.so(xgboost::LearnerConfiguration::Configure()+0x593) [0x7f6fce3d1de3]
  [bt] (3) /home/jon/minicondadai/lib/python3.6/site-packages/xgboost/lib/libxgboost.so(xgboost::LearnerImpl::Predict(std::shared_ptr<xgboost::DMatrix>, bool, xgboost::HostDeviceVector<float>*, unsigned int, bool, bool, bool, bool, bool)+0x78) [0x7f6fce3bb028]
  [bt] (4) /home/jon/minicondadai/lib/python3.6/site-packages/xgboost/lib/libxgboost.so(XGBoosterPredict+0xf0) [0x7f6fce2b18d0]
  [bt] (5) /home/jon/minicondadai/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f71767b9630]
  [bt] (6) /home/jon/minicondadai/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7f71767b8fed]
  [bt] (7) /home/jon/minicondadai/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f71750eff9e]
  [bt] (8) /home/jon/minicondadai/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x139d5) [0x7f71750f09d5]

[hcho3]

@pseudotensor Can you post your code so that I can try running it on my end?

[pseudotensor]

Yes, slimming down to MRE

[pseudotensor]

But question: Why should predict ever need the custom loss function in first place?

[pseudotensor]

Hard to isolate to MRE. If I just put the predict call in a try-except and try again, there is no problem. So seems like problem is setting up the xgboost state for predict.

[pseudotensor]

For MRE, I've tried this basic thing:

# https://www.kaggle.com/c/allstate-claims-severity/discussion/24520
import numpy as np
def custom_objective_fair_linear(actual, pred):
    c = np.percentile(actual, 10)  # first decile as transition point
    if np.isnan(c):
        c = 2
    x = pred - actual
    c_denom = c / (np.abs(x) + c)
    grad = c_denom * x
    hess = c_denom ** 2
    return grad, hess

import xgboost as xgb

params = {'objective': custom_objective_fair_linear}
model = xgb.XGBRegressor(**params)
import pandas as pd
df = pd.read_csv("creditcard.csv")
target = "default payment next month"
y = df[target].values.ravel()
X = df.drop(target, axis=1)

model.fit(X, y)
preds = model.predict(X)

print(preds[0])

which works and trying to make it fail by following behavior of what we do in full code.

I couldn't make it fail with just subprocess stuff. I couldn't make it fail with separate files for fit/predict vs. where the custom loss function is defined.

[pseudotensor]

This is close to what we do, but it doesn't fail:

# https://www.kaggle.com/c/allstate-claims-severity/discussion/24520
import numpy as np

from modelfitpredict import fit_func, predict_func


def custom_objective_fair_linear(actual, pred):
    c = np.percentile(actual, 10)  # first decile as transition point
    if np.isnan(c):
        c = 2
    x = pred - actual
    c_denom = c / (np.abs(x) + c)
    grad = c_denom * x
    hess = c_denom ** 2
    return grad, hess


def func():
    fit_func(objective=custom_objective_fair_linear)
    predict_func()


from concurrent.futures import ProcessPoolExecutor
p = ProcessPoolExecutor(max_workers=1)
f = p.submit(func)
f.result()

# https://www.kaggle.com/c/allstate-claims-severity/discussion/24520
import xgboost as xgb
import pickle


def get_data():
    import pandas as pd
    df = pd.read_csv("creditcard.csv")
    target = "default payment next month"
    y = df[target].values.ravel()
    X = df.drop(target, axis=1)
    return X, y


def func():
    fit_func()
    predict_func()


def fit_func(objective="reg:linear"):
    print("fit")
    X, y = get_data()

    params = {'objective': objective, 'eval_metric': 'rmse'}
    model = xgb.XGBRegressor(**params)
    model.fit(X, y, eval_set=[(X, y)])

    pickle.dump(model, open('model.pkl', 'wb'))


def predict_func():
    print("predict")
    model = pickle.load(open('model.pkl', 'rb'))
    X, y = get_data()
    preds = model.predict(X)
    print(preds[0])

[trivialfis]

Not sure why the custom function ended up in xgboost string parameter. Could you please check that?

[trivialfis]

I have a guess for what's happening. You did something like this:

def logregobj(y_true, y_pred):
    y_pred = 1.0 / (1.0 + np.exp(-y_pred))
    grad = y_pred - y_true
    hess = y_pred * (1.0 - y_pred)
    return grad, hess


cls = xgb.XGBClassifier(use_label_encoder=False)
cls.fit(X, y)
cls.set_params(objective=logregobj)
cls.predict(X)

It was a bug in old version that parameter set after fit in skl interface are ignored until next fit. 1.3 fixed that so the custom objective got punched into xgboost internal as python string.

[trivialfis]

This can happen as the custom objective / string name of builtin objective handling logic is still on constructor.