Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python Custom Multiclass Loss Function Doesn't Work #1644

Closed
wagnew3 opened this issue Sep 6, 2018 · 4 comments
Assignees

Comments

@wagnew3
Copy link

@wagnew3 wagnew3 commented Sep 6, 2018

Custom multiclass loss functions in python don't work-- LightGBM doesn't seem to be learning anything.
In the example code below on the iris dataset, I train lightgbm with the default multiclass loss function and get a training accuracy of 98%. When I train lightgbm on a "custom" loss function (the same multiclass loss), I get a training accuracy of 0.099% (basically random). However, when I train xgboost normally and with a "custom" multiclass loss, both give same training accuracy of 99%, as expected.

Environment info

Operating System: Ubuntu 16.04
CPU: Intel® Core™ i7-5930K
C++/Python/R version: Python 3.5.2
LightGBM 2.1.2

Error message

lightgbm train correct %: 0.9827490261547023
lightgbm custom loss train correct %: 0.09905397885364496
xgboost train error %: 0.9994435169727324
xgboost custom loss train error %: 0.9994435169727324

Reproducible examples

import numpy as np
from sklearn import datasets
from sklearn.preprocessing import OneHotEncoder
import xgboost as xgb
import lightgbm as lgb

#custom loss function
def logregobj_lgb(preds, train_data):
    labels = train_data.get_label()
    preds=np.reshape(preds, (len(labels), 10))
    labels = OneHotEncoder(n_values=10, sparse=False).fit_transform(labels.reshape(-1, 1))
    grad = (preds - labels)
    hess = 2.0 * preds * (1.0-preds)
    return grad.flatten('F'), hess.flatten('F')

def logregobj_xgb(preds, train_data):
    labels = train_data.get_label()
    preds=np.reshape(preds, (len(labels), 10))
    labels = OneHotEncoder(n_values=10, sparse=False).fit_transform(labels.reshape(-1, 1))
    grad = (preds - labels)
    hess = 2.0 * preds * (1.0-preds)
    return grad.flatten(), hess.flatten()

#Digits dataset
digits = datasets.load_digits()
X, Ymc = digits.data, digits.target
Y = OneHotEncoder(sparse=False).fit_transform(Ymc.reshape(-1, 1))

#train lightgbm with default multiclass loss function
params={'num_class':len(np.unique(Ymc)), 'objective': 'multiclass', 'verbose':-1, 'num_leaves': 63, 'num_iterations': 10}
lgb_train = lgb.Dataset(X, Ymc)
bst=lgb.train(params, lgb_train, verbose_eval=False)
preds=bst.predict(X)
pred_labels=np.argmax(preds, axis=1)
train_error=np.sum(pred_labels==Ymc)
#accuracy
print('lightgbm train correct %:', train_error/Ymc.shape[0])

#train lightgbm with custom loss function
params={'num_class':len(np.unique(Ymc)), 'objective': 'multiclass', 'verbose':1, 'num_leaves': 63, 'num_iterations': 10}
lgb_train = lgb.Dataset(X, Ymc)
bst=lgb.train(params, lgb_train, verbose_eval=False, fobj=logregobj_lgb)
preds=bst.predict(X)
pred_labels=np.argmax(preds, axis=1)
train_error=np.sum(pred_labels==Ymc)
#accuracy
print('lightgbm custom loss train correct %:', train_error/Ymc.shape[0])

#train xgboost
dtrain = xgb.DMatrix(X, label=Ymc)
param = {'max_depth': '6', 'objective':'multi:softprob', 'num_class':len(np.unique(Ymc)), 'tree_method':'hist', 'silent':1}
model = xgb.Booster(params, [dtrain])
bst = xgb.train(param, dtrain, num_boost_round=10)
preds=bst.predict(dtrain)
pred_labels=np.argmax(preds, axis=1)
train_error=np.sum(pred_labels==Ymc)
#accuracy
print('xgboost train error %:', train_error/Ymc.shape[0])

#train xgboost with same custom loss function
dtrain = xgb.DMatrix(X, label=Ymc)
param = {'max_depth': '6', 'objective':'multi:softprob', 'num_class':len(np.unique(Ymc)), 'tree_method':'hist', 'silent':1}
model = xgb.Booster(params, [dtrain])
bst = xgb.train(param, dtrain, num_boost_round=10, obj=logregobj_xgb)
preds=bst.predict(dtrain)
pred_labels=np.argmax(preds, axis=1)
train_error=np.sum(pred_labels==Ymc)
#accuracy
print('xgboost custom loss train error %:', train_error/Ymc.shape[0])
@guolinke guolinke self-assigned this Sep 27, 2018
@guolinke

This comment has been minimized.

Copy link
Member

@guolinke guolinke commented Sep 27, 2018

@wagnew3 thanks, it seems this is a bug. I will fix it later.

@guolinke guolinke added bug and removed bug labels Sep 27, 2018
@guolinke

This comment has been minimized.

Copy link
Member

@guolinke guolinke commented Sep 28, 2018

@wagnew3
The problem is you are using the zero hessians.
you can try:

def logregobj_lgb(preds, train_data):
    labels = train_data.get_label()
    preds=np.reshape(preds, (len(labels), 10))
    labels = OneHotEncoder(n_values=10, sparse=False).fit_transform(labels.reshape(-1, 1))
    grad = (preds - labels)
    hess = 2.0 * preds * (1.0-preds) + 1e-6
    return grad.flatten('F'), hess.flatten('F')
@guolinke guolinke closed this Sep 28, 2018
@sergey-lebedev

This comment has been minimized.

Copy link

@sergey-lebedev sergey-lebedev commented Oct 17, 2018

You need transforming preds

def logregobj_lgb(preds, train_data):
     labels = train_data.get_label()
     preds=np.reshape(preds, (len(labels), 10))
     preds = np.exp(preds)
     preds = np.multiply(preds, 1/np.sum(preds, axis=1)[:, np.newaxis])
     labels = OneHotEncoder(n_values=10, sparse=False).fit_transform(labels.reshape(-1, 1))
     grad = (preds - labels)
     hess = 2.0 * preds * (1.0-preds)
     return grad.flatten('F'), hess.flatten('F')
@jrzaurin

This comment has been minimized.

Copy link

@jrzaurin jrzaurin commented Nov 9, 2019

I know this issue has been close since forever, but I am wondering, should not the line:

preds=np.reshape(preds, (len(labels), 10))

be

preds=np.reshape(preds, (len(labels), 10), order='F')?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.