[python] added get_data() method to Dataset class

[StrikerRUS]

Fixed #1690.
Fixed #1763.

[StrikerRUS]

What about string and list of numpy arrays? How to "slice" them?..

[guolinke]

I think we cannot slide them, maybe we should throw a warning or error here.

[StrikerRUS]

Simplified example from #1763

import numpy as np
import pandas as pd
import lightgbm as lgb

full_data = pd.DataFrame({'x1': np.random.rand(100),
                          'x2': 5 + np.random.rand(100),
                          'target': np.random.randint(0, 2, 100)})

data_train = lgb.Dataset(full_data.drop(['target'], axis=1), full_data.target, free_raw_data=False)

def fpreproc_rebalance(dtrain, dtest, params,):
    train_data = dtrain.get_data()
    test_data = dtest.get_data()
    train_data.loc[train_data['x1'] > 0.5, ['x1']] = 0.5
    test_data.loc[test_data['x1'] > 0.5, ['x1']] = 0.5
    fdtrain = lgb.Dataset(train_data, dtrain.label, free_raw_data=False)
    fdtest = lgb.Dataset(test_data, dtest.label, free_raw_data=False)
    return fdtrain, fdtest, params

results = lgb.cv({}, data_train, fpreproc=fpreproc_rebalance)

fails with

LightGBMError: Cannot add validation data, since it has different bin mappers with training data

[guolinke]

maybe a better logic still pass None for data here?
Then in construct, use reference.data to get_data ?
Then, the logic of construct is the same as before.

[StrikerRUS]

Reworked

[StrikerRUS]

Now this snippet should be modified with construct() like the following:

...
def fpreproc_rebalance(dtrain, dtest, params,):
    train_data = dtrain.construct().get_data()
    test_data = dtest.construct().get_data()
    ...

[guolinke]

maybe we don't need the get_data here ?

[StrikerRUS]

I thought that we need a consistent state of the object after calling construct().

Or do you think that it's unneeded and we should allow users to access data only via get_data() (remove mentions about data field)?

[guolinke]

okay, I see. As only subset will call the slicing, I think the overhead cost is acceptable.
Change the definition of data may cause many dependency problems.

[StrikerRUS]

Yeah, it's expected that only subsetting branch of construct calls get_data(). Also, get_data() itself checks used_indices:
https://github.com/Microsoft/LightGBM/blob/1ff6c6d6bca05de2c13490264ba3b6bf7f993d4a/python-package/lightgbm/basic.py#L1392

[StrikerRUS]

@guolinke please help with this.

[guolinke]

@StrikerRUS I think that will not fail by the latest code

[StrikerRUS]

@guolinke Unfortunately, it still fails (data branch with dll file from latest master):

import numpy as np
import pandas as pd
import lightgbm as lgb

full_data = pd.DataFrame({'x1': np.random.rand(100),
                          'x2': 5 + np.random.rand(100),
                          'target': np.random.randint(0, 2, 100)})

data_train = lgb.Dataset(full_data.drop(['target'], axis=1), full_data.target, free_raw_data=False)

def fpreproc_rebalance(dtrain, dtest, params,):
    train_data = dtrain.construct().get_data()
    test_data = dtest.construct().get_data()
    train_data.loc[train_data['x1'] > 0.5, ['x1']] = 0.5
    test_data.loc[test_data['x1'] > 0.5, ['x1']] = 0.5
    fdtrain = lgb.Dataset(train_data, dtrain.label, free_raw_data=False)
    fdtest = lgb.Dataset(test_data, dtest.label, free_raw_data=False)
    return fdtrain, fdtest, params

results = lgb.cv({}, data_train, fpreproc=fpreproc_rebalance)

LightGBMError: Cannot add validation data, since it has different bin mappers with training data

[guolinke]

@StrikerRUS you need to set the reference for valid data:

    fdtrain = lgb.Dataset(train_data, dtrain.label, free_raw_data=False)
    fdtest = lgb.Dataset(test_data, dtest.label, free_raw_data=False, reference=fdtrain )

[StrikerRUS]

@guolinke Yeah, thanks! Completely forgot about this. Now everything is OK.

Maybe add this snippet as a regression test?

[guolinke]

@StrikerRUS yeah, sure!

[StrikerRUS]

@guolinke Test has been added. Please check.