Getting "ValueError: setting an array element with a sequence" for user-defined splits on categorical variables

[hermanashley]

Hi there,

I'm using manual bins that were previously working and can't seem to figure out what I'm doing wrong. The issue only exists for categorical variables. If I define features/bins like so:

features = [ 'LOCATION']
 
 binning_parameters =  { 
        'LOCATION':{
                "dtype": 'categorical',
                "user_splits":[
                        ['Canada', 'USA', 'Mexico'],
                        ['Taiwan', 'China', 'India', 'Thailand', 'Malaysia']
                ],
                "user_splits_fixed": [False, False],
                "special_codes":['missing'],
               "min_prebin_size":10e-6,
    },
}

And I'm creating my binning process object like so:

categorical_variables = data[features].select_dtypes(
            include=["object", "category", "string"]
          ).columns.values

        return BinningProcess(
            categorical_variables=categorical_variables,
            variable_names=features,
            min_prebin_size=10e-6,
        )

I get the error when trying to fit the data (I've tried with both fit_transform() and fit():

binned_data = binning_process.fit_transform(
            X=data[features],
            y=data[target].tolist(),
            metric=metric
        )  

The full error is as follows:

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

I can't seem to figure out what I'm missing here. I'm using Optbinning 0.15.1 and python 3.8.

Thanks!

[hermanashley]

update: I was also using numpy version 1.24.1 - the binning parameters are passed in a format that was not supported by numpy. Downgrading to 1.23.1 fixes the issue for now.

[guillermo-navas-palencia]

Thanks, @hermanashley for reporting this issue. I will look into this.

[lassilehtonen]

Hi, I submitted a pull request #261 which should fix this issue. The function np.asarray() does not accept input like

'user_splits': [
  ['Canada', 'USA', 'Mexico'],
  ['Taiwan', 'China', 'India', 'Thailand', 'Malaysia']
 ]

where the bins have different lengths (3 and 5 in this case). Converting the list to pd.Series should work.

[guillermo-navas-palencia]

Hi. I think the solution is as simple as:

user_splits = np.array([
  ['Canada', 'USA', 'Mexico'],
  ['Taiwan', 'China', 'India', 'Thailand', 'Malaysia']
], dtype=object)

as shown in the tutorial:

Tested with numpy 1.24.1

[lassilehtonen]

That is true, I have changed the line in the PR.