ValueError: operands could not be broadcast together with shapes (1,2) (1,14)

[shrija2901]

Problem Statement:

1. I am trying to fit a dataset with categorical variables, similar to adult dataset example.
2. However I want to fit the explainer model for multiple datasets with same number of features
3. During the fit of the model I am encountering the below error, detailed error points to probability calculation with X_hat

ValueError                                Traceback (most recent call last)
<command-3456335270115933> in <cell line: 8>()
     64 
     65   print("*******************fit explainer********************************")
---> 66   explainer1 = explainer.fit(X=group2.values) #group2.values
     67   print('explainer1 complete')
     68 

/local_disk0/.ephemeral_nfs/envs/pythonEnv-55e14a0e-51a9-4bf3-8516-341b53dfa255/lib/python3.9/site-packages/alibi/explainers/cfrl_tabular.py in fit(self, X)
    277 
    278         # call base class fit
--> 279         return super().fit(X)
    280 
    281     def explain(self,  # type: ignore[override]

/local_disk0/.ephemeral_nfs/envs/pythonEnv-55e14a0e-51a9-4bf3-8516-341b53dfa255/lib/python3.9/site-packages/alibi/explainers/cfrl_base.py in fit(self, X)
    679             for pp_func in self.params["postprocessing_funcs"]:
    680                 # Post-process noised counterfactual.
--> 681                 X_cf_tilde = pp_func(self.backend.to_numpy(X_cf_tilde),
    682                                      self.backend.to_numpy(data["X"]),
    683                                      self.backend.to_numpy(data["C"]))

/local_disk0/.ephemeral_nfs/envs/pythonEnv-55e14a0e-51a9-4bf3-8516-341b53dfa255/lib/python3.9/site-packages/alibi/explainers/cfrl_tabular.py in __call__(self, X_cf, X, C)
     70         Conditional sampled counterfactual instance.
     71         """
---> 72         return sample(X_hat_split=X_cf,
     73                       X_ohe=X,
     74                       C=C,

/local_disk0/.ephemeral_nfs/envs/pythonEnv-55e14a0e-51a9-4bf3-8516-341b53dfa255/lib/python3.9/site-packages/alibi/explainers/backends/cfrl_tabular.py in sample(X_hat_split, X_ohe, C, category_map, stats)
    417     if cat_feat > 0:
    418         # Sample categorical columns
--> 419         X_ohe_hat_split += sample_categorical(X_hat_cat_split=X_hat_split[-cat_feat:],
    420                                               C_cat_split=C_cat_split)
    421 

/local_disk0/.ephemeral_nfs/envs/pythonEnv-55e14a0e-51a9-4bf3-8516-341b53dfa255/lib/python3.9/site-packages/alibi/explainers/backends/cfrl_tabular.py in sample_categorical(X_hat_cat_split, C_cat_split)
    359         # compute probability distribution
    360         proba = softmax(X_hat_cat_split[i], axis=1)
--> 361         proba = proba * C_cat_split[i] if (C_cat_split is not None) else proba
    362 
    363         # sample the most probable outcome conditioned on the conditional vector

ValueError: operands could not be broadcast together with shapes (1,2) (1,14)

Code Snippet is as below:

#### Select some positive examples.

  X_positive = group2[np.argmax(predictor(group2.values), axis=1) == 1].reset_index(drop=True)
  group_arc_info = group_arc_info[np.argmax(predictor(group2.values), axis=1) == 1].reset_index(drop=True)
  print("X_positive shape:",X_positive.shape)

  Y_t = np.array([0])
  if (X_positive.shape[0] < 10 ):
      print("Skipped record is : {}".format(name))
      continue
  
  print("*******************fit explainer********************************") 
  explainer1 = explainer.fit(X=group2.values)    ###--------------------- error at this command line---------------------------
  print('explainer1 complete') 

[RobertSamoilescu]

Hi @shrija2901 ,

If you try to fit multiple datasets, even tough they have the same number of features, I recommend you to define and fit a new explainer.

Although I am not entirely sure that I understood exactly your workflow, I suspect that the error comes from the difference in the number of categories within the columns. To give you an example, suppose that we have an initial dataset in which the first categorical column has only 2 categories (e.g., gender taking the values male or female), and another dataset in which the first categorical column has 4 categories (e.g., color taking values red, yellow, blue, green). As you can observe, the one-hot-encoding representation differ for those two. Even more, although you can have the same number of features in the raw representations of the datasets (e.g., label encoding), the number of columns can differ in the one hot encoding because of the difference in the number of categories within each feature.

This error can occur because the metadata of the dataset is provided at initialization and not at the fit step (e.g., feature_names, category_map are provided to __init__ method). That metadata is used to construct the conditional vector.

Again, I am not sure if this is the case for you since I see some grouping based on the prediction of the model. It would be great if you can provide some runnable code so I can reproduce the error locally and understand better your use case. You can replace the data with some dummy data (randomly generated).

Thanks!