ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (339,) + inhomogeneous part

[jeremyhermann]

Do you know why I'd get this error when running PowerShap?

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[95], [line 16](vscode-notebook-cell:?execution_count=95&line=16)
      [5](vscode-notebook-cell:?execution_count=95&line=5) # X, y = ...  # your classification dataset
      [6](vscode-notebook-cell:?execution_count=95&line=6) 
      [7](vscode-notebook-cell:?execution_count=95&line=7) # selector = PowerShap(
      [8](vscode-notebook-cell:?execution_count=95&line=8) #     model=LGBMRegressor(n_estimators=1000, verbose=1)
      [9](vscode-notebook-cell:?execution_count=95&line=9) # )
     [11](vscode-notebook-cell:?execution_count=95&line=11) selector = PowerShap(
     [12](vscode-notebook-cell:?execution_count=95&line=12)     model=CatBoostRegressor(n_estimators=250, verbose=0, use_best_model=True),
     [13](vscode-notebook-cell:?execution_count=95&line=13)     power_iterations=2
     [14](vscode-notebook-cell:?execution_count=95&line=14) )
---> [16](vscode-notebook-cell:?execution_count=95&line=16) selector.fit(X, y)  # Fit the PowerShap feature selector

File [~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:392](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:392), in PowerShap.fit(self, X, y, stratify, groups, **kwargs)
    [375](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:375)     self._print(
    [376](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:376)         "Automatic mode enabled: Finding the minimal required powershap",
    [377](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:377)         f"iterations for significance of {self.power_alpha}.",
    [378](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:378)     )
    [380](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:380) shaps_df = self._explainer.explain(
    [381](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:381)     X=X,
    [382](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:382)     y=y,
   (...)
    [389](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:389)     **kwargs,
    [390](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:390) )
--> [392](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:392) processed_shaps_df = powerSHAP_statistical_analysis(
    [393](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:393)     shaps_df, self.power_alpha, self.power_req_iterations, include_all=self.include_all
    [394](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:394) )
    [396](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:396) if self.automatic:
    [398](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:398)     processed_shaps_df = self._automatic_fit(
    [399](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:399)         X=X,
    [400](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:400)         y=y,
   (...)
    [406](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:406)         **kwargs,
    [407](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/powershap.py:407)     )

File [~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:72](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:72), in powerSHAP_statistical_analysis(shaps_df, power_alpha, power_req_iterations, include_all)
     [62](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:62)         effect_size.append(0)
     [63](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:63)         power_list.append(0)
     [65](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:65) processed_shaps_df = pd.DataFrame(
     [66](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:66)     data=np.hstack(
     [67](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:67)         [
     [68](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:68)             np.reshape(shaps_df.mean().values, (-1, 1)),
     [69](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:69)             np.reshape(np.array(p_values), (len(p_values), 1)),
     [70](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:70)             np.reshape(np.array(effect_size), (len(effect_size), 1)),
     [71](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:71)             np.reshape(np.array(power_list), (len(power_list), 1)),
---> [72](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:72)             np.reshape(np.array(required_iterations), (len(required_iterations), 1)),
     [73](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:73)         ]
     [74](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:74)     ),
     [75](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:75)     columns=[
     [76](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:76)         "impact",
     [77](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:77)         "p_value",
     [78](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:78)         "effect_size",
     [79](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:79)         "power_" + str(power_alpha) + "_alpha",
     [80](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:80)         str(power_req_iterations) + "_power_its_req",
     [81](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:81)     ],
     [82](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:82)     index=shaps_df.mean().index,
     [83](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:83) )
     [84](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:84) processed_shaps_df = processed_shaps_df.reindex(
     [85](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:85)     processed_shaps_df.impact.abs().sort_values(ascending=False).index
     [86](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:86) )
     [88](https://vscode-remote+ssh-002dremote-002bdelphina-002djeremy.vscode-resource.vscode-cdn.net/home/ubuntu/delphina-beta/python/notebooks/gold/nyctaxi/~/mambaforge/envs/delphina-beta-env/lib/python3.10/site-packages/powershap/utils.py:88) return processed_shaps_df

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (339,) + inhomogeneous part.

[jvdd]

Hi @jeremyhermann,

Thanks for submitting this issue.
If you can provide a (minimal) reproducible example, I'll gladly further analyze why this error occurs.

• Jeroen

[mrzdev]

Hi,

chiming in, I observed the same issue occur a while ago using PowerShap with Optuna .

I tracked it down to TTestPower().solve_power call in utils.py issuing a warning
site-packages/statsmodels/stats/power.py:525: ConvergenceWarning: Failed to converge on a solution.
and returning a list instead of a scalar here:

powershap/powershap/utils.py

Line 51 in 4a60fbe

TTestPower().solve_power(

In that case, after appending the result to required_iterations list, we end up with inhomogeneous shape and it leads to an error during conversion of required_iterations to numpy array here:

powershap/powershap/utils.py

Line 72 in 4a60fbe

np.reshape(np.array(required_iterations), (len(required_iterations), 1)),

Logging the solve_power returns in each iteration:

[10.]
2.4867261850681004
2.6408546509597652
...

Here's a repro from trial that errored out for me.
The solution is maybe simply unpacking first value invariant if a scalar or a list: np.asarray(solved_power).flatten()[0] before appending.
Let me know what you think.

[jonasvdd]

@jvdd @JarneVerhaeghe - this appears highly similar to the error that I obtained when updating the dependencies.

[hentt30]

I'm having the same error since upgrading to NumPy 1.24.1. Looks like it was caused by a breaking change in NumPy.

https://numpy.org/neps/nep-0034-infer-dtype-is-object.html

[ppawlo97]

I'm still experiencing the same error, would it be possible to adjust the code to make it compatible with later versions of NumPy?

[gifig]

I'm still experiencing the same error, would it be possible to adjust the code to make it compatible with later versions of NumPy?

I managed to fix it by treating the list that contains an array before creating the dataframe.

in line 64 of utils.py:

flat_required_iterations = []

for item in required_iterations:
if isinstance(item, np.ndarray):
flat_required_iterations.extend(item.tolist())
else:
flat_required_iterations.append(item)

Then use the new list in line 72.

Not pretty, but did it for me in a pinch.

[JarneVerhaeghe]

This should be ixed with the new release https://github.com/predict-idlab/powershap/releases/tag/v0.1.0