ERR: pivot_table when number of levels larger than int32 range #20601
Code Sample, a copy-pastable example if possible
Above code raises the following error:
np.prod(self.full_shape) appears to be returning a negative value because the number of unique index combinations is larger than the largest int32 value.
If line 144 were changed to the following, the issue could be fixed:
1337600 x 3040 dataframe.
It does fix the problem, but I do understand the memory impact it could cause.
If this patch has too much of a negative impact on memory usage, I think throwing an error indicting that np.prod will lead to integer overflow when the pivot table is too large would be enough. That would inform the user that they need to reduce the size of the data to resolve the problem since the ValueError that actually got raised made it hard to track down the problem.