Groupby "negative dimensions are not allowed" error and bad key behaviour when there are NaNs values.

[jordeu]

On a groupby with a composed key if the product of all possible values is bigger than 2^63 we get a ValueError "negative dimensions are not allowed" when we call len(grouped_data).

A simple version to reproduce it:

values = range(55109)
data = pd.DataFrame.from_dict({'a': values, 'b': values, 'c': values, 'd': values})
grouped = data.groupby(['a', 'b', 'c', 'd'])
len(grouped)

A side effect of this error is that if there are NaN values as possible keys it won't ignore them, it will replace the NaN values with some other values present in the index.

Here there is a complete IPython notebook example to reproduce it:
http://nbviewer.ipython.org/gist/jordeu/cd86fc99f5f89451cf93

[jreback]

hmm, this overflow space is explicity handled (though maybe is hiding an error some). If you'd like to take a look would appreciate (this is a deep part of groupby).

[jreback]

cc @behzadnouri
were just talking about this in #9077 (slightly different issue / manifestation)

[mpschr]

any progress on this issue? It's creating lots of false data

[jreback]

well, pull-requests are welcome.

This of course should be fixed, but you can avoid by not grouping by all available columns (which causes the error).

[jreback]

give a try on master, just merged in #9380 which fixes this.

[jordeu]

Perfect, I've tested on master and now it's working. Thanks!

[rth]

Just got this issue when doing a groupby on a large number of columns with high cardinality with pandas 1.0.3. I'm not sure it was fully fixed. Also there is #31355.

[wangxiaoying]

Just got this issue when doing a groupby on a large number of columns with high cardinality with pandas 1.0.3. I'm not sure it was fully fixed. Also there is #31355.

got the same problem in pandas 1.0.3

[lisovskey]

got this error too with groupby (pandas 1.0.5)

[jreback]

you will have to open. new issue with a reproducible example in the latest released version and even better showing something exists in master