Filter of grouped dataframe raises ValueError

[mattemathias]

see also #4527

When trying to filter a grouped dataframe with more than 2 columns raises an ValueError:

In [90]: dff = pd.DataFrame({'A': np.arange(8), 'B': list('aabbbbcc'), 'C': np.arange(8)})

In [91]: dff.groupby('B').filter(lambda x: len(x) > 2)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-91-89d79df28299> in <module>()
----> 1 dff.groupby('B').filter(lambda x: len(x) > 2)

C:\Anaconda\lib\site-packages\pandas\core\groupby.pyc in filter(self, func, dropna, *args, **kwargs)
   2092                 res = path(group)
   2093
-> 2094             if res:
   2095                 indexers.append(self.obj.index.get_indexer(group.index))
   2096

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()


In [93]: pd.__version__
Out[93]: '0.12.0'

[jtratner]

good catch. Looking into this.

[jtratner]

So far, I've found that the proximate problem is with the slow_path here, but fixing it for this case causes issues for other tests. Working on it...

[danielballan]

The problem arises with len specifically. For example, count is fine.

In [1]: dff = pd.DataFrame({'A': np.arange(8), 'B': list('aabbbbcc'), 'C': np.arange(8)})

In [2]: dff.groupby('B').filter(lambda x: x.count() > 2)
Out[2]: 
   A  B  C
0  0  a  0
1  1  a  1
2  2  b  2
3  3  b  3
4  4  b  4
5  5  b  5
6  6  c  6
7  7  c  7

[jreback]

@danielballan you prob just need to deal with different return types/shapes from the filtered function;

this is BTD (bug test driven!)

[jreback]

@danielballan fyi... # 4657 will partially fix this (sort of) as it looks at the return value from the filter in a better way

[jreback]

@danielballan would be nice to fix for 0.13

[danielballan]

I cannot reproduce the error. I think #4657 changed the path taken, so now the shape of the len result is interpreted correctly. On the current master:

In [19]: pd.DataFrame({'A': np.arange(8), 'B': list('aabbbbcc'), 'C': np.arange(8)}).groupby('B').filter(lambda x: len(x) > 2)
Out[19]: 
   A  B  C
2  2  b  2
3  3  b  3
4  4  b  4
5  5  b  5

If you can point me toward an extant problem, I will follow up.

If not, close?

[jreback]

hmm....maybe add some test cases that return different things in the function to try to break it:

e.g. scalar, series, frame, raise various exceptions, etc

[jreback]

and then can close this issue

[danielballan]

OK, sounds good.

[jreback]

@danielballan tests for this?

[danielballan]

Within a week should be no problem. (Not exactly related, but is there a target for 0.13?)

[jreback]

gr8!

not exactly sure of target date...still lots of issues to get thru...few weeks maybe

[jreback]

@danielballan ?

[danielballan]

Added four tests in PR #5096 using len on a DataFrame and a Series. Will continue to explore various shapes. I have not discovered any other filter bugs so far. I will switch focus to #4621....