agg() swallows exceptions.

[blais]

agg() appears to swallow exceptions.
This was the source of a hard-to-find bug today.
Here's a program that replicates the problem.

#!/usr/bin/env python                                                                                                                                                                                                                         
from pandas import *
from numpy.random import randint

df = DataFrame(randint(10, size=(20, 10)))

def raiseException(df):
  print '----------------------------------------'
  print df.to_string()
  raise TypeError

df.groupby(0).agg(raiseException)

[jreback]

Try raising Exception (instead of TypeError), and your example will work. ``TypeErroris caught internally in order to deal with issues like trying to perform asum` on non-numeric data. Prob ought to be a doc mention on this

[blais]

I wasn't raising a TypeError myself; calling a function with an incorrect number of parameters raises a TypeError. Your solution is not acceptable.

[jreback]

what exactly are you trying to accomplish? maybe put up a sample frame, function that is closer to what you are doing

[blais]

I have a custom aggregator function. This function calls another function; the parameter list was wrong. The error was ignored silently! It should not.

[jreback]

@wesm ?

[ghost]

agg tries to be flexible by applying the aggregator to the entire dataset
at once, and then fallback to doing an item-by-item apply.
This can yield a big performance win if vectorization can be employed, especially in
groupby operations which tend to be large.

Unfortunately, it's hard to predict what kind of Error will be raised by the user-provided
aggregator if the first test fails, hence the broad exception clause.

Although I'm not very fond of this sort of magical behaviour for exactly the reason you
exemplified, I don't see a way to fix that without potentially breaking a lot of code, and
TypeError is specifically something you would expect to see in this situation.

If you have a solution that does not break existing code, let's discuss it.

[hayd]

I like the current magic, but maybe there could be some kind of raise_exceptions flag (which defaults to False)...

[ghost]

This is a debugging issue, it doesn't make sense to crudify signatures with "dev-time only"
args, and that would open the flood gates to similiar signature noise across the lib.

IMO.

[changhiskhan]

what about raising an exception if none of the items can be agg'ed while agg'ing item by item? It shouldn't break existing code (much) and it should solve the above case.
i know it's just adding on more magic but i can't really think of a general/clean solution off the top of my head.

[wesm]

Let me have a look at this soon and let you know what I think.

[wesm]

There's one unit test that breaks by no longer suppressing the TypeError but on inspection I'd rather not support the behavior-- i.e. to be able to pass an aggregation function that ONLY works on the string columns (but fails on the numeric columns). The auto-exclusion of non-numeric data when doing, say, df.groupby(keys).agg(np.sum) was only conceived as a convenience to begin with.

Any objections?

[ghost]

I think that sounds right, grouper functions should be responsible for handling the data,
it's not pandas' role to guess how exceptions should be handled.