numexpr 2.3.1 error with pandas 0.13.1

[bluefir]

I just installed numexpr 2.3.1 with pandas 0.13.1 and got the following error:

File "C:\Python27\lib\site-packages\pandas\core\ops.py", line 496, in wrapper
arr = na_op(lvalues, rvalues)
File "C:\Python27\lib\site-packages\pandas\core\ops.py", line 443, in na_op
raise_on_error=True, *_eval_kwargs)
File "C:\Python27\lib\site-packages\pandas\computation\expressions.py", line 176, in evaluate
*_eval_kwargs)
File "C:\Python27\lib\site-packages\pandas\computation\expressions.py", line 104, in _evaluate_numexpr
*_eval_kwargs)
File "C:\Python27\lib\site-packages\numexpr\necompiler.py", line 738, in evaluate
NumExpr(ex, signature, *context)
File "C:\Python27\lib\site-packages\numexpr\necompiler.py", line 554, in NumExpr
precompile(ex, signature, context)
File "C:\Python27\lib\site-packages\numexpr\necompiler.py", line 498, in precompile
ast = typeCompileAst(ast)
File "C:\Python27\lib\site-packages\numexpr\necompiler.py", line 163, in typeCompileAst
% (ast.value + '' + retsig+basesig))
NotImplementedError: couldn't find matching opcode for 'mul_bbb'

[jreback]

and what exactly are you doing when it gives that error?

[bluefir]

Good point.

n_nonzero_returns_idc = np.sum(portfolio_data[field_return_daily_idc].notnull() *
                               (portfolio_data[field_return_daily_idc] != 0.0))

[cpcloud]

why not just do & instead?

[jreback]

can't use booleans in numexpr with multiply; need to astype(int) if you reall y want to do that

as @cpcloud says, why are you not using &?

[cpcloud]

still ... should this fall back onto python/bottleneck ops?

[bluefir]

& worked - had to change it everywhere. cpcloud is right: & is better than *. I started using & instead of * when I realized that. May be it's a good reason to correct those things. Still, has numexpr become more hard ass about things like that?

[jreback]

maybe not catching the NotImplementedError?

and not testing with 2.3.1...hmmm will fix that...

[jreback]

I take that back...we ARE testing on window 2.3.1.....guess don't have a test for that particular case.....

@cpcloud when you have a chance?

[cpcloud]

i don't think numexpr has ever supported arithmetic ops on bool dtype arrays. not sure why this is throwing from pandas tho ... @jreback sure

[bluefir]

used to work with numexpr 2.0.1 and pandas 0.12

[jreback]

I agree....should just catch and revert to python

@bluefir what numpy are you using?

[jreback]

looks like 2.1 changed this (to a warning then later deprecated)....see here: https://github.com/pydata/numexpr/blob/master/RELEASE_NOTES.rst

[cpcloud]

oh didn't know they were on gh awesom

[cpcloud]

now we can add opcodes and datetime timedelta support hurrah!!!

[bluefir]

numpy 1.8.1

[bluefir]

Ok, just to intercept here. Are you guys saying there is a faster way of doing what I am trying to do?

[jreback]

looks like you are trying to index something else but keep shape?

df.where(cond1 & cond2)

or just select

df[cond1 & cond2]

to just get the indexed data

[cpcloud]

could also do np.sum(df[the_slice_or_index].fillna(0.0) != 0)

[bluefir]

I just want to calculate the number of values in a column that are not null and not equal to zero.

[jreback]

s[s!=0].count()

[bluefir]

Ha! Indeed. Thanks!

[cpcloud]

@jreback should the arith bool ops return the same thing as combine? combine actually does each operation "manually" then puts the result in an array of the same type as the lhs and rhs ... which makes truediv different for numpy arrays vs. combine ... just not sure what direction the consistency should go np or pandas

[cpcloud]

dividing bools is weird and is an edge case so it's not THAT important i just need to pick one

[jreback]

I think you just raise on all bool dtypes that are not boolean types (e.g. &,|,~|-). no reason for numexpr to support * which is the only meaningful one (+ and /) don't make sense at all and subtraction is just negation