Dataframe.eval(): Negative number in list passed to 'in'-expression causes crash on python 3.4.0

[tobgu]

The following crashes on python 3.4.0. It works fine on Python 2.7.5.

>>> import pandas
>>> from io import StringIO
>>> data = "foo,bar\n11,12"
>>> df = pandas.read_csv(StringIO(data))
>>> df.eval('foo in [11, 32]')
0    True
dtype: bool
>>> df.eval('foo in [11, -32]')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/core/frame.py", line 1987, in eval
    return _eval(expr, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/eval.py", line 230, in eval
    truediv=truediv)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 635, in __init__
    self.terms = self.parse()
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 652, in parse
    return self._visitor.visit(self.expr)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 314, in visit
    return visitor(node, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 320, in visit_Module
    return self.visit(expr, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 314, in visit
    return visitor(node, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 323, in visit_Expr
    return self.visit(node.value, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 314, in visit
    return visitor(node, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 560, in visit_Compare
    return self.visit(binop)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 314, in visit
    return visitor(node, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 404, in visit_BinOp
    op, op_class, left, right = self._possibly_transform_eq_ne(node)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 357, in _possibly_transform_eq_ne
    right = self.visit(node.right, side='right')
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 314, in visit
    return visitor(node, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 430, in visit_List
    name = self.env.add_tmp([self.visit(e).value for e in node.elts])
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 430, in <listcomp>
    name = self.env.add_tmp([self.visit(e).value for e in node.elts])
AttributeError: 'UnaryOp' object has no attribute 'value'
>>> 

[jreback]

pls show a reproducible example, e.g. show how the actual frame is created so it can be copy-pasted

pls show pd.show_versions()

[tobgu]

Sorry, my bad. The lines actually defining the data fell away in during copy-paste. I've updated the example above.

Here's show_versions()

>>> import pandas
>>> pandas.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.0.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-48-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.16.2
nose: None
Cython: None
numpy: 1.9.3
scipy: None
statsmodels: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2015.6
bottleneck: None
tables: None
numexpr: 2.4.4
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None

[jreback]

yeh I guess this is parsed differently in 3.4 that in 2.7.

pull-requests to fix are welcome

[brentp]

Hi all, this appears to still be broken for some cases, e.g. here is a test-case for python3.6, pandas 0.20.1:

def test_unary():
    df = pd.DataFrame({'x': np.array([0.11, 0], dtype=np.float32)})
    res = df.eval('(x > 0.1) | (x < -0.1)')
    assert np.array_equal(res, np.array([True, False])), res

the tail of the output is:

  File ".../envs/py3/lib/python3.6/site-packages/pandas/core/computation/expr.py", line 370, in _maybe_downcast_constants
    name = self.env.add_tmp(np.float32(right.value))
AttributeError: 'UnaryOp' object has no attribute 'value'

In that case the right is -(0.1)

please let me know if you prefer I open a new issue.

[jreback]

best 2 open a new issue