New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataframe.eval(): Negative number in list passed to 'in'-expression causes crash on python 3.4.0 #11235

Closed
tobgu opened this Issue Oct 4, 2015 · 5 comments

Comments

Projects
None yet
3 participants
@tobgu
Contributor

tobgu commented Oct 4, 2015

The following crashes on python 3.4.0. It works fine on Python 2.7.5.

>>> import pandas
>>> from io import StringIO
>>> data = "foo,bar\n11,12"
>>> df = pandas.read_csv(StringIO(data))
>>> df.eval('foo in [11, 32]')
0    True
dtype: bool
>>> df.eval('foo in [11, -32]')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/core/frame.py", line 1987, in eval
    return _eval(expr, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/eval.py", line 230, in eval
    truediv=truediv)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 635, in __init__
    self.terms = self.parse()
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 652, in parse
    return self._visitor.visit(self.expr)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 314, in visit
    return visitor(node, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 320, in visit_Module
    return self.visit(expr, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 314, in visit
    return visitor(node, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 323, in visit_Expr
    return self.visit(node.value, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 314, in visit
    return visitor(node, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 560, in visit_Compare
    return self.visit(binop)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 314, in visit
    return visitor(node, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 404, in visit_BinOp
    op, op_class, left, right = self._possibly_transform_eq_ne(node)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 357, in _possibly_transform_eq_ne
    right = self.visit(node.right, side='right')
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 314, in visit
    return visitor(node, **kwargs)
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 430, in visit_List
    name = self.env.add_tmp([self.visit(e).value for e in node.elts])
  File "/home/tobias/Envs/qcache-py3/lib/python3.4/site-packages/pandas/computation/expr.py", line 430, in <listcomp>
    name = self.env.add_tmp([self.visit(e).value for e in node.elts])
AttributeError: 'UnaryOp' object has no attribute 'value'
>>> 
@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 4, 2015

Contributor

pls show a reproducible example, e.g. show how the actual frame is created so it can be copy-pasted

pls show pd.show_versions()

Contributor

jreback commented Oct 4, 2015

pls show a reproducible example, e.g. show how the actual frame is created so it can be copy-pasted

pls show pd.show_versions()

@tobgu

This comment has been minimized.

Show comment
Hide comment
@tobgu

tobgu Oct 4, 2015

Contributor

Sorry, my bad. The lines actually defining the data fell away in during copy-paste. I've updated the example above.

Here's show_versions()

>>> import pandas
>>> pandas.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.0.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-48-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.16.2
nose: None
Cython: None
numpy: 1.9.3
scipy: None
statsmodels: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2015.6
bottleneck: None
tables: None
numexpr: 2.4.4
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
Contributor

tobgu commented Oct 4, 2015

Sorry, my bad. The lines actually defining the data fell away in during copy-paste. I've updated the example above.

Here's show_versions()

>>> import pandas
>>> pandas.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.0.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-48-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.16.2
nose: None
Cython: None
numpy: 1.9.3
scipy: None
statsmodels: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2015.6
bottleneck: None
tables: None
numexpr: 2.4.4
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 4, 2015

Contributor

yeh I guess this is parsed differently in 3.4 that in 2.7.

pull-requests to fix are welcome

Contributor

jreback commented Oct 4, 2015

yeh I guess this is parsed differently in 3.4 that in 2.7.

pull-requests to fix are welcome

@jreback jreback added this to the Next Major Release milestone Oct 4, 2015

@jreback jreback modified the milestones: 0.17.1, Next Major Release Oct 19, 2015

@brentp

This comment has been minimized.

Show comment
Hide comment
@brentp

brentp May 15, 2017

Hi all, this appears to still be broken for some cases, e.g. here is a test-case for python3.6, pandas 0.20.1:

def test_unary():
    df = pd.DataFrame({'x': np.array([0.11, 0], dtype=np.float32)})
    res = df.eval('(x > 0.1) | (x < -0.1)')
    assert np.array_equal(res, np.array([True, False])), res

the tail of the output is:

  File ".../envs/py3/lib/python3.6/site-packages/pandas/core/computation/expr.py", line 370, in _maybe_downcast_constants
    name = self.env.add_tmp(np.float32(right.value))
AttributeError: 'UnaryOp' object has no attribute 'value'

In that case the right is -(0.1)

please let me know if you prefer I open a new issue.

brentp commented May 15, 2017

Hi all, this appears to still be broken for some cases, e.g. here is a test-case for python3.6, pandas 0.20.1:

def test_unary():
    df = pd.DataFrame({'x': np.array([0.11, 0], dtype=np.float32)})
    res = df.eval('(x > 0.1) | (x < -0.1)')
    assert np.array_equal(res, np.array([True, False])), res

the tail of the output is:

  File ".../envs/py3/lib/python3.6/site-packages/pandas/core/computation/expr.py", line 370, in _maybe_downcast_constants
    name = self.env.add_tmp(np.float32(right.value))
AttributeError: 'UnaryOp' object has no attribute 'value'

In that case the right is -(0.1)

please let me know if you prefer I open a new issue.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback May 15, 2017

Contributor

best 2 open a new issue

Contributor

jreback commented May 15, 2017

best 2 open a new issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment