pd.eval() discards imaginary part in division "/"

[fillipe-gsm]

Code Sample

data = {"a": [1 + 2j], "b": [1 + 1j]}
df = pd.DataFrame(data = data)
df.eval("a/b")

/usr/local/lib64/python3.6/site-packages/pandas/core/dtypes/cast.py:730: ComplexWarning: Casting complex values to real discards the imaginary part
  return arr.astype(dtype, copy=True)

0    1.0
dtype: float64

Problem description

The output type was coerced into a float. This also happens by assigning the result to another existing column:

data = {"a": [1 + 2j], "b": [1 + 1j], "c": [1j]}
df = pd.DataFrame(data = data)
df.eval("c = a/b")

/usr/local/lib64/python3.6/site-packages/pandas/core/dtypes/cast.py:730: ComplexWarning: Casting complex values to real discards the imaginary part
  return arr.astype(dtype, copy=True)
Out[82]: 
        a       b    c
0  (1+2j)  (1+1j)  1.0

And even if the operation is in place:

data = {"a": [1 + 2j], "b": [1 + 1j], "c": [1j]}
df = pd.DataFrame(data = data)
df.eval("c = a/b", inplace = True)
/usr/local/lib64/python3.6/site-packages/pandas/core/dtypes/cast.py:730: ComplexWarning: Casting complex values to real discards the imaginary part
  return arr.astype(dtype, copy=True)

df
        a       b    c
0  (1+2j)  (1+1j)  1.0

Expected Output

The expected output is

df["a"]/df["b"]

0    (1.5+0.5j)
dtype: complex128

The problem seems to happen only with the "/" operator. In fact, the correct result can be obtained by replacing the division with a multiplication and a negative exponent:

df.eval("a*b**(-1)")

0    (1.5+0.5j)
dtype: complex128

Output of pd.show_versions()

Details

INSTALLED VERSIONS

commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Linux
OS-release: 4.16.12-300.fc28.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.utf8
LOCALE: en_US.UTF-8

pandas: 0.23.0
pytest: None
pip: 9.0.3
setuptools: 39.2.0
Cython: None
numpy: 1.14.4
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: None
patsy: 0.4.1
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.1
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: None
lxml: 4.1.1
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[gfyoung]

@fillipe-gsm : How odd! Investigation and patch are welcome!

[uds5501]

@gfyoung @fillipe-gsm maybe the problem lies in the usage of numpy.imag casting (if we are using it) , have a look at it's documentation here : https://docs.scipy.org/doc/numpy/reference/generated/numpy.imag.html

It also mentions that " If val is real, the type of val is used for the output. If val has complex elements, the returned type is float. "
Can it happen that "/" operator triggers the type caster and the "**(-1)" doesn't?

[fillipe-gsm]

@gfyoung @uds5501
I did some throghout investigation line by line starting in the pandas.core.computation.eval module, and I may have found some lead in the Div class inside the pandas.core.computation.ops module, which seems to be created specially to handle divisions. Given the odd behavior happening only with the "/" operator, this makes some sense. Since it is a small code, here it is:

class Div(BinOp):

    """Div operator to special case casting.

    Parameters
    ----------
    lhs, rhs : Term or Op
        The Terms or Ops in the ``/`` expression.
    truediv : bool
        Whether or not to use true division. With Python 3 this happens
        regardless of the value of ``truediv``.
    """

    def __init__(self, lhs, rhs, truediv, *args, **kwargs):
        super(Div, self).__init__('/', lhs, rhs, *args, **kwargs)

        if not isnumeric(lhs.return_type) or not isnumeric(rhs.return_type):
            raise TypeError("unsupported operand type(s) for {0}:"
                            " '{1}' and '{2}'".format(self.op,
                                                      lhs.return_type,
                                                      rhs.return_type))

        if truediv or PY3:
            # do not upcast float32s to float64 un-necessarily
            acceptable_dtypes = [np.float32, np.float_]
            _cast_inplace(com.flatten(self), acceptable_dtypes, np.float_)

It seems that this class is simply not ready to handle complex numbers given the instruction acceptable_dtypes = [np.float32, np.float_]. Thus, I tried to replace this line with acceptable_dtypes = [np.float32, np.float_, np.complex_, np.complex64]. Apparently, this solves the problem:

data = {"a": [1 + 2j], "b": [1 + 1j]}
 df = pd.DataFrame(data = data)
 df.eval("a/b")
  
0    (1.5+0.5j)
dtype: complex128

What do you guys think?

[gfyoung]

@fillipe-gsm : That seems reasonable to me. A PR for this would be great!