Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Overflow in arrow comparison operator hides actual error behind NotImplementedError #54944

Closed
2 of 3 tasks
rohanjain101 opened this issue Sep 1, 2023 · 0 comments · Fixed by #54952
Closed
2 of 3 tasks
Labels
Arrow pyarrow functionality Bug Error Reporting Incorrect or improved errors from pandas
Milestone

Comments

@rohanjain101
Copy link
Contributor

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> a = pd.Series([1 << 63], dtype="uint64[pyarrow]")
>>> b = pd.Series([None], dtype="int64[pyarrow]")
>>> a < b
Traceback (most recent call last):
  File "C:\ps_311_arrow_13_env\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 630, in _cmp_method
    result = pc_func(self._pa_array, self._box_pa(other))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ps_311_arrow_13_env\Lib\site-packages\pyarrow\compute.py", line 245, in wrapper
    return func.call(args, None, memory_pool)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow\_compute.pyx", line 367, in pyarrow._compute.Function.call
  File "pyarrow\error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow\error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 9223372036854775808 not in range: 0 to 9223372036854775807

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ps_311_arrow_13_env\Lib\site-packages\pandas\core\ops\common.py", line 76, in new_method
    return method(self, other)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\ps_311_arrow_13_env\Lib\site-packages\pandas\core\arraylike.py", line 48, in __lt__
    return self._cmp_method(other, operator.lt)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ps_311_arrow_13_env\Lib\site-packages\pandas\core\series.py", line 5804, in _cmp_method
    res_values = ops.comparison_op(lvalues, rvalues, op)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ps_311_arrow_13_env\Lib\site-packages\pandas\core\ops\array_ops.py", line 332, in comparison_op
    res_values = op(lvalues, rvalues)
                 ^^^^^^^^^^^^^^^^^^^^
  File "C:\ps_311_arrow_13_env\Lib\site-packages\pandas\core\ops\common.py", line 76, in new_method
    return method(self, other)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\ps_311_arrow_13_env\Lib\site-packages\pandas\core\arraylike.py", line 48, in __lt__
    return self._cmp_method(other, operator.lt)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ps_311_arrow_13_env\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 640, in _cmp_method
    raise NotImplementedError(
NotImplementedError: lt not implemented for <class 'pandas.core.arrays.arrow.array.ArrowExtensionArray'>
>>>

Issue Description

The operation is implemented, the actual error should be an overflow from error about 9223372036854775808 not being in range of an int64, this is the error produced by pandas 2.0.3.

Expected Behavior

I expect the error pandas 2.0.3 produced:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\pd_210\Lib\site-packages\pandas\core\ops\common.py", line 81, in new_method
    return method(self, other)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\pd_210\Lib\site-packages\pandas\core\arraylike.py", line 48, in __lt__
    return self._cmp_method(other, operator.lt)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pd_210\Lib\site-packages\pandas\core\series.py", line 6096, in _cmp_method
    res_values = ops.comparison_op(lvalues, rvalues, op)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pd_210\Lib\site-packages\pandas\core\ops\array_ops.py", line 279, in comparison_op
    res_values = op(lvalues, rvalues)
                 ^^^^^^^^^^^^^^^^^^^^
  File "C:\pd_210\Lib\site-packages\pandas\core\ops\common.py", line 81, in new_method
    return method(self, other)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\pd_210\Lib\site-packages\pandas\core\arraylike.py", line 48, in __lt__
    return self._cmp_method(other, operator.lt)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pd_210\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 467, in _cmp_method
    result = pc_func(self._data, other._data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pd_210\Lib\site-packages\pyarrow\compute.py", line 245, in wrapper
    return func.call(args, None, memory_pool)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow\_compute.pyx", line 367, in pyarrow._compute.Function.call
  File "pyarrow\error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow\error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 9223372036854775808 not in range: 0 to 9223372036854775807
>>>

Installed Versions

INSTALLED VERSIONS

commit : ba1cccd
python : 3.11.2.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22621
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 7, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.1.0
numpy : 1.25.2
pytz : 2023.3
dateutil : 2.8.2
setuptools : 68.0.0
pip : 23.2.1
Cython : None
pytest : 7.0.1
hypothesis : None
sphinx : 4.2.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.14.0
pandas_datareader : None
bs4 : 4.12.2
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

@rohanjain101 rohanjain101 added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 1, 2023
@phofl phofl added Error Reporting Incorrect or improved errors from pandas Arrow pyarrow functionality and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 2, 2023
@phofl phofl added this to the 2.1.1 milestone Sep 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Arrow pyarrow functionality Bug Error Reporting Incorrect or improved errors from pandas
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants