New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SystemError on invalid ndarray / Path operation #7360

Open
anntzer opened this Issue Feb 28, 2016 · 5 comments

Comments

Projects
None yet
5 participants
@anntzer
Contributor

anntzer commented Feb 28, 2016

Running

from pathlib import Path
import numpy as np
np.arange(300000) / Path("foo")

raises

TypeError: argument should be a path or str object, not <class 'int'>

During handling of the above exception, another exception occurred:

SystemError: <built-in method __new__ of type object at 0x7fb4df227fc0> returned a result with an error set

<elided: many repeats of the same>

During handling of the above exception, another exception occurred:

SystemError: <built-in method __new__ of type object at 0x7fb4df227fc0> returned a result with an error set

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/foo.py", line 3, in <module>
    np.arange(300000) / Path("foo")
  File "/usr/lib/python3.5/pathlib.py", line 879, in __rtruediv__
    return self._from_parts([key] + self._parts)
  File "/usr/lib/python3.5/pathlib.py", line 637, in _from_parts
    self = object.__new__(cls)
SystemError: <built-in method __new__ of type object at 0x7fb4df227fc0> returned a result with an error set

Note that this does NOT appear for small arrays; I haven't determined the threshold.
Crossposted as http://bugs.python.org/issue26453.
Python 3.5.1, numpy 1.10.4.

@rgommers

This comment has been minimized.

Show comment
Hide comment
@rgommers

rgommers Feb 28, 2016

Member

Looks like a pathlib issue, although the many repeating errors in a loop are odd. For small arrays it's more sensible:

>>> foo = Path('foo')
>>> x = np.arange(3)
>>> x / foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rgommers/Code/pythons/lib/python3.5/pathlib.py", line 879, in __rtruediv__
    return self._from_parts([key] + self._parts)
  File "/home/rgommers/Code/pythons/lib/python3.5/pathlib.py", line 638, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/home/rgommers/Code/pythons/lib/python3.5/pathlib.py", line 630, in _parse_args
    % type(a))
TypeError: argument should be a path or str object, not <class 'int'>

Compared with what happens when dividing with a string:

>>> x / 'foo'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: ufunc 'true_divide' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

So Path implements __rtruediv__ which seems to take precedence over any ndarray methods here. And then inside __rtruediv__ something goes wrong when the left-hand side of the division is a large array.

Python bug, hence close this issue?

Member

rgommers commented Feb 28, 2016

Looks like a pathlib issue, although the many repeating errors in a loop are odd. For small arrays it's more sensible:

>>> foo = Path('foo')
>>> x = np.arange(3)
>>> x / foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rgommers/Code/pythons/lib/python3.5/pathlib.py", line 879, in __rtruediv__
    return self._from_parts([key] + self._parts)
  File "/home/rgommers/Code/pythons/lib/python3.5/pathlib.py", line 638, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/home/rgommers/Code/pythons/lib/python3.5/pathlib.py", line 630, in _parse_args
    % type(a))
TypeError: argument should be a path or str object, not <class 'int'>

Compared with what happens when dividing with a string:

>>> x / 'foo'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: ufunc 'true_divide' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

So Path implements __rtruediv__ which seems to take precedence over any ndarray methods here. And then inside __rtruediv__ something goes wrong when the left-hand side of the division is a large array.

Python bug, hence close this issue?

@serhiy-storchaka

This comment has been minimized.

Show comment
Hide comment
@serhiy-storchaka

serhiy-storchaka Feb 28, 2016

Contributor

TypeError is not a bug. SystemError is a bug. And this looks as NumPy bug.

I can't reproduce the bug on 32-bit Python 3.4.3 with NumPy 1.8.2.

Contributor

serhiy-storchaka commented Feb 28, 2016

TypeError is not a bug. SystemError is a bug. And this looks as NumPy bug.

I can't reproduce the bug on 32-bit Python 3.4.3 with NumPy 1.8.2.

@vstinner

This comment has been minimized.

Show comment
Hide comment
@vstinner

vstinner Feb 28, 2016

Contributor

Traceback:

python
>>> from pathlib import Path; import numpy as np; np.arange(300000) / Path("foo")
python: Objects/abstract.c:2158: PyObject_Call: Assertion `!PyErr_Occurred()' failed.
(...)
(gdb) where
...
#3  0x00007ffff71222d2 in __assert_fail () from /lib64/libc.so.6
#4  0x000000000046ad09 in PyObject_Call (func=<method at remote 0x7ffff7f2fde8>, arg=(8192,), kw=0x0) at Objects/abstract.c:2158

#5  0x00000000004f2ed0 in call_maybe (o=<PosixPath at remote 0x7fffea9814b8>, nameid=0x961820 <rop_id>, format=0x6b010d "(O)") at Objects/typeobject.c:1483
#6  0x00000000005042c3 in slot_nb_true_divide (self=8192, other=<PosixPath at remote 0x7fffea9814b8>) at Objects/typeobject.c:5973
#7  0x0000000000466d09 in binary_op1 (v=8192, w=<PosixPath at remote 0x7fffea9814b8>, op_slot=240) at Objects/abstract.c:765
#8  0x0000000000466e37 in binary_op (v=8192, w=<PosixPath at remote 0x7fffea9814b8>, op_slot=240, op_name=0x698c1b "/") at Objects/abstract.c:788
#9  0x00000000004677b7 in PyNumber_TrueDivide (v=8192, w=<PosixPath at remote 0x7fffea9814b8>) at Objects/abstract.c:956
#10 0x00007fffef053ed1 in PyUFunc_OO_O (args=0xb41658, dimensions=0xb41618, steps=0xb41640, func=0x46778d <PyNumber_TrueDivide>) at numpy/core/src/umath/loops.c.src:536

#11 0x00007fffef0c324f in iterator_loop (ufunc=0xbcb0a0, op=0x7fffffffbb30, dtype=0x7fffffffb700, order=NPY_KEEPORDER, buffersize=8192, arr_prep=0x7fffffffb5f0, arr_prep_args=0x0, 
    innerloop=0x7fffef053dd7 <PyUFunc_OO_O>, innerloopdata=0x46778d <PyNumber_TrueDivide>) at numpy/core/src/umath/ufunc_object.c:1505
#12 0x00007fffef0c3f7e in execute_legacy_ufunc_loop (ufunc=0xbcb0a0, trivial_loop_ok=0, op=0x7fffffffbb30, dtypes=0x7fffffffb700, order=NPY_KEEPORDER, buffersize=8192, 
    arr_prep=0x7fffffffb5f0, arr_prep_args=0x0) at numpy/core/src/umath/ufunc_object.c:1660
#13 0x00007fffef0c6887 in PyUFunc_GenericFunction (ufunc=0xbcb0a0, args=(<numpy.ndarray at remote 0x7ffff06d3940>, <PosixPath at remote 0x7fffea9814b8>), kwds=0x0, op=0x7fffffffbb30)
    at numpy/core/src/umath/ufunc_object.c:2627
#14 0x00007fffef0ccbd2 in ufunc_generic_call (ufunc=0xbcb0a0, args=(<numpy.ndarray at remote 0x7ffff06d3940>, <PosixPath at remote 0x7fffea9814b8>), kwds=0x0)
    at numpy/core/src/umath/ufunc_object.c:4253

#15 0x000000000046adac in PyObject_Call (func=<numpy.ufunc at remote 0xbcb0a0>, arg=(<numpy.ndarray at remote 0x7ffff06d3940>, <PosixPath at remote 0x7fffea9814b8>), kw=0x0)
    at Objects/abstract.c:2170
#16 0x000000000046bed6 in PyObject_CallFunctionObjArgs (callable=<numpy.ufunc at remote 0xbcb0a0>) at Objects/abstract.c:2450
#17 0x00007fffef79f32f in PyArray_GenericBinaryFunction (m1=0x7ffff06d3940, m2=<PosixPath at remote 0x7fffea9814b8>, op=<numpy.ufunc at remote 0xbcb0a0>)
    at numpy/core/src/multiarray/number.c:321
#18 0x00007fffef7a0de0 in array_true_divide (m1=0x7ffff06d3940, m2=<PosixPath at remote 0x7fffea9814b8>) at numpy/core/src/multiarray/number.c:737

#19 0x0000000000466c67 in binary_op1 (v=<numpy.ndarray at remote 0x7ffff06d3940>, w=<PosixPath at remote 0x7fffea9814b8>, op_slot=240) at Objects/abstract.c:759
#20 0x0000000000466e37 in binary_op (v=<numpy.ndarray at remote 0x7ffff06d3940>, w=<PosixPath at remote 0x7fffea9814b8>, op_slot=240, op_name=0x698c1b "/") at Objects/abstract.c:788
#21 0x00000000004677b7 in PyNumber_TrueDivide (v=<numpy.ndarray at remote 0x7ffff06d3940>, w=<PosixPath at remote 0x7fffea9814b8>) at Objects/abstract.c:956
#22 0x00000000005b9a68 in PyEval_EvalFrameEx (f=Frame 0x7ffff06d6238, for file <stdin>, line 1, in <module> (), throwflag=0) at Python/ceval.c:1524
Contributor

vstinner commented Feb 28, 2016

Traceback:

python
>>> from pathlib import Path; import numpy as np; np.arange(300000) / Path("foo")
python: Objects/abstract.c:2158: PyObject_Call: Assertion `!PyErr_Occurred()' failed.
(...)
(gdb) where
...
#3  0x00007ffff71222d2 in __assert_fail () from /lib64/libc.so.6
#4  0x000000000046ad09 in PyObject_Call (func=<method at remote 0x7ffff7f2fde8>, arg=(8192,), kw=0x0) at Objects/abstract.c:2158

#5  0x00000000004f2ed0 in call_maybe (o=<PosixPath at remote 0x7fffea9814b8>, nameid=0x961820 <rop_id>, format=0x6b010d "(O)") at Objects/typeobject.c:1483
#6  0x00000000005042c3 in slot_nb_true_divide (self=8192, other=<PosixPath at remote 0x7fffea9814b8>) at Objects/typeobject.c:5973
#7  0x0000000000466d09 in binary_op1 (v=8192, w=<PosixPath at remote 0x7fffea9814b8>, op_slot=240) at Objects/abstract.c:765
#8  0x0000000000466e37 in binary_op (v=8192, w=<PosixPath at remote 0x7fffea9814b8>, op_slot=240, op_name=0x698c1b "/") at Objects/abstract.c:788
#9  0x00000000004677b7 in PyNumber_TrueDivide (v=8192, w=<PosixPath at remote 0x7fffea9814b8>) at Objects/abstract.c:956
#10 0x00007fffef053ed1 in PyUFunc_OO_O (args=0xb41658, dimensions=0xb41618, steps=0xb41640, func=0x46778d <PyNumber_TrueDivide>) at numpy/core/src/umath/loops.c.src:536

#11 0x00007fffef0c324f in iterator_loop (ufunc=0xbcb0a0, op=0x7fffffffbb30, dtype=0x7fffffffb700, order=NPY_KEEPORDER, buffersize=8192, arr_prep=0x7fffffffb5f0, arr_prep_args=0x0, 
    innerloop=0x7fffef053dd7 <PyUFunc_OO_O>, innerloopdata=0x46778d <PyNumber_TrueDivide>) at numpy/core/src/umath/ufunc_object.c:1505
#12 0x00007fffef0c3f7e in execute_legacy_ufunc_loop (ufunc=0xbcb0a0, trivial_loop_ok=0, op=0x7fffffffbb30, dtypes=0x7fffffffb700, order=NPY_KEEPORDER, buffersize=8192, 
    arr_prep=0x7fffffffb5f0, arr_prep_args=0x0) at numpy/core/src/umath/ufunc_object.c:1660
#13 0x00007fffef0c6887 in PyUFunc_GenericFunction (ufunc=0xbcb0a0, args=(<numpy.ndarray at remote 0x7ffff06d3940>, <PosixPath at remote 0x7fffea9814b8>), kwds=0x0, op=0x7fffffffbb30)
    at numpy/core/src/umath/ufunc_object.c:2627
#14 0x00007fffef0ccbd2 in ufunc_generic_call (ufunc=0xbcb0a0, args=(<numpy.ndarray at remote 0x7ffff06d3940>, <PosixPath at remote 0x7fffea9814b8>), kwds=0x0)
    at numpy/core/src/umath/ufunc_object.c:4253

#15 0x000000000046adac in PyObject_Call (func=<numpy.ufunc at remote 0xbcb0a0>, arg=(<numpy.ndarray at remote 0x7ffff06d3940>, <PosixPath at remote 0x7fffea9814b8>), kw=0x0)
    at Objects/abstract.c:2170
#16 0x000000000046bed6 in PyObject_CallFunctionObjArgs (callable=<numpy.ufunc at remote 0xbcb0a0>) at Objects/abstract.c:2450
#17 0x00007fffef79f32f in PyArray_GenericBinaryFunction (m1=0x7ffff06d3940, m2=<PosixPath at remote 0x7fffea9814b8>, op=<numpy.ufunc at remote 0xbcb0a0>)
    at numpy/core/src/multiarray/number.c:321
#18 0x00007fffef7a0de0 in array_true_divide (m1=0x7ffff06d3940, m2=<PosixPath at remote 0x7fffea9814b8>) at numpy/core/src/multiarray/number.c:737

#19 0x0000000000466c67 in binary_op1 (v=<numpy.ndarray at remote 0x7ffff06d3940>, w=<PosixPath at remote 0x7fffea9814b8>, op_slot=240) at Objects/abstract.c:759
#20 0x0000000000466e37 in binary_op (v=<numpy.ndarray at remote 0x7ffff06d3940>, w=<PosixPath at remote 0x7fffea9814b8>, op_slot=240, op_name=0x698c1b "/") at Objects/abstract.c:788
#21 0x00000000004677b7 in PyNumber_TrueDivide (v=<numpy.ndarray at remote 0x7ffff06d3940>, w=<PosixPath at remote 0x7fffea9814b8>) at Objects/abstract.c:956
#22 0x00000000005b9a68 in PyEval_EvalFrameEx (f=Frame 0x7ffff06d6238, for file <stdin>, line 1, in <module> (), throwflag=0) at Python/ceval.c:1524
@vstinner

This comment has been minimized.

Show comment
Hide comment
@vstinner

vstinner Feb 28, 2016

Contributor

iterator_loop() calls PyUFunc_OO_O() which calls Path.rtruediv() which raises an error, but it doesn't handle immediatly the error: it continue to call PyUFunc_OO_O() with an exception set.

Python 3.5 helps to detect bugs in C code, it is now more strict. It's not more possible to call arbitrary Python function with an exception set. You have two main choices:

  • completly ignore the exception (PyErr_Clear()) -- usually a bad choice
  • detect that PyUFunc_OO_O() failed and stop the loop -- probably the better choice

It's up to you to decide how to handle this case.

I modified Python 3.5 to make it more strict to avoid loosing exceptions by mistake. For example, calling hasattr() can clear the current exception. So if you call hasattr() while you handle an exception, you may simply loose the current exception. It may work or not. Python 3.5 now requires to be more careful.

Contributor

vstinner commented Feb 28, 2016

iterator_loop() calls PyUFunc_OO_O() which calls Path.rtruediv() which raises an error, but it doesn't handle immediatly the error: it continue to call PyUFunc_OO_O() with an exception set.

Python 3.5 helps to detect bugs in C code, it is now more strict. It's not more possible to call arbitrary Python function with an exception set. You have two main choices:

  • completly ignore the exception (PyErr_Clear()) -- usually a bad choice
  • detect that PyUFunc_OO_O() failed and stop the loop -- probably the better choice

It's up to you to decide how to handle this case.

I modified Python 3.5 to make it more strict to avoid loosing exceptions by mistake. For example, calling hasattr() can clear the current exception. So if you call hasattr() while you handle an exception, you may simply loose the current exception. It may work or not. Python 3.5 now requires to be more careful.

@seberg

This comment has been minimized.

Show comment
Hide comment
@seberg

seberg Feb 28, 2016

Member

The threshold is going to be the iterator buffer size (8196 elements I think). Though it surprises me that it should be different if one loop or more loops are called. Maybe we don't check the error state after every loop but only at the very end. We will have to check.

Member

seberg commented Feb 28, 2016

The threshold is going to be the iterator buffer size (8196 elements I think). Though it surprises me that it should be different if one loop or more loops are called. Maybe we don't check the error state after every loop but only at the very end. We will have to check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment