Typed list fix type inference #4
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit does three things: To begin with, there were actually two severe bugs in `is_prceise`. First, the negation was missing and secondly the trailing comma meant that a tuple was being returned. Thus, a list would have always been deemed to be precise, if there had been any tests for it. Then, add a simple test to check if refinement works within and outside of a jit context. Lastly, After fixing the refinement of the typed-list in a jit context, it turned out that there is an issue with `overload_method`. The issue is that, in case of calling an overloaded method on an imprecise container type, e.g. typed-list, a function with `overload_method` must refine the type and return a tuple of the signature that the implementation should have had and the implementation. While `overload` does have support for receiving such a tuple, `overload_method` does not and instead an error regarding a tuple not being callable is raised. This is fixed now too.
esc
force-pushed
the
typed_list_fix_type_inferene
branch
from
f1bd467
to
1b22212
Compare
esc
added a commit
that referenced
this pull request
May 10, 2024
This fixes a test, where a non-thread safe container is written to
during testing
To reproduce on at least `linux-64` and `osx-arm64` (and probably
others):
```
NUMBA_THREADING_LAYER=workqueue SUBPROC_TEST=1 ./runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting
```
On `linux-64`, this can be debugged with `gdb`:
```
(gdb) bt
closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438
(gdb) f 9
closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438
4438 <string>: No such file or directory.
(gdb) i args
sched = {meminfo = 0x0, parent = 0x0, nitems = 0, itemsize = 0, data = 0x0, shape = {0}, strides = {0}}
closure____locals______listcomp____v15____v2build__list__0 = {meminfo = 0x0, parent = 0x0}
(gdb) f 8
498 mi->data = NRT_Reallocate(mi->data, size);
(gdb) i args
mi = 0x21d1660
size = 2510933856
```
On `osx-arm64` we can use `lldb`:
```
(lldb) run runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting
Process 13575 launched: '/Users/esc/miniconda3-arm64/envs/numba_3.9/bin/python3' (arm64)
Parallel: 0. Serial: 1
python3(13575,0x17025b000) malloc: Non-aligned pointer 0x600000256880 being freed (2)
python3(13575,0x17025b000) malloc: *** set a breakpoint in malloc_error_break to debug
Process 13575 stopped
* thread numba#18, stop reason = signal SIGABRT
frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`:
-> 0x19c35c704 <+8>: b.lo 0x19c35c724 ; <+40>
0x19c35c708 <+12>: pacibsp
0x19c35c70c <+16>: stp x29, x30, [sp, #-0x10]!
0x19c35c710 <+20>: mov x29, sp
Target 0: (python3) stopped.
(lldb) bt
* thread numba#18, stop reason = signal SIGABRT
* frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8
frame #1: 0x000000019c393c28 libsystem_pthread.dylib`pthread_kill + 288
frame #2: 0x000000019c2a1ae8 libsystem_c.dylib`abort + 180
frame #3: 0x000000019c1c2e28 libsystem_malloc.dylib`malloc_vreport + 908
frame #4: 0x000000019c1d95d4 libsystem_malloc.dylib`malloc_zone_error + 104
frame #5: 0x000000019c1ca620 libsystem_malloc.dylib`_szone_free + 628
frame #6: 0x000000019c1b87f4 libsystem_malloc.dylib`nanov2_realloc + 356
frame #7: 0x000000019c1b85a4 libsystem_malloc.dylib`malloc_zone_realloc + 112
frame numba#8: 0x000000019c1b7110 libsystem_malloc.dylib`realloc + 388
frame numba#9: 0x000000013a9ff0f8 _nrt_python.cpython-39-darwin.so`NRT_MemInfo_varsize_realloc + 60
frame numba#10: 0x000000013d4f41e0
frame numba#11: 0x000000019c393fa8 libsystem_pthread.dylib`_pthread_start + 148
```
esc
added a commit
that referenced
this pull request
May 10, 2024
This fixes a test, where a non-thread safe container is written to
during testing
To reproduce on at least `linux-64` and `osx-arm64` (and probably
others):
```
NUMBA_THREADING_LAYER=workqueue SUBPROC_TEST=1 ./runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting
```
On `linux-64`, this can be debugged with `gdb`:
```
(gdb) bt
closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438
```
On `osx-arm64` we can use `lldb`:
```
(lldb) run runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting
Process 13575 launched: '/Users/esc/miniconda3-arm64/envs/numba_3.9/bin/python3' (arm64)
Parallel: 0. Serial: 1
python3(13575,0x17025b000) malloc: Non-aligned pointer 0x600000256880 being freed (2)
python3(13575,0x17025b000) malloc: *** set a breakpoint in malloc_error_break to debug
Process 13575 stopped
* thread numba#18, stop reason = signal SIGABRT
frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`:
-> 0x19c35c704 <+8>: b.lo 0x19c35c724 ; <+40>
0x19c35c708 <+12>: pacibsp
0x19c35c70c <+16>: stp x29, x30, [sp, #-0x10]!
0x19c35c710 <+20>: mov x29, sp
Target 0: (python3) stopped.
(lldb) bt
* thread numba#18, stop reason = signal SIGABRT
* frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8
frame #1: 0x000000019c393c28 libsystem_pthread.dylib`pthread_kill + 288
frame #2: 0x000000019c2a1ae8 libsystem_c.dylib`abort + 180
frame #3: 0x000000019c1c2e28 libsystem_malloc.dylib`malloc_vreport + 908
frame #4: 0x000000019c1d95d4 libsystem_malloc.dylib`malloc_zone_error + 104
frame #5: 0x000000019c1ca620 libsystem_malloc.dylib`_szone_free + 628
frame #6: 0x000000019c1b87f4 libsystem_malloc.dylib`nanov2_realloc + 356
frame #7: 0x000000019c1b85a4 libsystem_malloc.dylib`malloc_zone_realloc + 112
frame numba#8: 0x000000019c1b7110 libsystem_malloc.dylib`realloc + 388
frame numba#9: 0x000000013a9ff0f8 _nrt_python.cpython-39-darwin.so`NRT_MemInfo_varsize_realloc + 60
frame numba#10: 0x000000013d4f41e0
frame numba#11: 0x000000019c393fa8 libsystem_pthread.dylib`_pthread_start + 148
```
esc
added a commit
that referenced
this pull request
May 10, 2024
This fixes a test, where a non-thread safe container is written to during testing To reproduce on at least `linux-64` and `osx-arm64` (and probably others): ``` NUMBA_THREADING_LAYER=workqueue SUBPROC_TEST=1 ./runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting ``` On `linux-64`, this can be debugged with `gdb`: ``` (gdb) bt 0 0x00007ffff7c8018b in raise () from /lib/x86_64-linux-gnu/libc.so.6 1 0x00007ffff7c5f859 in abort () from /lib/x86_64-linux-gnu/libc.so.6 2 0x00007ffff7cca3ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6 3 0x00007ffff7cd247c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 4 0x00007ffff7cd412c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 5 0x00007ffff7cd6105 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 6 0x00007ffff7cd82d6 in realloc () from /lib/x86_64-linux-gnu/libc.so.6 7 0x00007fffe706de21 in NRT_Reallocate (ptr=0x224aef0, size=2552489952) at numba/core/runtime/nrt.cpp:539 8 0x00007fffe706dcf6 in NRT_MemInfo_varsize_realloc (mi=0x1fe9580, size=2552489952) at numba/core/runtime/nrt.cpp:498 9 0x00007fffdcabde0d in _3cdynamic_3e::__numba_parfor_gufunc_0x7fffdc60eb20[abi:v19][abi:c8tJTC_2fWQAliW1xhDEoY6EEMEUOEMISPGsAQMVj4QniQ4IXKQEMXwoMGLoQDDVsQR1NHAZtvoQrhyQ_2fKR8sTqKIYOQAmjYgkW7ADge6ERATM1UUQpZoA](Array<unsigned long long, 1, C, mutable, aligned>, list_28Tuple_28DictType_5bint64_2cfloat64_5d_3civ_3dNone_3e_2c_20array_28float64_2c_201d_2c_20C_29_29_29_3civ_3dNone_3e) (sched=..., closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438 10 0x00007fffdcab624e in __gufunc__._ZN13_3cdynamic_3e36__numba_parfor_gufunc_0x7fffdc60eb20B3v19B120c8tJTC_2fWQAliW1xhDEoY6EEMEUOEMISPGsAQMVj4QniQ4IXKQEMXwoMGLoQDDVsQR1NHAZtvoQrhyQ_2fKR8sTqKIYOQAmjYgkW7ADge6ERATM1UUQpZoAE5ArrayIyLi1E1C7mutable7alignedE119list_28Tuple_28DictType_5bint64_2cfloat64_5d_3civ_3dNone_3e_2c_20array_28float64_2c_201d_2c_20C_29_29_29_3civ_3dNone_3e () 11 0x00007fffdc926a3b in thread_worker (arg=0x1bd49c0) at numba/np/ufunc/workqueue.c:567 12 0x00007ffff7f8f609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 13 0x00007ffff7d5c293 in clone () from /lib/x86_64-linux-gnu/libc.so.6 ``` On `osx-arm64` we can use `lldb`: ``` (lldb) run runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting Process 13575 launched: '/Users/esc/miniconda3-arm64/envs/numba_3.9/bin/python3' (arm64) Parallel: 0. Serial: 1 python3(13575,0x17025b000) malloc: Non-aligned pointer 0x600000256880 being freed (2) python3(13575,0x17025b000) malloc: *** set a breakpoint in malloc_error_break to debug Process 13575 stopped * thread numba#18, stop reason = signal SIGABRT frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 libsystem_kernel.dylib`: -> 0x19c35c704 <+8>: b.lo 0x19c35c724 ; <+40> 0x19c35c708 <+12>: pacibsp 0x19c35c70c <+16>: stp x29, x30, [sp, #-0x10]! 0x19c35c710 <+20>: mov x29, sp Target 0: (python3) stopped. (lldb) bt * thread numba#18, stop reason = signal SIGABRT * frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 frame #1: 0x000000019c393c28 libsystem_pthread.dylib`pthread_kill + 288 frame #2: 0x000000019c2a1ae8 libsystem_c.dylib`abort + 180 frame #3: 0x000000019c1c2e28 libsystem_malloc.dylib`malloc_vreport + 908 frame #4: 0x000000019c1d95d4 libsystem_malloc.dylib`malloc_zone_error + 104 frame #5: 0x000000019c1ca620 libsystem_malloc.dylib`_szone_free + 628 frame #6: 0x000000019c1b87f4 libsystem_malloc.dylib`nanov2_realloc + 356 frame #7: 0x000000019c1b85a4 libsystem_malloc.dylib`malloc_zone_realloc + 112 frame numba#8: 0x000000019c1b7110 libsystem_malloc.dylib`realloc + 388 frame numba#9: 0x000000013a9ff0f8 _nrt_python.cpython-39-darwin.so`NRT_MemInfo_varsize_realloc + 60 frame numba#10: 0x000000013d4f41e0 frame numba#11: 0x000000019c393fa8 libsystem_pthread.dylib`_pthread_start + 148 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit does three things:
To begin with, there were actually two severe bugs in
is_prceise.First, the negation was missing and secondly the trailing comma meant
that a tuple was being returned. Thus, a list would have always been
deemed to be precise, if there had been any tests for it.
Then, add a simple test to check if refinement works within and outside
of a jit context.
Lastly, After fixing the refinement of the typed-list in a jit context,
it turned out that there is an issue with
overload_method. The issueis that, in case of calling an overloaded method on an imprecise
container type, e.g. typed-list, a function with
overload_methodmustrefine the type and return a tuple of the signature that the
implementation should have had and the implementation. While
overloaddoes have support for receiving such a tuple,
overload_methoddoes notand instead an error regarding a tuple not being callable is raised.
This is fixed now too.