initial stab at docs for typed-list #7

esc · 2019-07-01T21:14:48Z

As title.

Improve the wording somewhat.

stuartarchibald

Thanks for these docs, they look good. I think that this is about the right level of information given this is experimental and users need to know about it and developer level detail would be wasted effort until the impl is concrete! I've made a few suggestions to consider before merge. Thanks.

docs/source/reference/pysupported.rst

stuartarchibald · 2019-07-05T15:26:09Z

docs/source/reference/pysupported.rst

+    As of version 0.45.0 the internal implementation for the list datatype in
+    Numba is changing. Until recently, only a single implementation of the list
+    datatype was available, the so-called *reflected-list* (see below).
+    However, it was deprecated from version 0.44.0 onwards due to its


IIRC It was not deprecated, it was scheduled for deprecation, see schedule: http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#schedule

Correct, I have reworded this and added a link to the deprecation schedule at the end of the warning.

stuartarchibald · 2019-07-05T15:49:12Z

docs/source/reference/pysupported.rst

+.. warning::
+    As of version 0.45.0 the internal implementation for the list datatype in
+    Numba is changing. Until recently, only a single implementation of the list
+    datatype was available, the so-called *reflected-list* (see below).


Perhaps reference the documentation on reflection?

The documentation for the that is almost exactly below, I am hoping this relative reference should suffice.

What I can do, is reference the deprecation schedule.

docs/source/reference/pysupported.rst

examples/typed_list_usage.py

Wording fixes and improvements. Co-Authored-By: stuartarchibald <stuartarchibald@users.noreply.github.com>

This commit adds a link to the deprecation schedule and removes a debatable statement about the efficiency of the typed-list conatiner. Also, some typos are fixed.

esc · 2019-07-06T18:52:58Z

@stuartarchibald thanks for the review, I have addressed all the comments and will now merge this into the typed-list branch proper.

This fixes a test, where a non-thread safe container is written to during testing To reproduce on at least `linux-64` and `osx-arm64` (and probably others): ``` NUMBA_THREADING_LAYER=workqueue SUBPROC_TEST=1 ./runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting ``` On `linux-64`, this can be debugged with `gdb`: ``` (gdb) bt closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438 (gdb) f 9 closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438 4438 <string>: No such file or directory. (gdb) i args sched = {meminfo = 0x0, parent = 0x0, nitems = 0, itemsize = 0, data = 0x0, shape = {0}, strides = {0}} closure____locals______listcomp____v15____v2build__list__0 = {meminfo = 0x0, parent = 0x0} (gdb) f 8 498 mi->data = NRT_Reallocate(mi->data, size); (gdb) i args mi = 0x21d1660 size = 2510933856 ``` On `osx-arm64` we can use `lldb`: ``` (lldb) run runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting Process 13575 launched: '/Users/esc/miniconda3-arm64/envs/numba_3.9/bin/python3' (arm64) Parallel: 0. Serial: 1 python3(13575,0x17025b000) malloc: Non-aligned pointer 0x600000256880 being freed (2) python3(13575,0x17025b000) malloc: *** set a breakpoint in malloc_error_break to debug Process 13575 stopped * thread numba#18, stop reason = signal SIGABRT frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 libsystem_kernel.dylib`: -> 0x19c35c704 <+8>: b.lo 0x19c35c724 ; <+40> 0x19c35c708 <+12>: pacibsp 0x19c35c70c <+16>: stp x29, x30, [sp, #-0x10]! 0x19c35c710 <+20>: mov x29, sp Target 0: (python3) stopped. (lldb) bt * thread numba#18, stop reason = signal SIGABRT * frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 frame #1: 0x000000019c393c28 libsystem_pthread.dylib`pthread_kill + 288 frame #2: 0x000000019c2a1ae8 libsystem_c.dylib`abort + 180 frame #3: 0x000000019c1c2e28 libsystem_malloc.dylib`malloc_vreport + 908 frame #4: 0x000000019c1d95d4 libsystem_malloc.dylib`malloc_zone_error + 104 frame #5: 0x000000019c1ca620 libsystem_malloc.dylib`_szone_free + 628 frame #6: 0x000000019c1b87f4 libsystem_malloc.dylib`nanov2_realloc + 356 frame #7: 0x000000019c1b85a4 libsystem_malloc.dylib`malloc_zone_realloc + 112 frame numba#8: 0x000000019c1b7110 libsystem_malloc.dylib`realloc + 388 frame numba#9: 0x000000013a9ff0f8 _nrt_python.cpython-39-darwin.so`NRT_MemInfo_varsize_realloc + 60 frame numba#10: 0x000000013d4f41e0 frame numba#11: 0x000000019c393fa8 libsystem_pthread.dylib`_pthread_start + 148 ```

This fixes a test, where a non-thread safe container is written to during testing To reproduce on at least `linux-64` and `osx-arm64` (and probably others): ``` NUMBA_THREADING_LAYER=workqueue SUBPROC_TEST=1 ./runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting ``` On `linux-64`, this can be debugged with `gdb`: ``` (gdb) bt closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438 ``` On `osx-arm64` we can use `lldb`: ``` (lldb) run runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting Process 13575 launched: '/Users/esc/miniconda3-arm64/envs/numba_3.9/bin/python3' (arm64) Parallel: 0. Serial: 1 python3(13575,0x17025b000) malloc: Non-aligned pointer 0x600000256880 being freed (2) python3(13575,0x17025b000) malloc: *** set a breakpoint in malloc_error_break to debug Process 13575 stopped * thread numba#18, stop reason = signal SIGABRT frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 libsystem_kernel.dylib`: -> 0x19c35c704 <+8>: b.lo 0x19c35c724 ; <+40> 0x19c35c708 <+12>: pacibsp 0x19c35c70c <+16>: stp x29, x30, [sp, #-0x10]! 0x19c35c710 <+20>: mov x29, sp Target 0: (python3) stopped. (lldb) bt * thread numba#18, stop reason = signal SIGABRT * frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 frame #1: 0x000000019c393c28 libsystem_pthread.dylib`pthread_kill + 288 frame #2: 0x000000019c2a1ae8 libsystem_c.dylib`abort + 180 frame #3: 0x000000019c1c2e28 libsystem_malloc.dylib`malloc_vreport + 908 frame #4: 0x000000019c1d95d4 libsystem_malloc.dylib`malloc_zone_error + 104 frame #5: 0x000000019c1ca620 libsystem_malloc.dylib`_szone_free + 628 frame #6: 0x000000019c1b87f4 libsystem_malloc.dylib`nanov2_realloc + 356 frame #7: 0x000000019c1b85a4 libsystem_malloc.dylib`malloc_zone_realloc + 112 frame numba#8: 0x000000019c1b7110 libsystem_malloc.dylib`realloc + 388 frame numba#9: 0x000000013a9ff0f8 _nrt_python.cpython-39-darwin.so`NRT_MemInfo_varsize_realloc + 60 frame numba#10: 0x000000013d4f41e0 frame numba#11: 0x000000019c393fa8 libsystem_pthread.dylib`_pthread_start + 148 ```

This fixes a test, where a non-thread safe container is written to during testing To reproduce on at least `linux-64` and `osx-arm64` (and probably others): ``` NUMBA_THREADING_LAYER=workqueue SUBPROC_TEST=1 ./runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting ``` On `linux-64`, this can be debugged with `gdb`: ``` (gdb) bt 0 0x00007ffff7c8018b in raise () from /lib/x86_64-linux-gnu/libc.so.6 1 0x00007ffff7c5f859 in abort () from /lib/x86_64-linux-gnu/libc.so.6 2 0x00007ffff7cca3ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6 3 0x00007ffff7cd247c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 4 0x00007ffff7cd412c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 5 0x00007ffff7cd6105 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 6 0x00007ffff7cd82d6 in realloc () from /lib/x86_64-linux-gnu/libc.so.6 7 0x00007fffe706de21 in NRT_Reallocate (ptr=0x224aef0, size=2552489952) at numba/core/runtime/nrt.cpp:539 8 0x00007fffe706dcf6 in NRT_MemInfo_varsize_realloc (mi=0x1fe9580, size=2552489952) at numba/core/runtime/nrt.cpp:498 9 0x00007fffdcabde0d in _3cdynamic_3e::__numba_parfor_gufunc_0x7fffdc60eb20[abi:v19][abi:c8tJTC_2fWQAliW1xhDEoY6EEMEUOEMISPGsAQMVj4QniQ4IXKQEMXwoMGLoQDDVsQR1NHAZtvoQrhyQ_2fKR8sTqKIYOQAmjYgkW7ADge6ERATM1UUQpZoA](Array<unsigned long long, 1, C, mutable, aligned>, list_28Tuple_28DictType_5bint64_2cfloat64_5d_3civ_3dNone_3e_2c_20array_28float64_2c_201d_2c_20C_29_29_29_3civ_3dNone_3e) (sched=..., closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438 10 0x00007fffdcab624e in __gufunc__._ZN13_3cdynamic_3e36__numba_parfor_gufunc_0x7fffdc60eb20B3v19B120c8tJTC_2fWQAliW1xhDEoY6EEMEUOEMISPGsAQMVj4QniQ4IXKQEMXwoMGLoQDDVsQR1NHAZtvoQrhyQ_2fKR8sTqKIYOQAmjYgkW7ADge6ERATM1UUQpZoAE5ArrayIyLi1E1C7mutable7alignedE119list_28Tuple_28DictType_5bint64_2cfloat64_5d_3civ_3dNone_3e_2c_20array_28float64_2c_201d_2c_20C_29_29_29_3civ_3dNone_3e () 11 0x00007fffdc926a3b in thread_worker (arg=0x1bd49c0) at numba/np/ufunc/workqueue.c:567 12 0x00007ffff7f8f609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 13 0x00007ffff7d5c293 in clone () from /lib/x86_64-linux-gnu/libc.so.6 ``` On `osx-arm64` we can use `lldb`: ``` (lldb) run runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting Process 13575 launched: '/Users/esc/miniconda3-arm64/envs/numba_3.9/bin/python3' (arm64) Parallel: 0. Serial: 1 python3(13575,0x17025b000) malloc: Non-aligned pointer 0x600000256880 being freed (2) python3(13575,0x17025b000) malloc: *** set a breakpoint in malloc_error_break to debug Process 13575 stopped * thread numba#18, stop reason = signal SIGABRT frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 libsystem_kernel.dylib`: -> 0x19c35c704 <+8>: b.lo 0x19c35c724 ; <+40> 0x19c35c708 <+12>: pacibsp 0x19c35c70c <+16>: stp x29, x30, [sp, #-0x10]! 0x19c35c710 <+20>: mov x29, sp Target 0: (python3) stopped. (lldb) bt * thread numba#18, stop reason = signal SIGABRT * frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 frame #1: 0x000000019c393c28 libsystem_pthread.dylib`pthread_kill + 288 frame #2: 0x000000019c2a1ae8 libsystem_c.dylib`abort + 180 frame #3: 0x000000019c1c2e28 libsystem_malloc.dylib`malloc_vreport + 908 frame #4: 0x000000019c1d95d4 libsystem_malloc.dylib`malloc_zone_error + 104 frame #5: 0x000000019c1ca620 libsystem_malloc.dylib`_szone_free + 628 frame #6: 0x000000019c1b87f4 libsystem_malloc.dylib`nanov2_realloc + 356 frame #7: 0x000000019c1b85a4 libsystem_malloc.dylib`malloc_zone_realloc + 112 frame numba#8: 0x000000019c1b7110 libsystem_malloc.dylib`realloc + 388 frame numba#9: 0x000000013a9ff0f8 _nrt_python.cpython-39-darwin.so`NRT_MemInfo_varsize_realloc + 60 frame numba#10: 0x000000013d4f41e0 frame numba#11: 0x000000019c393fa8 libsystem_pthread.dylib`_pthread_start + 148 ```

esc added 3 commits July 1, 2019 16:13

initial stab at docs for typed-list

4a6ceea

As title.

adding some typed-list examples

405c81d

tweak the docs for the typed-list

a021aa0

Improve the wording somewhat.

stuartarchibald reviewed Jul 5, 2019

View reviewed changes

esc and others added 2 commits July 6, 2019 13:28

Apply suggestions from code review

3983da2

Wording fixes and improvements. Co-Authored-By: stuartarchibald <stuartarchibald@users.noreply.github.com>

rewording and typos for typed-list

fbebeea

This commit adds a link to the deprecation schedule and removes a debatable statement about the efficiency of the typed-list conatiner. Also, some typos are fixed.

esc merged commit 803c186 into typed_list Jul 6, 2019

esc deleted the typed_list_docs branch July 6, 2019 18:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial stab at docs for typed-list #7

initial stab at docs for typed-list #7

esc commented Jul 1, 2019

stuartarchibald left a comment

stuartarchibald Jul 5, 2019

esc Jul 6, 2019

stuartarchibald Jul 5, 2019

esc Jul 6, 2019

esc Jul 6, 2019

esc commented Jul 6, 2019

initial stab at docs for typed-list #7

initial stab at docs for typed-list #7

Conversation

esc commented Jul 1, 2019

stuartarchibald left a comment

Choose a reason for hiding this comment

stuartarchibald Jul 5, 2019

Choose a reason for hiding this comment

esc Jul 6, 2019

Choose a reason for hiding this comment

stuartarchibald Jul 5, 2019

Choose a reason for hiding this comment

esc Jul 6, 2019

Choose a reason for hiding this comment

esc Jul 6, 2019

Choose a reason for hiding this comment

esc commented Jul 6, 2019