I want to return a typed.Dict() from the function, but it raised error. Can anyone help me?

[classysky]

import numpy as np
import numba

keytype = numba.core.types.float64
float_array = numba.core.types.float32[:]


@numba.njit(cache=True, parallel=True)
def mydict():
    mydict = numba.typed.Dict.empty(keytype, float_array)
    v1 = np.array([0.1, 0.2], dtype=np.float32)
    for i in numba.prange(100000):
        mydict[i] = v1
    return mydict


mydict()

version:
numba=0.58.1
numpy=1.25.1

the error is:
[Done] exited with code=3221225477 in 1.75 seconds
sometimes error would become:
[Done] exited with code=3221226356 in 0.632 seconds

PS: If remove the code "return tdict" or reduce the iteration number 100000 to 100, it would not raise the error. I need to output the typed.Dict() and the iteration number is large generally.

[guilhermeleobas]

@classysky, thanks for the report. I can reproduce the error.

Error is coming from numba_dict_resize:

$ lldb python
(lldb) target create "python"
Current executable set to 'python' (arm64).
(lldb) b malloc_error_break
Breakpoint 1: where = libsystem_malloc.dylib`malloc_error_break, address = 0x0000000180155120
(lldb) run e.py
Process 50244 launched: '/Users/guilhermeleobas/micromamba/envs/numba/bin/python' (arm64)
nrt_allocate_meminfo_and_data 0x0
NRT_Allocate_External bytes=56 ptr=0x600001735d80
NRT_MemInfo_alloc_dtor 0x600001735db0 8
NRT_MemInfo_init mi=0x600001735d80 external_allocator=0x0
nrt_allocate_meminfo_and_data_align 0x0
nrt_allocate_meminfo_and_data 0x0
NRT_Allocate_External bytes=120 ptr=0x600002c1a300
NRT_MemInfo_alloc_aligned 0x600002c1a340
NRT_MemInfo_init mi=0x600002c1a300 external_allocator=0x0
python(50244,0x1736bf000) malloc: *** error for object 0x100aca200: pointer being freed was not allocated
python(50244,0x1736bf000) malloc: *** set a breakpoint in malloc_error_break to debug
python(50244,0x17374b000) malloc: *** error for object 0x100ad8e00: pointer being freed was not allocated
python(50244,0x173a93000) malloc: *** error for object 0x100ad8e00: pointer being freed was not allocated
python(50244,0x1738ef000) malloc: *** error for object 0x100ad8e00: pointer being freed was not allocated
python(50244,0x173a93000) malloc: *** set a breakpoint in malloc_error_break to debug
python(50244,0x173a07000) malloc: *** error for object 0x100ad8e00: pointer being freed was not allocated
python(50244,0x173a07000) malloc: *** set a breakpoint in malloc_error_break to debug
Process 50244 stopped
* thread #9, stop reason = breakpoint 1.1
    frame #0: 0x000000018956d120 libsystem_malloc.dylib`malloc_error_break
libsystem_malloc.dylib`malloc_error_break:
->  0x18956d120 <+0>:  pacibsp
    0x18956d124 <+4>:  stp    x29, x30, [sp, #-0x10]!
    0x18956d128 <+8>:  mov    x29, sp
    0x18956d12c <+12>: nop
  thread #15, stop reason = breakpoint 1.1
    frame #0: 0x000000018956d120 libsystem_malloc.dylib`malloc_error_break
libsystem_malloc.dylib`malloc_error_break:
->  0x18956d120 <+0>:  pacibsp
    0x18956d124 <+4>:  stp    x29, x30, [sp, #-0x10]!
    0x18956d128 <+8>:  mov    x29, sp
    0x18956d12c <+12>: nop
  thread #16, stop reason = breakpoint 1.1
    frame #0: 0x000000018956d120 libsystem_malloc.dylib`malloc_error_break
libsystem_malloc.dylib`malloc_error_break:
->  0x18956d120 <+0>:  pacibsp
    0x18956d124 <+4>:  stp    x29, x30, [sp, #-0x10]!
    0x18956d128 <+8>:  mov    x29, sp
    0x18956d12c <+12>: nop
Target 0: (python) stopped.
(lldb) bt
* thread #9, stop reason = breakpoint 1.1
  * frame #0: 0x000000018956d120 libsystem_malloc.dylib`malloc_error_break
    frame #1: 0x000000018955e844 libsystem_malloc.dylib`malloc_vreport + 428
    frame #2: 0x0000000189561f34 libsystem_malloc.dylib`malloc_report + 64
    frame #3: 0x0000000189550ca0 libsystem_malloc.dylib`free + 300
    frame #4: 0x000000014000c1d0 _helperlib.cpython-39-darwin.so`numba_dict_resize + 460
    frame #5: 0x000000014000bdac _helperlib.cpython-39-darwin.so`numba_dict_insert + 224
    frame #6: 0x0000000101ff4b10
    frame #7: 0x000000018974226c libsystem_pthread.dylib`_pthread_start + 148

edit: Just realized you're using parallel=True. Wondering if the code on dictobject.c is thread safe.

[classysky]

@guilhermeleobas Thank you for your works. I think the code is thread safe since it works when the iteration number is 10000.

[classysky]

import numpy as np
import numba

keytype = numba.core.types.float64
float_array = numba.core.types.float32[:]


@numba.njit(cache=True, parallel=True)
def mydict_fun(mydict):
    calculation_resulst = np.array([0.1, 0.2], dtype=np.float32)
    for i in numba.prange(100000):
        mydict[i] = calculation_resulst
    return mydict


mydict = numba.typed.Dict.empty(keytype, float_array)
initial_array = np.array([0.1, 0.2], dtype=np.float32)
for i in range(100000):
    mydict[i] = initial_array
mydict_fun(mydict)

I try to initialize a dictionary and pass it to the function. It works.

[stuartarchibald]

The typed containers are currently not thread-safe, concurrent writes to a container that would cause a resize of the container would lead to invalid state occurring (threads trying to resize the underlying storage for the container whilst other threads are reading from/writing to it). This is documented here:

numba/docs/source/reference/pysupported.rst

Lines 795 to 800 in 609c8c9

	It should be noted that ``numba.typed.Dict`` is not thread-safe.
	Specifically, functions which modify a dictionary from multiple
	threads will potentially corrupt memory, causing a
	range of possible failures. However, the dictionary can be safely read from
	multiple threads as long as the contents of the dictionary do not
	change during the parallel access.

It looks like you've found a workaround, will close this ticket. Thanks for using Numba!