Type inference error for dict when using default value

[ivirshup]

Reporting a bug

• I am using the latest released version of Numba (most recent is visible in
  the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
• I have included below a minimal working reproducer (if you are unsure how
  to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

Numba fails to infer return type of Dict.get or Dict.setdefault if that value (or value generated downstream) is set to the Dict.

import numba

@numba.njit
def foo(x):
    d = dict()
    for i in x:
        val = d.get(i, default=0)
        d[i] = val + 1
    return d

foo([1, 2, 3]) # Throws TypingError

traceback

---------------------------------------------------------------------------
TypingError                               Traceback (most recent call last)
<ipython-input-114-3a9000771a91> in <module>
----> 1 foo([1, 2, 3])

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
    349                 e.patch_message(msg)
    350 
--> 351             error_rewrite(e, 'typing')
    352         except errors.UnsupportedError as e:
    353             # Something unsupported is present in the user code, add help info

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in error_rewrite(e, issue_type)
    316                 raise e
    317             else:
--> 318                 reraise(type(e), e, None)
    319 
    320         argtypes = []

/usr/local/lib/python3.7/site-packages/numba/six.py in reraise(tp, value, tb)
    656             value = tp()
    657         if value.__traceback__ is not tb:
--> 658             raise value.with_traceback(tb)
    659         raise value
    660 

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
cannot safely cast int64 to undefined
[1] During: resolving callee type: BoundFunction((<class 'numba.types.containers.DictType'>, 'get') for DictType[undefined,undefined])
[2] During: typing of call at <ipython-input-113-dda3043518e0> (5)


File "<ipython-input-113-dda3043518e0>", line 5:
def foo(x):
    <source elided>
    for i in x:
        val = d.get(i, 0)

However, numba seems to be able to infer the type just fine as long as the value isn't assigned back into the dict:

@numba.njit
def bar(x):
    d = dict()
    for i in x:
        val = d.get(i, default=0)
        print(type(val))
        d[i] = 1
    return d

_ = bar([1,2,3])
# int64
# int64
# int64

Assigning the value to another dict also fails inference, but other operations I've tried (like appending to a list) seem to work fine. Additionally, if I assign a value before trying .get or .setdefault, inference works.

[esc]

@ivirshup thanks for reporting this! In essence the typed dictionary is of undefined type until you add something to it. In that case, the dict will be refined and will have a fixed type associated with it's keys and values. However, in this case it seems like the refinement is currently not working for d.get. As a temporary workaround you can, as you suggested, add a value to the dict and remove it again to refine it for further use.

cc @sklam

[ivirshup]

No worries. I ended up figuring out that I could set the type of the Dict if I declared the dependent type I'm using (tuple of ints) outside of the compiled code.

[esc]

@ivirshup good to hear, will still leave this open to track the bug

[ivirshup]

Minor update – I've come across a similar case where type inference does work:

import numba

@numba.njit
def foo(x):
    d = {}
    for k in x:
        if k in d:
            d[k] += 1
        else:
            d[k] = 1
    return d

d = foo(list("aabc"))
d
# DictType[unicode_type,int64]({a: 2, b: 1, c: 1})

# So we can just do:
@numba.njit
def get(d, k, default=None):
    if k in d:
        return d[k]
    else:
        return default

assert get(d, "d") is None
assert get(d, "d", 6) == 6
assert get(d, "a") == 2
assert get(d, "a", 6) == 2

[BarclayII]

A similar issue when interfacing with numpy arrays and numba.typed.Dict.get:

# crashes
@njit("float64[:](int64[:], float64[:], int64[:])")
def a(x, y, z):
    d = Dict.empty(key_type=types.int64, value_type=types.float64)
    for i in range(len(x)):
        d[x[i]] = y[i]
    return np.array([d.get(_z, default=0.) for _z in z])

Error message:

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
No conversion from array(OptionalType(float64) i.e. the type 'float64 or None', 1d, C) to array(float64, 1d, A) for '$56.12', defined at None

File "<ipython-input-32-96d816327c42>", line 6:
def a(x, y, z):
    <source elided>
        d[x[i]] = y[i]
    return np.array([d.get(_z, default=0.) for _z in z])
    ^

[1] During: typing of assignment at <ipython-input-32-96d816327c42> (6)

File "<ipython-input-32-96d816327c42>", line 6:
def a(x, y, z):
    <source elided>
        d[x[i]] = y[i]
    return np.array([d.get(_z, default=0.) for _z in z])
    ^