Use of lists for shape args in empty(), zero(), Ones()

[cems2]

Noting an inconsistency between numba and numpy in the type of arguments allowed for empty(), zero() and ones() and probably others.

This is not a bug per se, just a case were generally expected behaviors don't match between numba and numpy.

In numpy the documentation states that the shape argument of numpy.empty is a tuple of dimensions.

However although not documented, numpy will also take a list in place of a tuple.

Numba Will only take a tuple. It will not take a list.

This is both good yet unfortunate.

It's bad because it breaks a lot of existing code that, albeit sloppy, depends on a list being accepted in numpy.

I think is it probably good because Cuda types will work better if Tuples are used I believe.

The real problem is the error output is a bit cryptic and the result unexpected by experienced numpy users.

CODE TO REPRODUCE

def testempty_List(i):
    moo = np.empty([i,i])
    return moo

def testempty_Tuple(i):
    moo = np.empty((i,i))
    return moo

# pure numpy
testempty_Tuple(3)   
testempty_List(3)                # this will not give an error
        
nb.njit(testempty_Tuple)(3)
nb.njit(testempty_List)(3)     # this will give an error

[stuartarchibald]

Thanks for the report. The issue here is that Numba has to determine the types of everything to be able to compile the code. Numba "knows" that a type is a list, but has no idea how many entries are in the list at compile time, and, as a result, if used in an array constructor, it has no idea what shape the array should be. I'd recommend reading the discussion here: #2771 which discusses implementing the tuple() constructor which suffers from the same problem. Any fix for #2771 is likely to also be applicable to the ndarray allocation routines mentioned.

[stuartarchibald]

As to:

The real problem is the error output is a bit cryptic and the result unexpected by experienced numpy users.

I get this:

numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<built-in function empty>) with argument(s) of type(s): (list(int64))
 * parameterized
In definition 0:
    All templates rejected with literals.
In definition 1:
    All templates rejected without literals.
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: resolving callee type: Function(<built-in function empty>)
[2] During: typing of call at issue3993.py (6)


File "issue3993.py", line 6:
def testempty_List(i):
    moo = np.empty([i,i])
    ^

do you have any suggestions for improvements please? Feedback is much appreciated.

I've been on-and-off working on #3942 to help here, at present it shows this for the above:

numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<built-in function empty>) with argument(s) of type(s): (list(int64)).

There were 1 definitions(s) that responded with:

    All templates rejected with literals.

There were 1 definitions(s) that responded with:

    All templates rejected without literals.


No concrete type signatures were found.

In addition, undetermined parameterised signatures were found.

This error is usually caused by passing an argument of a type that is unsupported by the named function.

HINT: Given argument type(s) were (list(int64)) and the NumPy function 'numpy.empty' is supported for the following argument type(s):

 * numpy.empty(any, any)
 * numpy.empty(any)


NOTE: Hinting is experimental, you can switch it off by setting the environment variable NUMBA_SHOW_HINTS to 0 or by adding "show_hints: 0" to your .numba_config.yaml configuration file. See http://numba.pydata.org/numba-doc/latest/reference/envvars.html for details of both.


[1] During: resolving callee type: Function(<built-in function empty>)
[2] During: typing of call at issue3993.py (6)


File "issue3993.py", line 6:
def testempty_List(i):
    moo = np.empty([i,i])
    ^

again, feedback is welcomed.

[astrojuanlu]

For me, the most common error is using np.zeros((...), dtype=np.int) instead of, say, np.zeros((...), dtype=np.int_). Perhaps that can be included in the hinting as well.

[stuartarchibald]

@Juanlu001 thanks for the feedback, I think what you are after is something that especially checks the dtype and if the type is identified as a Function class type it hints that you probably meant an equivalent concrete NumPy type? If my assumption is correct, please could you open a ticket to specifically request that, it's not quite the same sort of code path as #3942? Thanks.

[cems2]

One of the problem I have with enviroment vars is that I haven't figured out when they get invoked in conda envs and jupyter notebooks. Is there some stateful way to make it switchable with numba itself? Also I 'm still puzling over Juan's exampe as I don't yet know what's wrong with what he wrote :-)

…

On Apr 19, 2019, at 5:04 AM, stuartarchibald ***@***.***> wrote: @Juanlu001 <https://github.com/Juanlu001> thanks for the feedback, I think what you are after is something that especially checks the dtype and if the type is identified as a Function class type it hints that you probably meant an equivalent concrete NumPy type? If my assumption is correct, please could you open a ticket to specifically request that, it's not quite the same sort of code path as #3942 <#3942>? Thanks. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3993 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACRAR7QTIDU2XLICACLUCU3PRGRMDANCNFSM4HG7K7FQ>.

[cems2]

Some feed back: here's what the hint says HINT: Given argument type(s) were (list(int64)) and the NumPy function 'numpy.empty' is supported for the following argument type(s): * numpy.empty(any, any) * numpy.empty(any) I can't figure out what it's saying. It seems like it is saying the empty takes "any" argument and then above that it says it doesn't take a list or perhaps a list(int64). THe good thing about this hint is the the first sentence nails the error correctly so that's good. It's the second part that obfuscates that with "any". My sense is that just as it takes a while to learn to read the tea leaves of a numba Error traceback, that there will be a learning curve for deciphering hints. So It's probably just fine to have this seemingly contradictory hint in this format. When you have a hard to spot bug what you really want is some set of scooby clues follow. So offering more info is better even if some clues are red herrings. The first line's clue might possibly have been even more informative. FOr example, the fact that it's a list is the actual problem. Not that it's a list of int64. thus omitting the int64 part would have give a better pointer. To see why consider if the error had been because I had used a tuple of floats or strings as the shape parameter. THen the error would not be the tuple itself but the dtype fof the contents. So sometimes the error will be the outer part (list) and sometimes the inner part (not int64). Anyhow that might beyond the hinting capacity to post mortem. And as I said, any hint, even a partly bad hint, is better than no hint.

…

---- regarding my last comment about Environment variables. Since I develop inside jupyter notebooks environment variables are a nuiscance as they don't seem to always stick. And exiting the notebook to set them and relaunch means you lost the state that was causeing an error (perhaps the error is caused by some hard to reproduce condition that is producing a Nan). This means you can't keep a hold on the error and also switch on the enviroment varialbles until you can write code to reproduce the error every time-- which is half the battle of debugging anyhow. If there were a way to switch numba's state directly then one could switch this on and off while interactively debugging. this applies to things like cuda emulators too.

On Apr 19, 2019, at 8:38 AM, Charlie Strauss ***@***.***> wrote: One of the problem I have with enviroment vars is that I haven't figured out when they get invoked in conda envs and jupyter notebooks. Is there some stateful way to make it switchable with numba itself? Also I 'm still puzling over Juan's exampe as I don't yet know what's wrong with what he wrote :-) > On Apr 19, 2019, at 5:04 AM, stuartarchibald ***@***.*** ***@***.***>> wrote: > > @Juanlu001 <https://github.com/Juanlu001> thanks for the feedback, I think what you are after is something that especially checks the dtype and if the type is identified as a Function class type it hints that you probably meant an equivalent concrete NumPy type? If my assumption is correct, please could you open a ticket to specifically request that, it's not quite the same sort of code path as #3942 <#3942>? Thanks. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub <#3993 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACRAR7QTIDU2XLICACLUCU3PRGRMDANCNFSM4HG7K7FQ>. >

[stuartarchibald]

Thanks for the feedback.

It's the second part that obfuscates that with "any".

Yes, this is an unfortunate consequence of the data that is available, hence it's not ready for production etc. In this case, the specification of types accepted is very loose and doesn't really offer useful information, in other cases the hint can be very specific about what is accepted. This in general is a tricky problem to solve but we have some ideas #3855.

My sense is that just as it takes a while to learn to read the tea leaves of a numba Error traceback, that there will be a learning curve for deciphering hints.

We're trying hard to make it less mysterious and ideas of things that would help are always welcome. Part of the challenge is always that Numba doesn't behave like e.g. a statically typed language compiler (e.g. C or Fortran) which a lot of people are familiar with, as Python is dynamic and Numba has to do a lot of work to compute the types of everything to get to the point where it can behave like a static typed language compiler.

The first line's clue might possibly have been even more informative. FOr example, the fact that it's a list is the actual problem. Not that it's a list of int64. thus omitting the int64 part would have give a better pointer. To see why consider if the error had been because I had used a tuple of floats or strings as the shape parameter. THen the error would not be the tuple itself but the dtype fof the contents. So sometimes the error will be the outer part (list) and sometimes the inner part (not int64).

The idea of telling the user what was supplied is because it's not always obvious what the type inference mechanism has decided a type should be. Though I do see your point about it being the container class opposed to a specific typed instance of the container class being the problem, I think eventually that could be resolved in the hinting. The example above, whilst matching your code was perhaps less useful, a better one might be:

@njit
def foo(x):
    x[8.3] # oops, float index on a list!?
    return x
foo([1,2,3])

which gives:

Invalid use of Function(<built-in function getitem>) with argument(s) of type(s): (reflected list(int64), float64).

There were 4 definitions(s) that responded with:

    All templates rejected with literals.

There were 4 definitions(s) that responded with:

    All templates rejected without literals.


No concrete type signatures were found.

In addition, undetermined parameterised signatures were found.

This error is usually caused by passing an argument of a type that is unsupported by the named function.

HINT: Given argument type(s) were (reflected list(int64), float64) and the function 'getitem' from the module '_operator' is supported for the following argument type(s):

 * _operator.getitem(Buffer, SliceType)
 * _operator.getitem(Buffer, Integer)
 * _operator.getitem(Buffer, BaseTuple)
 * _operator.getitem(Buffer, Array)
 * _operator.getitem(NumpyFlatType, Integer)
 * _operator.getitem(CPointer, Integer)
 * _operator.getitem(List, Integer)
 * _operator.getitem(List, SliceType)
 * _operator.getitem(NamedUniTuple, uint64)
 * _operator.getitem(NamedUniTuple, int64)
 * _operator.getitem(UniTuple, uint64)
 * _operator.getitem(UniTuple, int64)

[stuartarchibald]

(HINT: If you put -- in markdown it folds the reply, so RE environment variables...)

regarding my last comment about Environment variables. Since I develop inside jupyter notebooks environment variables are a nuiscance as they don't seem to always stick. And exiting the notebook to set them and relaunch means you lost the state that was causeing an error (perhaps the error is caused by some hard to reproduce condition that is producing a Nan). This means you can't keep a hold on the error and also switch on the enviroment varialbles until you can write code to reproduce the error every time-- which is half the battle of debugging anyhow. If there were a way to switch numba's state directly then one could switch this on and off while interactively debugging. this applies to things like cuda emulators too.

Please could you open a ticket for this? This is a reasonable request but won't be possible for all flags right now due to the way Numba has to load shared libraries and instantiate some global state in them. Thanks.

[astrojuanlu]

Also I 'm still puzling over Juan's exampe as I don't yet know what's wrong with what he wrote :-)

I arrived here by Googling "All templates rejected with literals" after failing to run these functions:

@njit
def foo():
  return np.zeros((2, 2), dtype=int)

@njit
def foo():
  return np.zeros((2, 2), dtype=np.int)

Because both of them raise this error:

numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<built-in function zeros>) with argument(s) of type(s): (tuple(int64 x 2), dtype=Function(<class 'int'>))
 * parameterized
In definition 0:
    All templates rejected with literals.
In definition 1:
    All templates rejected without literals.
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: resolving callee type: Function(<built-in function zeros>)
[2] During: typing of call at <stdin> (3)

It turns out I was using dtypes wrong:

>>> @njit
... def foo():
...   return np.zeros((2, 2), dtype=np.int_)
... 
>>> foo()
array([[0, 0],
       [0, 0]])

[pfeatherstone]

I had a similar error. I was using np.float instead of np.float_. I would have thought the lexer would detect this

[stuartarchibald]

@pfeatherstone indeed, this should ideally be caught. However, note Numba works on bytecode, CPython has already done the lexicographical analysis etc, https://numba.readthedocs.io/en/stable/developer/architecture.html

[stuartarchibald]

@pfeatherstone @astrojuanlu #6184 tracks improving the error message for this use case, it's a "good first issue" if either of you are interested in contributing to Numba? The issue also contains a starter patch. Thanks.