custom logging level for numba internals.

[stuartarchibald]

Numba could do with a custom logging level to use internally so that developers of numba can switch on logging but it doesn't impact consumers of numba with e.g. logging.DEBUG enabled.

[stuartarchibald]

The task here is to create a custom logging level for Numba (something below logging.DEBUG) and then use that throughout the code base where ever logging is currently writing into the logging.DEBUG level.

[benjaminr]

Happy to take this one on!

[stuartarchibald]

Great, thanks @benjaminr

[tobiasraabe]

Hi,
I am having an issue with Numba for some time now. Whenever I run pytest and some test fails, Numba produces an unwieldy amount of logging information. Here is a tiny sample:

DEBUG    numba.core.ssa:ssa.py:178 Running <numba.core.ssa._GatherDefsHandler object at 0x0000025067900160>
DEBUG    numba.core.ssa:ssa.py:180 on stmt: $60load_global.0 = global(AssertionError: <class 'AssertionError'>)
DEBUG    numba.core.ssa:ssa.py:180 on stmt: $const62.1 = const(str, internal dict error during lookup)
DEBUG    numba.core.ssa:ssa.py:180 on stmt: $64call_function.2 = call $60load_global.0($const62.1, func=$60load_global.0, args=[Var($const62.1, dictobject.py:735)], kws=(), vararg=None)
DEBUG    numba.core.ssa:ssa.py:180 on stmt: <static> raise <class 'AssertionError'>('internal dict error during lookup')
DEBUG    numba.core.ssa:ssa.py:149 ==== SSA block analysis pass on 70
DEBUG    numba.core.ssa:ssa.py:178 Running <numba.core.ssa._GatherDefsHandler object at 0x0000025067900160>
DEBUG    numba.core.ssa:ssa.py:180 on stmt: $70load_global.0 = global(_nonoptional: <intrinsic _nonoptional>)
DEBUG    numba.core.ssa:ssa.py:180 on stmt: $74call_function.2 = call $70load_global.0(val, func=$70load_global.0, args=[Var(val, dictobject.py:731)], kws=(), vararg=None)
DEBUG    numba.core.ssa:ssa.py:180 on stmt: $76return_value.3 = cast(value=$74call_function.2)
DEBUG    numba.core.ssa:ssa.py:180 on stmt: return $76return_value.3
DEBUG    numba.core.ssa:ssa.py:141 defs defaultdict(<class 'list'>,

Is this related to this issue?

[stuartarchibald]

@tobiasraabe yes

[ugm2]

Does anyone know how this is going? I can't help but get a huge logging every time I execute my script. It's messing up my CI runners

I am using umap-learn==0.5.1 and numba==0.53.1.

Thanks in advance!

[ugm2]

As a workaround, I am setting the following before importing umap (which imports numba):

import logging
logging.getLogger('numba').setLevel(logging.WARNING)

[HPLegion]

I just stumbled over this issue and recalled there was a discussion about this on Gitter last year, so I thought I'd just cross link that here: https://gitter.im/numba/numba?at=5ed533094c9b0f060d39ada5

For what it's worth: Personally I am not convinced that custom loglevels are the proper way to screen users of numba from log messages. To me, using the abilities of the logging module as suggested by @ugm2 or the filter suggested in the Gitter chat feel more like the solution intended by the devs of the logging module. I think custom levels should only be used if there is a feeling that numba itself needs more fine grain control over its own log messages.

[adigitoleo]

Just adding my 2c. Agree with @HPLegion in general, custom log levels seem to be discouraged in the logging module docs. One thing I note is that numba is "special" compared to (some) other modules that use logging, in that its logger doesn't seem to appear in logging.Logger.manager.loggerDict. In a project which imports numba and some other stuff, I can use:

print(list(logging.root.manager.loggerDict.keys()))

to grab all (?) the other logger names:

['PIL.Image', 'PIL', 'PIL.PngImagePlugin', 'matplotlib.ticker', 'matplotlib']                                 

It would be convenient if numba could add itself in there, then we (users) can use our debug logging while blocking all of those.

It's all a bit of a mess, because logging doesn't provide a way to operate using a whhitelist rather than a blacklist. But that's unlikely to change anytime soon, so this could be a "workaround" to make it easier.