Numba warnings

[mbr085]

First I want to thank you for sharing this great work. It is very useful for me.
The first time I execute UMAP fit and transform I get some numba warnings. They do not seem to be critical, but they are somewhat disturbing.

numba.__version__
'0.44.0'

sys.version
'3.7.3 (default, Mar 27 2019, 22:11:17) \n[GCC 7.3.0]'

reducer = umap.UMAP(random_state=42)
reducer.fit(X)
Y = reducer.transform(X)

/home/mbr085/anaconda3/envs/divisivegater/lib/python3.7/site-packages/umap/umap_.py:349: NumbaWarning: 
Compilation is falling back to object mode WITH looplifting enabled because Function "fuzzy_simplicial_set" failed type inference due to: Untyped global name 'nearest_neighbors': cannot determine Numba type of <class 'function'>

File "../../../../../anaconda3/envs/divisivegater/lib/python3.7/site-packages/umap/umap_.py", line 467:
def fuzzy_simplicial_set(
    <source elided>
    if knn_indices is None or knn_dists is None:
        knn_indices, knn_dists, _ = nearest_neighbors(
        ^

  @numba.jit()
/home/mbr085/anaconda3/envs/divisivegater/lib/python3.7/site-packages/numba/compiler.py:725: NumbaWarning: Function "fuzzy_simplicial_set" was compiled in object mode without forceobj=True.

File "../../../../../anaconda3/envs/divisivegater/lib/python3.7/site-packages/umap/umap_.py", line 350:
@numba.jit()
def fuzzy_simplicial_set(
^

  self.func_ir.loc))
/home/mbr085/anaconda3/envs/divisivegater/lib/python3.7/site-packages/numba/compiler.py:734: NumbaDeprecationWarning: 
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "../../../../../anaconda3/envs/divisivegater/lib/python3.7/site-packages/umap/umap_.py", line 350:
@numba.jit()
def fuzzy_simplicial_set(
^

  warnings.warn(errors.NumbaDeprecationWarning(msg, self.func_ir.loc))

[lmcinnes]

These are, indeed, non-critical -- numba has just gotten a little more verbose about it's warnings, and umap continues to be somewhat optimistic about how well numba can compile things. In this case numba is just letting you know that umap was a little over-optimistic and not as much could be as completely compiled as one might have hoped. I'll try to fix this up in an upcoming release. Thanks for the heads up, it is appreciated.

[00krishna]

I was getting the same warnings. I added the following code to my notebooks and it solved it.

import warnings
warnings.filterwarnings('ignore')

This was based upon the StackExchange Post
https://stackoverflow.com/questions/9031783/hide-all-warnings-in-ipython

[Kimi71Jedidiah]

I came across the same problem!
S.umap(max_pc=max_pc,n_neighbors=k,min_dist=0.05)

[lmcinnes]

This is actually fine -- the latest version of numba now issues warning for routines it couldn't completely compile. There will be a release of umap-learn soon that will remove the ineffective jit instructions and thus remove the warnings.

[sjfleming]

Could you not do
@numba.jit(forceobj=True) ?

[lmcinnes]

That might we work yes. Right now it is mostly just a matter of having the time to sit down and go through all the options and the code and make changes. I was very busy last month, and am on vacation right now and trying not to look at code. Hopefully I can make some time in a couple of weeks.

[sleighsoft]

Enjoy your vacation :)

To help you resolve the warnings quickly I compiled a list of all numba errors.

A list of warnings I currently see:

The keyword argument 'parallel=True' was specified but no transformation for 
parallel execution was possible.

To find out why, try turning on parallel diagnostics, see
http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for
help.File "../umap/nndescent.py", line 47:   
    @numba.njit(parallel=True)
    def nn_descent(

Compilation is falling back to object mode WITH looplifting enabled because Function 
"fuzzy_simplicial_set" failed type inference due to: Untyped global name 
'nearest_neighbors': cannot determine Numba type o$ <class 'function'>
File "../umap/umap_.py", line 467:
    def fuzzy_simplicial_set(
    <source elided>
    if knn_indices is None or knn_dists is None:
        knn_indices, knn_dists, _ = nearest_neighbors(
        ^

NumbaWarning: Function "fuzzy_simplicial_set" was compiled in object mode without 
forceobj=True. 
File "../umap/umap_.py", line 350:
@numba.jit()
def fuzzy_simplicial_set(
^

The keyword argument 'parallel=True' was specified but no transformation for
parallel execution was possible. To find out why, try turning on parallel diagnostics,
see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.
File "../umap/utils.py", line 409:
@numba.njit(parallel=True)
def build_candidates(current_graph, n_vertices, n_neighbors, max_candidates, rng_state):
^

umap/rp_tree.py:450: NumbaWarning:  Compilation is falling back to object mode 
WITH looplifting enabled because Function "make_euclidean_tree" failed type 
inference due to: Cannot unify RandomProjectionTreeNode(array(int64, 1d, C), 
bool, none, none, none, none) and RandomProjectionTreeNode(none, bool, 
array(float32, 1d, C), float64, RandomProjectionTreeNode(array(int64, 1d, C), bool, 
none, none, none, none), RandomProjectionTreeNode(array(int64, 1d, 
C), bool, none, none, none, none)) for '$14.16', defined at ../umap/rp_tree.py (457)  
                                                                                               
File "../umap/rp_tree.py", line 457:            
def make_euclidean_tree(data, indices, rng_state, leaf_size=30):  
<source elided>                              
        left_node = make_euclidean_tree(data, left_indices, rng_state, leaf_size)
        ^
                                                                                              
[1] During: resolving callee type: recursive(type(CPUDispatcher(
<function make_euclidean_tree at 0x7fafae56a378>)))
[2] During: typing of call at ../umap/rp_tree.py (457)   
                                                                                                                
File "../umap/rp_tree.py", line 457:       
def make_euclidean_tree(data, indices, rng_state, leaf_size=30):
    <source elided>
    left_node = make_euclidean_tree(data, left_indices, rng_state, leaf_size)
    ^ 
@numba.jit()
NumbaWarning: Function "make_euclidean_tree" was compiled in object mode without 
forceobj=True.       

[sleighsoft]

Changed in: afb819e

[sleighsoft]

Should be resolved in 0.3.10 release 0.4

EDIT:
This warning is caused by umap-learn package and is just that. A warning that numba, a package used to speed up code execution by JIT compiling code, could not optimize the code that was annotated to be JIT compiled.

This is 100% my bad. I thought @lmcinnes added this to 0.3.10 but I think it only made it into the upcoming release of 0.4.

The PR for the changes is here #287

[asfix]

The keyword argument 'parallel=True' was specified but no transformation for
parallel execution was possible.

To find out why, try turning on parallel diagnostics, see
http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for
help.File "../umap/nndescent.py", line 47:
@numba.njit(parallel=True)
def nn_descent(

I am having this problem too..

/usr/local/lib/python3.6/dist-packages/umap/spectral.py:229: UserWarning: Embedding a total of 178 separate connected components using meta-embedding (experimental)
n_components
/usr/local/lib/python3.6/dist-packages/sklearn/metrics/pairwise.py:1575: DataConversionWarning: Data was converted to boolean for metric jaccard
warnings.warn(msg, DataConversionWarning)

When I press a key, I started to get another message. Would not I run my script and obtain reduced dimensions?

[lmcinnes]

Sorry @asfix ; these are just warnings and don't represent a problem beyond the fact that various things have been done in the processing that a user may wish to be aware of. It should all run without problem, but there may be some warning messages flashing by.

[suluxan]

Hey,
We are getting this same error but the warnings are causing our pipeline to fail rather than just skip over them. Is there any way to ignore warnings? Or has this been addressed?
i.e.

server_1    | [0] stderr: /usr/local/lib/python2.7/dist-packages/numba/compiler.py:602: NumbaPerformanceWarning:
server_1    | [0] The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
server_1    | [0]
server_1    | [0] To find out why, try turning on parallel diagnostics, see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.
server_1    | [0]
server_1    | [0] File "../usr/local/lib/python2.7/dist-packages/umap/nndescent.py", line 47:
server_1    | [0]     @numba.njit(parallel=True)
server_1    | [0]     def nn_descent(
server_1    | [0]     ^
server_1    | [0]
server_1    | [0]   self.func_ir.loc))
server_1    | [0]
server_1    | [0] stderr: UMAP(a=None, angular_rp_forest=False, b=None, init='spectral',
server_1    | [0]    learning_rate=1.0, local_connectivity=1, metric='correlation',
server_1    | [0]    metric_kwds=None, min_dist=0.3, n_components=2, n_epochs=None,
server_1    | [0]    n_neighbors=30, negative_sample_rate=5, random_state=None,
server_1    | [0]    repulsion_strength=1.0, set_op_mix_ratio=1.0, spread=1.0,
server_1    | [0]    target_metric='categorical', target_metric_kwds=None,
server_1    | [0]    target_n_neighbors=-1, target_weight=0.5, transform_queue_size=4.0,
server_1    | [0]    transform_seed=42, verbose=True)

[sleighsoft]

It has been addressed in 0.4dev. Could you try with that branch, please. And report back any issues.

[suluxan]

Is the 0.4dev only available for python 3.5?

[sleighsoft]

If I remember correctly there was a PR that included the @ operator which is only available in Python 3.5.
I cannot find the PR but I assume that when you don't use that code path you are fine to use a version lower than 3.5

[lmcinnes]

I believe future releases will be targeting 3.5 and upwards. So things may work for now, but i a version or so you'll likely need a newer python if you haven't gotten onto 3.5 or newer.

[suluxan]

Yeah there was build problems on python 2.7 due to the @ but we've updated to 3.7.3 and everything is smooth now! Thank you very much

[moldach]

This warning is caused by umap-learn package and is just that. A warning that numba, a package used to speed up code execution by JIT compiling code, could not optimize the code that was annotated to be JIT compiled.

I'm seeking clarity on: The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

I'm trying to do some benchmarking and would like to know if this means that UMAP only ran on one core?

Just found issue #317 from November 2019 so I'm guessing parallelization is still not available out-of-the-box?

[sleighsoft]

This can just mean that a single function within UMAP could not be optimized to be run in parallel by Numba. It does not mean that everything else, that can be, wasn't executed in parallel. This should not happen with the current dev branch. If so, please report.

[naity]

I use the following to ignore the numba warning without affecting other warnings for now.

# silence NumbaPerformanceWarning
import warnings
from numba.errors import NumbaPerformanceWarning

warnings.filterwarnings("ignore", category=NumbaPerformanceWarning)

[scvannost]

This solution works for me!

[vasiliskatr]

Thank you for numba!

This warning: "The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible."

is always occurring only on the 1st execution of the jited function. Every subsequent execution runs smoothly and and several times faster. Is there a way to avoid this warning entirely?

Many computationally-heavy functions are called only once and thus it would be a great improvement if it could be avoided.