Error using nb.prange()

[pythonweb2]

Here is a reproducible example of a problem I encountered today. The traceback suggested that I post an issue, so I came here.

import numba as nb
import numpy as np
from time import time
import matplotlib.pyplot as plt


def f(d):
    a = np.int(10*d)
    N = np.int(10000/d)
    for _ in range(N):
        X = np.dot(np.random.randn(a,10), np.random.randn(10,10))
    return X


@nb.jit(parallel=True, nopython=True)
def f_numba(d):
    a = np.int(10*d)
    N = np.int(10000/d)
    for _ in nb.prange(N):
        X = np.dot(np.random.randn(a,10), np.random.randn(10,10))
    return X


# Dimensions
ds = [1,2,3,4,5,6,8,10,20,35,40,45,50,60,62,64,66,68,70,80,90,100]

# Serial processing
serial = []
for d in ds:
    t1 = time()
    for i in range(20):
        f(d)
    serial.append(time()-t1)

# Parallel processing
parallel = []
for d in ds:
    t1 = time()
    for i in range(20):
        f_numba(d)
    parallel.append(time() - t1)

# Plot
plt.title('Matrix multiplication time with 10000/d repetitions')
plt.plot(ds, serial, label='serial')
plt.plot(ds, parallel, label='parallel')
plt.xlabel('d (dimension)')
plt.ylabel('Total time (sec)')
plt.legend()
plt.show()

Traceback is:

numba.errors.LoweringError: Failed in nopython mode pipeline (step: nopython mode backend)
Buffer dtype cannot be buffer, have dtype: array(float64, 2d, C)

File "scratch.py", line 19:
def f_numba(d):
    <source elided>
    N = np.int(10000/d)
    for _ in nb.prange(N):
    ^

[1] During: lowering "id=2[LoopNest(index_variable = parfor_index.22, range = (0, $N.24, 1))]{42: <ir.Block at C:/Users/wadear/.PyCharm2019.1/config/scratches/scratch.py (19)>}Var(parfor_index.22, C:/Users/wadear/.PyCharm2019.1/config/scratches/scratch.py (19))" at C:/Users/wadear/.PyCharm2019.1/config/scratches/scratch.py (19)
-------------------------------------------------------------------------------
This should not have happened, a problem has occurred in Numba's internals.

Please report the error message and traceback, along with a minimal reproducer
at: https://github.com/numba/numba/issues/new

If more help is needed please feel free to speak to the Numba core developers
directly at: https://gitter.im/numba/numba

Thanks in advance for your help in improving Numba!

My current dependencies and versions are:

Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 03:37:03) [MSC v.1900 64 bit (AMD64)] on win32
cftime==1.0.3.4
cycler==0.10.0
et-xmlfile==1.0.1
hydrostats==0.67
jdcal==1.4
kiwisolver==1.0.1
llvmlite==0.28.0
matplotlib==2.2.2
mpmath==1.0.0
netCDF4==1.4.2
numba==0.43.1
numpy==1.16.2
openpyxl==2.5.4
pandas==0.23.3
pyparsing==2.2.0
python-dateutil==2.7.3
pytz==2018.5
scipy==1.1.0
six==1.11.0
sympy==1.2
xarray==0.11.3
xlrd==1.1.0

[stuartarchibald]

Thanks for the report, I can reproduce. Works fine without parallel=True so it's a ParallelAccelerator issue. Suspect it's the np.random.randn(a, 10) that's the issue. Ping @DrTodd13.

[DrTodd13]

Important note. The parallel version is undefined since you are writing to the same variable in every loop iteration.

[DrTodd13]

This is actually unrelated to randn. Here's a smaller reproducer:

import numba as nb
import numpy as np

@nb.jit(parallel=True, nopython=True)
def f_numba(c):
    for _ in nb.prange(2):
        X = c
    return X

c = np.random.randn(10, 10)
f_numba(c)

Given that this assignment to X is a race and has undefined behavior, this is going to be low priority for me since it is essentially trading a compiler error for undefined runtime behavior.