Permit non intp typed ints as args in np.* allocators. #3505

stuartarchibald · 2018-11-16T15:33:34Z

As title.

seibert · 2018-11-19T15:56:03Z

FYI: this is currently failing on Linux 32-bit because sizeof(intp) < sizeof(int64):

2018-11-16T17:20:46.8716360Z ======================================================================
2018-11-16T17:20:46.8716429Z ERROR: test_empty2d_typed_tuple_arg_overflow (numba.tests.test_dyn_array.TestDynArray)
2018-11-16T17:20:46.8717375Z ----------------------------------------------------------------------
2018-11-16T17:20:46.8717447Z Traceback (most recent call last):
2018-11-16T17:20:46.8717545Z   File "/home/vsts/work/1/s/numba/errors.py", line 601, in new_error_context
2018-11-16T17:20:46.8717595Z     yield
2018-11-16T17:20:46.8717649Z   File "/home/vsts/work/1/s/numba/lowering.py", line 257, in lower_block
2018-11-16T17:20:46.8717695Z     self.lower_inst(inst)
2018-11-16T17:20:46.8717784Z   File "/home/vsts/work/1/s/numba/lowering.py", line 306, in lower_inst
2018-11-16T17:20:46.8717829Z     val = self.lower_assign(ty, inst)
2018-11-16T17:20:46.8717879Z   File "/home/vsts/work/1/s/numba/lowering.py", line 452, in lower_assign
2018-11-16T17:20:46.8717967Z     return self.lower_expr(ty, value)
2018-11-16T17:20:46.8718015Z   File "/home/vsts/work/1/s/numba/lowering.py", line 875, in lower_expr
2018-11-16T17:20:46.8718061Z     res = self.lower_call(resty, expr)
2018-11-16T17:20:46.8718152Z   File "/home/vsts/work/1/s/numba/lowering.py", line 841, in lower_call
2018-11-16T17:20:46.8718198Z     res = impl(self.builder, argvals, self.loc)
2018-11-16T17:20:46.8718334Z   File "/home/vsts/work/1/s/numba/targets/base.py", line 1116, in __call__
2018-11-16T17:20:46.8718425Z     return self._imp(self._context, builder, self._sig, args)
2018-11-16T17:20:46.8718484Z   File "/home/vsts/work/1/s/numba/targets/arrayobj.py", line 3284, in numpy_empty_nd
2018-11-16T17:20:46.8718535Z     ary = _empty_nd_impl(context, builder, arrtype, shapes, shapety)
2018-11-16T17:20:46.8718627Z   File "/home/vsts/work/1/s/numba/targets/arrayobj.py", line 3176, in _empty_nd_impl
2018-11-16T17:20:46.8718675Z     raise ValueError(msg)
2018-11-16T17:20:46.8718723Z ValueError: Type int64 given in shape would lead to allocation potentially exceeding intp size
2018-11-16T17:20:46.8718799Z 
2018-11-16T17:20:46.8718847Z During handling of the above exception, another exception occurred:
2018-11-16T17:20:46.8718879Z 
2018-11-16T17:20:46.8718919Z Traceback (most recent call last):
2018-11-16T17:20:46.8719011Z   File "/home/vsts/work/1/s/numba/tests/test_dyn_array.py", line 222, in test_empty2d_typed_tuple_arg_overflow
2018-11-16T17:20:46.8719070Z     cfunc(m, n)
2018-11-16T17:20:46.8719120Z   File "/home/vsts/work/1/s/numba/dispatcher.py", line 367, in _compile_for_args
2018-11-16T17:20:46.8719266Z     raise e
2018-11-16T17:20:46.8719315Z   File "/home/vsts/work/1/s/numba/dispatcher.py", line 324, in _compile_for_args
2018-11-16T17:20:46.8719362Z     return self.compile(tuple(argtypes))
2018-11-16T17:20:46.8719459Z   File "/home/vsts/work/1/s/numba/compiler_lock.py", line 32, in _acquire_compile_lock
2018-11-16T17:20:46.8719507Z     return func(*args, **kwargs)
2018-11-16T17:20:46.8719555Z   File "/home/vsts/work/1/s/numba/dispatcher.py", line 655, in compile
2018-11-16T17:20:46.8719643Z     cres = self._compiler.compile(args, return_type)
2018-11-16T17:20:46.8719694Z   File "/home/vsts/work/1/s/numba/dispatcher.py", line 82, in compile
2018-11-16T17:20:46.8719742Z     pipeline_class=self.pipeline_class)
2018-11-16T17:20:46.8719830Z   File "/home/vsts/work/1/s/numba/compiler.py", line 905, in compile_extra
2018-11-16T17:20:46.8719886Z     return pipeline.compile_extra(func)
2018-11-16T17:20:46.8719934Z   File "/home/vsts/work/1/s/numba/compiler.py", line 367, in compile_extra
2018-11-16T17:20:46.8720027Z     return self._compile_bytecode()
2018-11-16T17:20:46.8720076Z   File "/home/vsts/work/1/s/numba/compiler.py", line 836, in _compile_bytecode
2018-11-16T17:20:46.8720126Z     return self._compile_core()
2018-11-16T17:20:46.8720173Z   File "/home/vsts/work/1/s/numba/compiler.py", line 823, in _compile_core
2018-11-16T17:20:46.8720263Z     res = pm.run(self.status)
2018-11-16T17:20:46.8720311Z   File "/home/vsts/work/1/s/numba/compiler_lock.py", line 32, in _acquire_compile_lock
2018-11-16T17:20:46.8720358Z     return func(*args, **kwargs)
2018-11-16T17:20:46.8720444Z   File "/home/vsts/work/1/s/numba/compiler.py", line 253, in run
2018-11-16T17:20:46.8720489Z     raise patched_exception
2018-11-16T17:20:46.8720536Z   File "/home/vsts/work/1/s/numba/compiler.py", line 244, in run
2018-11-16T17:20:46.8720623Z     stage()
2018-11-16T17:20:46.8720670Z   File "/home/vsts/work/1/s/numba/compiler.py", line 697, in stage_nopython_backend
2018-11-16T17:20:46.8720718Z     self._backend(lowerfn, objectmode=False)
2018-11-16T17:20:46.8720814Z   File "/home/vsts/work/1/s/numba/compiler.py", line 647, in _backend
2018-11-16T17:20:46.8720858Z     lowered = lowerfn()
2018-11-16T17:20:46.8720906Z   File "/home/vsts/work/1/s/numba/compiler.py", line 634, in backend_nopython_mode
2018-11-16T17:20:46.8720987Z     self.flags)
2018-11-16T17:20:46.8721035Z   File "/home/vsts/work/1/s/numba/compiler.py", line 1026, in native_lowering_stage
2018-11-16T17:20:46.8721081Z     lower.lower()
2018-11-16T17:20:46.8721128Z   File "/home/vsts/work/1/s/numba/lowering.py", line 176, in lower
2018-11-16T17:20:46.8721215Z     self.lower_normal_function(self.fndesc)
2018-11-16T17:20:46.8721264Z   File "/home/vsts/work/1/s/numba/lowering.py", line 217, in lower_normal_function
2018-11-16T17:20:46.8721311Z     entry_block_tail = self.lower_function_body()
2018-11-16T17:20:46.8721462Z   File "/home/vsts/work/1/s/numba/lowering.py", line 242, in lower_function_body
2018-11-16T17:20:46.8721509Z     self.lower_block(block)
2018-11-16T17:20:46.8721564Z   File "/home/vsts/work/1/s/numba/lowering.py", line 257, in lower_block
2018-11-16T17:20:46.8721649Z     self.lower_inst(inst)
2018-11-16T17:20:46.8721699Z   File "/home/vsts/miniconda3/envs/travisci/lib/python3.7/contextlib.py", line 130, in __exit__
2018-11-16T17:20:46.8721749Z     self.gen.throw(type, value, traceback)
2018-11-16T17:20:46.8721838Z   File "/home/vsts/work/1/s/numba/errors.py", line 609, in new_error_context
2018-11-16T17:20:46.8721886Z     six.reraise(type(newerr), newerr, tb)
2018-11-16T17:20:46.8721932Z   File "/home/vsts/work/1/s/numba/six.py", line 658, in reraise
2018-11-16T17:20:46.8722020Z     raise value.with_traceback(tb)
2018-11-16T17:20:46.8722068Z   File "/home/vsts/work/1/s/numba/errors.py", line 601, in new_error_context
2018-11-16T17:20:46.8722113Z     yield
2018-11-16T17:20:46.8722205Z   File "/home/vsts/work/1/s/numba/lowering.py", line 257, in lower_block
2018-11-16T17:20:46.8722251Z     self.lower_inst(inst)
2018-11-16T17:20:46.8722297Z   File "/home/vsts/work/1/s/numba/lowering.py", line 306, in lower_inst
2018-11-16T17:20:46.8722437Z     val = self.lower_assign(ty, inst)
2018-11-16T17:20:46.8722487Z   File "/home/vsts/work/1/s/numba/lowering.py", line 452, in lower_assign
2018-11-16T17:20:46.8722533Z     return self.lower_expr(ty, value)
2018-11-16T17:20:46.8722579Z   File "/home/vsts/work/1/s/numba/lowering.py", line 875, in lower_expr
2018-11-16T17:20:46.8722664Z     res = self.lower_call(resty, expr)
2018-11-16T17:20:46.8722712Z   File "/home/vsts/work/1/s/numba/lowering.py", line 841, in lower_call
2018-11-16T17:20:46.8722758Z     res = impl(self.builder, argvals, self.loc)
2018-11-16T17:20:46.8722848Z   File "/home/vsts/work/1/s/numba/targets/base.py", line 1116, in __call__
2018-11-16T17:20:46.8722897Z     return self._imp(self._context, builder, self._sig, args)
2018-11-16T17:20:46.8722955Z   File "/home/vsts/work/1/s/numba/targets/arrayobj.py", line 3284, in numpy_empty_nd
2018-11-16T17:20:46.8723050Z     ary = _empty_nd_impl(context, builder, arrtype, shapes, shapety)
2018-11-16T17:20:46.8723107Z   File "/home/vsts/work/1/s/numba/targets/arrayobj.py", line 3176, in _empty_nd_impl
2018-11-16T17:20:46.8723330Z     raise ValueError(msg)
2018-11-16T17:20:46.8723626Z numba.errors.LoweringError: Failed in nopython mode pipeline (step: nopython mode backend)
2018-11-16T17:20:46.8723682Z Type int64 given in shape would lead to allocation potentially exceeding intp size
2018-11-16T17:20:46.8723715Z 
2018-11-16T17:20:46.8723800Z File "numba/tests/test_dyn_array.py", line 205:
2018-11-16T17:20:46.8723846Z         def pyfunc(m, n):
2018-11-16T17:20:46.8723895Z             arr = np.empty((m, n), np.int32)
2018-11-16T17:20:46.8723979Z             ^
2018-11-16T17:20:46.8724007Z 
2018-11-16T17:20:46.8724075Z [1] During: lowering "$0.8 = call $0.2($0.5, $0.7, func=$0.2, args=[Var($0.5, /home/vsts/work/1/s/numba/tests/test_dyn_array.py (205)), Var($0.7, /home/vsts/work/1/s/numba/tests/test_dyn_array.py (205))], kws=(), vararg=None)" at /home/vsts/work/1/s/numba/tests/test_dyn_array.py (205)
2018-11-16T17:20:46.8724142Z 
2018-11-16T17:20:46.8724513Z ----------------------------------------------------------------------

codecov-io · 2018-11-19T17:55:14Z

Codecov Report

❗ No coverage uploaded for pull request base (master@57d4ba0). Click here to learn what that means.
The diff coverage is 75.58%.

@@            Coverage Diff            @@
##             master    #3505   +/-   ##
=========================================
  Coverage          ?   80.68%           
=========================================
  Files             ?      392           
  Lines             ?    80205           
  Branches          ?     9127           
=========================================
  Hits              ?    64717           
  Misses            ?    14071           
  Partials          ?     1417

stuartarchibald · 2018-11-20T17:34:11Z

Following on from discussion in Numba meeting. Value base check needs adding near NRT allocation call on 32bit platforms to check that it won't overflow e.g. ssize_t.

As title. Fixes numba#2758

As title.

sklam · 2019-01-14T17:17:19Z

Is this ready for review?

stuartarchibald · 2019-01-16T09:19:20Z

@sklam not yet, I need to fix up and merge the run time allocation checks and work out what should be permitted on 32bit platforms.

stuartarchibald · 2019-12-10T18:00:35Z

Closing #4729 is better.

stuartarchibald added 2 - In Progress Pending BuildFarm For PRs that have been reviewed but pending a push through our buildfarm labels Nov 16, 2018

seibert added this to the Numba 0.42 RC milestone Nov 19, 2018

stuartarchibald added 2 commits November 29, 2018 17:36

Permit non intp typed ints as args in np.* allocators.

8635428

As title. Fixes numba#2758

Fix error type in exception checking.

be40500

As title.

stuartarchibald force-pushed the fix/2758 branch from f6a0a28 to be40500 Compare November 29, 2018 17:36

seibert mentioned this pull request Dec 14, 2018

Numba 0.42RC checklist #3600

Closed

29 tasks

seibert requested a review from sklam December 17, 2018 16:00

stuartarchibald added this to In Progress in Active Dec 20, 2018

stuartarchibald modified the milestones: Numba 0.42 RC, Numba 0.43 RC Jan 3, 2019

stuartarchibald modified the milestones: Numba 0.43 RC, Numba 0.44 RC, Numba 0.45 RC Mar 5, 2019

stuartarchibald force-pushed the master branch from e8ac495 to 14d8e85 Compare April 29, 2019 22:29

seibert modified the milestones: Numba 0.45 RC, Numba 0.46 RC Jul 2, 2019

stuartarchibald mentioned this pull request Sep 16, 2019

LoweringError in scanpy's calculate_qc_metrics function #4529

Closed

stuartarchibald modified the milestones: Numba 0.46 RC, Numba 0.47 RC Sep 24, 2019

This was referenced Oct 15, 2019

Internal Type Detection Failure? #4704

Closed

Allow array construction with mixed type shape tuples #4729

Merged

stuartarchibald closed this Dec 10, 2019

stuartarchibald removed this from the Numba 0.47 RC milestone Dec 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Permit non intp typed ints as args in np.* allocators. #3505

Permit non intp typed ints as args in np.* allocators. #3505

stuartarchibald commented Nov 16, 2018

seibert commented Nov 19, 2018

codecov-io commented Nov 19, 2018 •

edited

stuartarchibald commented Nov 20, 2018

sklam commented Jan 14, 2019

stuartarchibald commented Jan 16, 2019

stuartarchibald commented Dec 10, 2019

Permit non intp typed ints as args in np.* allocators. #3505

Permit non intp typed ints as args in np.* allocators. #3505

Conversation

stuartarchibald commented Nov 16, 2018

seibert commented Nov 19, 2018

codecov-io commented Nov 19, 2018 • edited

Codecov Report

stuartarchibald commented Nov 20, 2018

sklam commented Jan 14, 2019

stuartarchibald commented Jan 16, 2019

stuartarchibald commented Dec 10, 2019

codecov-io commented Nov 19, 2018 •

edited