Minutes_2022_11_29

Numba Meeting: 2022-11-29

Attendees: Siu Kwan Lam, Andre Masella, Graham Markall, Jim Pivarski, LI Da, stuart, Tobias Sargeant, Todd A. Anderson, Guilherme, Brandon Willard FPOC (last week): Stuart FPOC (incoming): Guilherme

NOTE: All communication is subject to the Numba Code of Conduct.

Please refer to this calendar for the next meeting date.

0. Discussion

• Updates on:

  • Python 3.11
    • #8621, down to a handful of issues now.
  • NumPy 1.24
    • #8620 is technically ok for supporting 1.24, need to do some more work for the resolve_dtypes functionality on ufuncs. This also needs a buildfarm run.
  • LLVM 14
    • SVML is broken in a way beyond where we can easily fix.
  • Python 3.7 removal
    • Waiting on re-review.

• LLVM Passmanager replacement - legacy (currently in llvmlite) vs new (not yet in llvmlite, but supported in 11 onwards) - Tobias Sargeant (@folded on Github)

  • There's two APIs (legacy and new). "new" was introduced in LLVM 11 so we could move if we wanted to.
  • LLVM devs wants to remove the legacy pass.
  • One option is to support both APIs at the same time.
  • RE opaque pointers, can deal with this at some other point.
  • Numba itself needs a relatively high level API, but everything is exposed.
  • Question could the custom pass for refcount support move to the new API? A. it's Numba internal so as long as it's move is synchronized with Numba moving it'll be fine.
  • Tobias to report back on difficulty and usage etc.
  • No fundamental objections to the change.
  • Need to move to LLVM 14 first.

• Can we aim towards swapping compile_isolated calls for jit / njit and eventually removing compile_isolated and the NestedContext on which it relies?

  • Example of removal in PR 8625, refactoring test_ufuncs: https://github.com/numba/numba/pull/8625
  • (Graham Markall)

• PR #8120 milestone / reviewer (Graham Markall)

• Support dynamic exceptions without CPython API (Guilherme)

  • Explanation of prior attempts which used the CPython API at runtime.
  • Explanations of various issues with moving to CPython API free codegen.
    • labelling all exceptions needs a way to handle collisions
    • quite a lot of "book keeping" to deal with call conv "flattening" aggregate values etc.

• Using cython/mypyc to compile parts of Numba into C code (Guilherme)

  • cython3 which is alpha use type annotations to compile to C.
  • mypyc generates python C ext from type annotated python file, seemed easier to use as you can use standard python types for annotation. Claims 2-10x faster than Python.
    • Questions...
    • earlier investigation suggested it was quite easy to break (same goes for cython!)
    • need them to be more stable first?
    • when is cython 3 likely to ship?
    • mypyc carries its own runtime which may present additional challenges

• Support arrow-format data and related computation in Numba (LI Da)

  • fletcher (https://github.com/xhochy/fletcher) has verified the potential, but seems deprecated (I don't know why)
    • using arrow to store data (e.g., an array of variable strings), and using Numba to implement the computation (define numba type, register related datamodel, and overload related APIs)
  • post a discussion on https://numba.discourse.group/t/feature-request-about-supporting-arrow-in-numba/1668/6
  • Does Numba want to accept related PRs if provided?
  • Any idea/question/concern?
  • Jim: Awkward Array sits "in between", it is mostly zero-copy from Arrow and works in Numba.
  • There's also AwkwardPandas, https://github.com/intake/awkward-pandas
  • Here's the scope of Awkward's implementation of Numba iterators: https://github.com/scikit-hep/awkward/tree/main/src/awkward/_connect/numba It's mostly arrayview.py and layout.py. And also https://github.com/scikit-hep/awkward/blob/main/src/awkward/_lookup.py, which was separated from the Numba-specific code because we share it with C++ JIT. Total: ~2300 LoC.
  • The above would give RO Arrow access via iterators. Vectorized ops aren't implemented, but could be if you could map e.g. sequence-of-numbers to a np.ndarray (automatically!). Everything else that's e.g. scalar will end up as standard Numba types which will compile as usual.

New "Ready for Review" PRs

• #8120 - Support nesting of nested array types

1. New Issues

• #8616 - Meta Issue: Support for Advanced/Fancy Indexing in Numba
• #8617 - Investigate/discuss setuptools build and runtime dependency versions.
• #8618 - Enhance ufunc loop order specialisation with ndim information
• #8622 - getitem access with str value to a struct array causes impl "no match"

Closed Issues

2. New PRs

• #8619 - Added flat iteration logic for random distributions
• #8620 - [WIP] NumPy 1.24
• #8621 - py3.11 closure updates
• #8623 - Adds support for np.nan_to_num
• #8624 - DOC: Add guvectorize scalar return example
• #8625 - Refactor test_ufuncs

Closed PRs

3. Next Release: Version 0.57.0/0.40.0, RC Jan 2023