Release 0.16.0

Highlights

The Numba backend uses the threads of the machine (#172, #174, #176)

The compiled kernels of the Numba implementation run in parallel where that pays for itself. Two paths are there:

• The operations that can be reduced in chunks - the sums, the products, the counts, the extremes and the mean - split their data into chunks that are reduced separately and folded afterwards.
• The operations that cannot - the order statistics (the median, the quantile, the sort, the number of the distinct values) - cut their groups into ranges instead and let the threads work on the ranges one after another. Every group belongs to one range, so nothing is folded afterwards and the result is the one of the sequential kernel bit for bit.

The chunks and the ranges are more than the threads, so that a core that is busy with something else does not hold up the whole call - the threads that are done take the pieces that are left. The operations that give an output per value (the cumulatives) stay single-threaded.

Whether the parallel path is taken is decided by a cost model that is fitted to measurements of the machine and scaled to err on the side of the sequential path (numpy_groupies/aggregate_numba/mt.py). Nothing is measured at run time, and the number of threads is the one of the pool of Numba - NUMBA_NUM_THREADS at the start of the process or numba.set_num_threads().

Every function is a name of its own (#170)

The function is in the name instead of in the func argument:

npg.mean(group_idx, a)              # the best implementation available
npg.quantile(group_idx, a, q=0.9)   # the same as func="quantile", q=0.9

The same names are the attributes of the module of every implementation, so an implementation can be chosen and used the same way:

import numpy_groupies.aggregate_numpy as npg
npg.mean(group_idx, a)              # the numpy implementation

The kernels of the Numba backend are reachable from jitted code

The Numba module carries one kernel per function of the package, under the name of that function - the lean functions jitted code calls, and nothing else:

import numba as nb
import numpy_groupies.aggregate_numba as npg

@nb.njit
def zscore(group_idx, a):
    mean = npg.mean(group_idx, a)
    std = npg.std(group_idx, a)
    return (a - mean[group_idx]) / std[group_idx]

The same call from Python reaches the very same compiled driver - the same arguments and the same result, only no jitted caller around it. A kernel is a lean thing: it takes flat 1-D input with integer indices and runs none of the validation and the dtype resolution of aggregate().

One registry and one set of complaints

The names of the functions, their aliases, their result types, their default fill values and the arguments they have of their own live in one place (numpy_groupies/registry.py), and every path into a computation - the python-level aggregate(), the pure python one and the compiled kernels - checks its input and says the same words about what is wrong with it (numpy_groupies/validation.py).

A compact test suite (#177)

The tests are asked once each instead of in every combination of function, form and backend, the tests of one function go to one worker of the parallel run, and the data of the tests comes from the seeded generator of the module.

Breaking changes

• The cache keyword argument of aggregate() is gone. The compiled loop of a custom callable is kept in a bounded LRU cache of the backend, and the policy of the cache is no longer the business of the caller. The other backends never honoured the argument.
• The NumPy implementation raises NotImplementedError for out= instead of ignoring the argument. Every backend takes the argument now, so that the signatures of the backends are the same.
• The name of a kernel changed: numpy_groupies.aggregate_numba.kernels is gone, and the kernels are the attributes of the Numba module (aggregate_numba.mean instead of kernels.aggregate_mean).

Other changes

• The kernel regressions of the Numba backend are kept out of the compiled drivers (#171), so that the drivers stay lean and the checks stay in Python.
• The python half of a kernel call is shared between the variants of the kernel (#173), and the jitted callers of the Numba kernels are built once per kernel and cached.
• The kernels of the threading model go through the stable jit (#176), so that they are cached on disk like the other kernels.
• The pure Python implementation has an unpack of its own, so that the gather works without NumPy.
• The hooks of pre-commit are updated (ruff, uv-pre-commit, rumdl), and filelock and virtualenv are bumped.

Benchmarks and documentation

• The benchmark tables of the readme and of docs/benchmarks.md are refreshed, and the new tool numpy_groupies.benchmarks.mt measures the threading model: the scaling of the kernels with the number of the threads, the warm and the cold times, and the point where the parallel path pays for itself.
• docs/functions.md describes the functions by their name and the arguments they have of their own.