Repository navigation
Release 4.15.0
Changes from 4.14.2 to 4.15.0
Typed ordinary-expression evaluation by default
- Ordinary lazy expressions now default to typed native execution where admitted,
with bounded typed-graph/NumPy fallback otherwise. Intermediate computation
dtypes remain separate from output and reduction accumulator dtypes. - NumPy older than 2 uses the installed NumPy for automatic ordinary-expression
evaluation, retaining bounded Blosc2 storage/reduction scheduling. Explicit
Menudet/native-only APIs keep their own numerical contract. - Keep
BLOSC2_EXPRESSION_EVALUATOR=legacyas a temporary qualification escape
hatch, selected before constructing expressions. Explicit compatibility DSL
and Python UDF contracts are unchanged. Unsupported typed execution never
silently retries through the uniform evaluator after numerical execution starts. - Summary-index OR queries no longer prune with a single column's candidate-block
bitmap; only conjunctions may combine such masks. OR predicates use the plan's
combined candidate segments, preserving rows that match only the other branch. - On NumPy < 2, non-numeric expressions keep their existing string route because
NumPy 1 has no string ufuncs; the installed-NumPy policy is numerical. - Some non-flat N-D fallback layouts remain slower than legacy fused execution;
seedoc/development/expression_evaluator_migration.mdand the migration plan
for profiling evidence and known limitations. Platform qualification is ongoing.
Introducing Menudet 0.1 beta
Menudet — a little language for portable computation on arrays and tables.
- Establish Menudet 0.1 beta as the supported computation language going forward.
Use@blosc2.menudet(MenudetFunction) for decorated authors,MenudetKernel
for executable specializations,MenudetErrorfor errors, andvalidate_menudet
for source validation. During beta, API and artifact changes remain possible and
will be documented with migration guidance. - Keep kernel authoring (
@menudet) distinct from NumPy-style eager/tracing
convenience (@jit). Existing@jitreturn types,strict,out=,disable=
and pandas contracts are unchanged. Scoped compiler backend defaults now reach
Menudet artifact preparation without being embedded in serialized author source. - Materialize eligible input-free numeric 1-D Menudet generators through native
block workers when output blocks match their logical partitions. Contiguous
partial reads also avoid building per-lane index grids; other layouts retain
partition-preserving execution. - Retire the unqualified alpha
@blosc2.dsl_kernel: importing it still works,
but using it raises actionable migration guidance to@blosc2.menudet. Direct
DSLKernel(...)andDSLKernel.from_source(...)also raise retirement guidance.
Shared source normalization is internal, not a legacy authoring API.
See the migration guide.
Generic Python LazyUDFs and supported LazyExpr computations are unaffected. - Reject historical source-backed DSL/Python UDF recipes and legacy table-kernel
metadata withUnsupportedLegacyRecipeError, even withdeserialize="full".
Before upgrading, use a previous release to materialize ordinary arrays,
reauthor with Menudet, and compare results in an isolated candidate environment.
No source reconstruction, silent column dropping or automatic migration occurs. lazyudf()rejects MenudetFunction/MenudetKernel with guidance to call the
function directly or usekernel.lazy(...); it is not a second Menudet API.- Strengthen NumExpr-removal qualification: runtime/install behavior is independent
of NumExpr, including native interpretation without JIT. Remove the
blosc2.numexprcompatibility placeholder; accessing it now raises
AttributeError. Rename internal chunk evaluation and options to
evaluate_chunkandeval_kwargs; retain temporaryne_evaluateand public
ne_args=compatibility aliases.- Native compression/reduction prefilter failures now propagate instead of
replaying a partially executed operation through NumPy. - Serial bounded stored-output maps accept interpreter artifacts as well as JIT.
- CI requires NumExpr-free environments; comparison tooling treats it as optional.
- Native compression/reduction prefilter failures now propagate instead of
- Use native structural classification for automatic execution policy: interpret
single-return expressions and prefer JIT for multi-statement Menudet programs.
Explicit JIT settings and source compiler pragmas retain priority. Expose
MenudetKernel.execution_kind;evaluate()no longer needs a special default
override. This requires the paired native classification API; older runtimes
retain their existing default behavior. Remove the legacy
truediv,sanitizeanddisable_cachekeywords; supplying them now raises an
error. Keeporder,castingand theoptimizationcompatibility hint. - Enable complex64/complex128 miniexpr expressions and Menudet interpretation
on Windows (clang-cl), and remove the remaining WebAssembly complex expression
and reduction exclusions. Complex Menudet kernels remain interpreter-only.
Scalar string-expression reductions on WASM retain the existing chunked
fallback for all dtypes; portable Menudet block reductions remain supported. - C-order
arangewith real bounds and complex output, and complexlinspace,
now use Menudet interpretation, includinglazy=True. Complexlinspace
preserves the requested endpoint across slicing and rechunking. - Menudet source pragmas now select math libraries and floating-point modes:
me:math=auto|system|sleef|accelerate,me:ulp=u10|u35, and
me:fp=strict|contract|fast(withme:compiler=cc). Exported artifacts
preserve explicit policy through JSON round trips; omitted fields inherit
ME_SIMD_MATH_BACKEND/ME_DSL_FP_MODEat preparation. Unavailable explicit
backends fail before execution, andautonow follows system math instead of
preferring SLEEF. - Export normalized Menudet authoring as validated, standalone JSON artifacts
with typed signatures and exact captured constants. Native miniexpr executes
imports without reconstructing Python functions or falling back to Python. - Support checked integer arithmetic, operand-typed floating computation, control
flow, fixed-width strings, logical array coordinates and ordered block reductions. - Keep logical reduction groups stable through persistence, partial reads and
output rechunking. True block-scalar results expose the logical block-grid shape. - Validate kernel inputs to
CTable.add_computed_column()and
add_generated_column()as Menudet artifacts with implicit independent-row
semantics, including fixed-shape row reductions. Named bindings are supported;
internal source normalization uses the same semantics.
New cross-row computations must use array partitions or table aggregation,
rather than implicit execution blocks or append batches. - Remove the draft
add_portable_*column methods androw_domainargument.
Fixed-shape row reductions usecardinality="block_scalar". - New Menudet lazy results persist validated artifacts; unsupported saves fail
before writing the destination. MessagePack decoding is safe by default,
including nested containers. Historical source recipes remain unsupported under
both policies, not re-enabled by trusted deserialization permission. - Ordinary LazyExpr persistence retains supported numerical calls, reductions,
methods and indexing under default loading, with existing expression validation.
Full-only nested payloads remain gated. String/lazy expressions and non-kernel
row transformers retain their existing table registration behavior.
Menudet 0.1 is beta: the core API and semantics are expected to remain
relatively stable, but may change before a 1.0 compatibility commitment. The typed
interpreter defines behavior and supported kernels can use JIT; required JIT
rejects unsupported acceleration. Complex computation is supported through the
interpreter on all supported platforms. Callbacks and arbitrary memory access
remain outside the language contract.
See the Menudet reference
for array/table examples, persistence policy and current limitations.
Serializing containers that have a postfilter
- Behaviour change:
SChunk.to_cframe()andNDArray.to_cframe()now return
the chunks as they are stored for every storage, also when a postfilter is set
(SChunk.postfilter). Before, they returned the postfilter's output unless the
container was an in-memory contiguous frame (contiguous=Truewithout
urlpath). In-memory containers are sparse by default, so the frame usually
held the postfiltered values, and setting the postfilter again on the container
loaded from it applied it twice. Code that loads such a frame without setting
the postfilter again now reads the stored values. To serialize the postfiltered
values, serialize a copy that holds them:NDArray.copy()applies the
postfilter, and for anSChunk, a new one made from its decompressed data does.
Containers without a postfilter give the same frames as before. With a
prefilter or a postfilter,to_cframe()no longer decompresses and recompresses
every chunk, which makes it much faster. This comes with the bundled C-Blosc2
(Blosc/c-blosc2#858).
Bundled libraries and other improvements
- Bundled C-Blosc2 updated to 3.6.1.
- Bundled miniexpr updated to 0.3.1.
- Report the compiled miniexpr version in
blosc2.print_versions().