fix: accept numpy integers as chunk sizes - #288
Open
d-v-b wants to merge 4 commits into
Open
Conversation
* fix: allow `require_array` to accept a `ZDType` AsyncGroup.require_array normalised its dtype with np.dtype(), which cannot consume a ZDType, so requiring an existing array with one raised a TypeError. Every sibling creation method already accepts ZDTypeLike. Widen the annotation and normalise via parse_data_type().to_native_dtype(). parse_data_type(None) resolves to float64 just as np.dtype(None) did, so the default is unchanged. This leaves numpy.typing unused, so drop it. * chore: rename changelog fragment to the PR number * fix: keep the float64 default explicit for mypy parse_data_type does not accept None, so pass "float64" directly, which is what np.dtype(None) resolved to before. * test: parametrize require_array dtype cases over (input, expected) pairs Covers the `dtype=None` path, which resolves to float64 and was previously untested, and asserts on the resulting ZDType rather than the native dtype. --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
`normalize_chunks_nd` dispatches the scalar convenience form on `numbers.Integral`, but `normalize_chunks_1d` narrowed on `int`. Numpy integer scalars satisfy the former and not the latter, so a per-dimension numpy integer passed the outer dispatch and then fell into the branch meant for explicit per-dimension chunk sequences, where `list(chunks)` raised `TypeError: 'numpy.int64' object is not iterable`. Numpy integers arise naturally whenever a chunk shape is computed rather than written as a literal, since numpy reductions and elementwise ops yield numpy scalars. Narrow on `numbers.Integral` and coerce with `int()`, matching the caller and the sequence branch, which already accepted `Integral` elements. Move the `-1` sentinel check inside that branch. It previously ran on the raw input, so a numpy array chunk specification made `chunks == -1` return an array and raise an ambiguous-truth-value error; rectilinear specs given as numpy arrays now work. A chunk specification that is neither an integer nor iterable now names the offending value and its type instead of surfacing an opaque "object is not iterable" from `list(chunks)`. Fixes zarr-developers#4255 Assisted-by: ClaudeCode:claude-opus-5
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Fixes zarr-developers#4255.
Problem
Since 3.3.0, a per-dimension chunk size given as a numpy integer raises a
TypeError:Two functions in
src/zarr/core/chunk_grids.pydisagreed about what counts as an integer:normalize_chunks_nddispatches the scalar convenience form onisinstance(chunks, numbers.Integral)normalize_chunks_1dnarrowed onisinstance(chunks, int)np.int64isnumbers.Integralbut is not anintsubclass, so a numpy integer passed the outer dispatch, failed the inner narrowing, and fell into theelsebranch meant for explicit per-dimension chunk sequences — wherelist(chunks)was called on a scalar.This matters because numpy integers arise naturally whenever a chunk shape is computed rather than written as a literal; any numpy reduction or elementwise op yields numpy scalars. The reporter hit it through ultrack, which sizes chunks from the array shape and is currently pinned to
zarr<3.3.Fix
numbers.Integraland coerce withint(), matching the caller. This is consistent with the rest of the module:guess_chunksalready ends withtuple(int(x) for x in chunks), and the sequence branch already acceptsIntegralelements and coerces them.-1sentinel check inside that branch. It previously ran on the raw input, so a numpy array chunk specification madechunks == -1evaluate to an array and raise an ambiguous-truth-value error. Rectilinear specs given as numpy arrays now work.'float' object is not iterablefromlist(chunks). The issue called out that the old error named neither chunks nor the bad value.Verification
All of these previously raised and now work:
Regression cases added to
test_normalize_chunks(scalar, per-dimension, mixed python/numpy,-1sentinel, numpy arrays, rectilinear numpy arrays) and error cases fornp.int64(0)and the non-iterable scalar.One existing assertion in
tests/test_api.pyasserted the old opaque'float' object is not iterablemessage and was updated to the new one.Full test suite: 6357 passed, 373 skipped, 4 xfailed.
mypyandruffclean.