feat(extension_types): add numpy.ndarray as a native value type (ITL-460)#201
Conversation
…numpy.ndarray native value type Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements LogicalNumpyArray, a new logical type that maps numpy.ndarray to Arrow large_binary using numpy's .npy format as the storage envelope. Includes unit tests covering protocol conformance, serialization guards, and round-trip correctness for 1-D/2-D/N-D/structured/Fortran arrays.
…gicalNumpyArray (ITL-460) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…efault context (ITL-460) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…(ITL-460) Verifies that numpy.ndarray values (both simple 1-D float64 and structured record arrays) round-trip correctly through parquet and Delta Lake storage with the expected Arrow extension name "numpy.ndarray". Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…L-460) Add NumpyArrayHandler to register_builtin_python_type_handlers so the programmatic API is consistent with the v0.1.json-configured path. Also update module docstring and v0.1.json changelog. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Pull request overview
Adds first-class support for numpy.ndarray as an OrcaPod native value type by introducing an Arrow extension type backed by NumPy’s .npy format, plus a semantic-hash handler to keep hashing consistent with storage.
Changes:
- Introduce
LogicalNumpyArrayto serialize/deserializenumpy.ndarrayvia.npybytes stored in Arrowlarge_binary. - Add
NumpyArrayHandlerand register it (code + default context) to semantic-hash arrays using the same.npyrepresentation. - Add comprehensive unit + integration tests and make NumPy an explicit dependency.
Reviewed changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Locks NumPy as a dependency for the environment. |
| pyproject.toml | Adds numpy>=1.26.4 as an explicit project dependency. |
| src/orcapod/extension_types/numpy_type.py | New logical type mapping numpy.ndarray ↔ Arrow large_binary via .npy. |
| src/orcapod/hashing/semantic_hashing/builtin_handlers.py | Adds/registers NumpyArrayHandler for array semantic hashing. |
| src/orcapod/extension_types/init.py | Re-exports LogicalNumpyArray. |
| src/orcapod/contexts/data/v0.1.json | Registers the new logical type + hash handler in the default context. |
| tests/test_hashing/test_numpy_handler.py | New unit tests for NumpyArrayHandler. |
| tests/test_extension_types/test_roundtrips.py | Adds integration round-trip coverage for numpy.ndarray. |
| tests/test_extension_types/test_numpy_type.py | New unit tests for LogicalNumpyArray. |
| superpowers/specs/2026-06-30-itl-460-numpy-ndarray-native-value-type-design.md | Design spec documenting the approach and constraints. |
| superpowers/plans/2026-07-01-itl-460-numpy-ndarray-native-value-type.md | Implementation plan/checklist for ITL-460. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| class NumpyArrayHandler: | ||
| """Hasher for ``numpy.ndarray`` — content hash via numpy's ``.npy`` binary format. | ||
|
|
||
| Returns the raw ``.npy`` bytes produced by ``np.save``, which encode dtype | ||
| (including structured/record field names), shape, byte order, and data in | ||
| numpy's stable on-disk format. This is identical to what ``LogicalNumpyArray`` | ||
| stores in Arrow, so the hash input and storage bytes are always consistent. | ||
|
|
||
| Object-dtype arrays are rejected with ``ValueError`` immediately (before | ||
| ``np.save`` is called). Structured dtypes containing object-typed fields | ||
| are caught by ``allow_pickle=False`` in ``np.save``. | ||
| """ | ||
|
|
||
| def handle(self, obj: Any, hasher: "SemanticHasherProtocol") -> bytes: | ||
| """Return the ``.npy`` bytes for ``obj``. | ||
|
|
||
| Args: | ||
| obj: A ``numpy.ndarray`` instance. | ||
| hasher: Ignored. Present for protocol conformance. | ||
|
|
||
| Returns: | ||
| Raw ``.npy`` bytes encoding dtype, shape, byte order, and data. | ||
|
|
||
| Raises: | ||
| TypeError: If ``obj`` is not a ``numpy.ndarray``. | ||
| ValueError: If ``obj`` has ``dtype.kind == "O"`` (object dtype). | ||
| """ | ||
| import io | ||
| import numpy as np | ||
| if not isinstance(obj, np.ndarray): | ||
| raise TypeError( | ||
| f"NumpyArrayHandler: expected numpy.ndarray, got {type(obj)!r}" | ||
| ) | ||
| if obj.dtype.kind == "O": | ||
| raise ValueError( | ||
| f"NumpyArrayHandler does not support object-dtype arrays " | ||
| f"(dtype={obj.dtype!r}). Object arrays require pickling, " | ||
| "which is disabled for security." | ||
| ) | ||
| buf = io.BytesIO() | ||
| np.save(buf, obj, allow_pickle=False) | ||
| return buf.getvalue() |
There was a problem hiding this comment.
Fixed in commit c75562b. NumpyArrayHandler.handle() now returns ContentHash("sha256", hashlib.sha256(npy_bytes).digest()) instead of the raw .npy bytes.
This matches the FileHandler/DirectoryHandler pattern and short-circuits the semantic hasher at its isinstance(obj, ContentHash) terminal check (semantic_hasher.py line 165), bypassing the BytesHandler hex-expansion + JSON-serialisation path entirely. For a 100 MB neural data array this avoids the ~400 MB of temporary allocations that the old path would require.
Tests in test_numpy_handler.py updated: test_returns_bytes → test_returns_content_hash, with assertions checking isinstance(result, ContentHash) and result.method == "sha256". The equality/inequality comparisons continue to work because ContentHash is a frozen dataclass with __eq__.
…nsion overhead
- Remove unused `import io` from test_numpy_type.py
- Change NumpyArrayHandler.handle() to return ContentHash("sha256", sha256_digest)
instead of raw .npy bytes, avoiding the BytesHandler hex-expansion + JSON
serialisation path for large arrays. This matches the FileHandler/DirectoryHandler
pattern and short-circuits the semantic hasher at the ContentHash terminal check.
- Update test_numpy_handler.py: test_returns_bytes → test_returns_content_hash,
equality/inequality assertions updated to compare ContentHash objects
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Review round 2 — changes made (commit c75562b)Two comments from Copilot addressed: 1. Unused 2. The semantic hasher (
All 39 numpy-related tests pass after these changes. |
eywalker
deleted the
eywalker/itl-460-support-numpyndarray-as-a-native-value-type
branch
Summary
LogicalNumpyArray— mapsnumpy.ndarray↔ Arrowlarge_binaryusing numpy's.npybinary format, preserving dtype (including structured/record field names), shape, byte order, and data for 1-D, 2-D, N-D, and structured arraysNumpyArrayHandler— content hashes arrays via the same.npybytes stored in Arrow, ensuring hash/storage consistency; registered in bothv0.1.jsonandregister_builtin_python_type_handlersdtype.kind == "O"guard fires beforenp.save;allow_pickle=Falseinnp.saveis a secondary guard for structured dtypes with object sub-fields)Test plan
LogicalNumpyArray(protocol conformance, storage guards, round-trips: 1-D float64, 2-D int32, 3-D uint8, zero-size, single-element, structured record array, F-order)NumpyArrayHandler(bytes return, same content → same bytes, dtype/shape sensitivity, structured field name sensitivity, TypeError/ValueError guards)test_roundtrips.pyfor both Parquet and Delta Lake backends, covering simple 1-D and structured (record) arraysLinear
Fixes ITL-460