feat(extension_types): add pandas.DataFrame and pandas.Series as native value types

[kurodo3]

Summary

• Adds LogicalPandasDataFrame and LogicalPandasSeries to extension_types/pandas_type.py, mapping pd.DataFrame / pd.Series to Arrow large_binary via Arrow IPC stream bytes with preserve_index=True
• Adds PandasDataFrameHandler and PandasSeriesHandler in builtin_handlers.py for content hashing — both convert to pa.Table and delegate to the existing StarfixArrowHasher for consistency with pa.Table hashing
• Registers both types and handlers in contexts/data/v0.1.json
• Full test coverage: protocol conformance, round-trips (12 DataFrame cases, 9 Series cases), hashing determinism, error paths, end-to-end integration over Parquet and Delta backends

Closes PLT-1869

Test plan

• uv run pytest tests/test_extension_types/test_pandas_type.py -v — all pass
• uv run pytest tests/test_hashing/test_pandas_handlers.py -v — all pass
• uv run pytest tests/test_extension_types/test_roundtrips.py -v — all pass (parquet + delta)
• uv run pytest tests/ -x -q — no regressions

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 93.16239% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...capod/hashing/semantic_hashing/builtin_handlers.py	86.36%	6 Missing ⚠️
src/orcapod/extension_types/pandas_type.py	97.22%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

Adds first-class support for pandas.DataFrame and pandas.Series as native value types in orcapod by introducing Arrow extension logical types (stored as large_binary IPC bytes) and registering semantic hashing handlers + default-context wiring, with accompanying unit and integration tests.

Changes:

• Introduces LogicalPandasDataFrame / LogicalPandasSeries logical types using Arrow IPC stream bytes (preserve_index=True) for lossless round-trips.
• Adds PandasDataFrameHandler / PandasSeriesHandler that hash by converting to pa.Table and delegating to the configured Arrow hasher (arrow_v0.1 by default).
• Registers new logical types and handlers in the default v0.1 data context and adds targeted tests for protocol conformance, hashing, and backend round-trips.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/orcapod/extension_types/pandas_type.py	Adds DataFrame/Series logical types and IPC (de)serialization helpers.
src/orcapod/hashing/semantic_hashing/builtin_handlers.py	Adds pandas semantic hash handlers delegating to the Arrow hasher and registers them.
src/orcapod/contexts/data/v0.1.json	Registers pandas logical types + handlers and adds a changelog entry.
src/orcapod/extension_types/__init__.py	Exposes the new logical types in the package public API.
tests/test_extension_types/test_pandas_type.py	Unit tests for logical type protocol + round-trips across common dtypes/index cases.
tests/test_hashing/test_pandas_handlers.py	Unit tests for pandas hashing handlers (determinism, differences, error paths).
tests/test_extension_types/test_roundtrips.py	End-to-end storage backend round-trip tests for DataFrame/Series.
superpowers/specs/2026-07-01-pandas-native-types-design.md	Design spec describing the feature and expected behavior (needs alignment with implementation).
superpowers/plans/2026-07-01-pandas-native-types.md	Implementation plan document for the feature.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kurodo3]

Fixed in commit 9d6e921. Added an explicit guard at the top of LogicalPandasSeries.python_to_storage:

if value.name == _SERIES_UNNAMED_SENTINEL:
    raise ValueError(
        f"LogicalPandasSeries: Series name {_SERIES_UNNAMED_SENTINEL!r} is "
        "reserved by orcapod for unnamed Series storage. "
        "Rename the Series before storing it."
    )

Also added test_reserved_name_raises_value_error to test_pandas_type.py to cover this path.

[kurodo3]

Fixed in commit 9d6e921. Added the same sentinel guard to PandasSeriesHandler.handle right after the isinstance check:

if obj.name == self._UNNAMED_SENTINEL:
    raise ValueError(
        f"PandasSeriesHandler: Series name {self._UNNAMED_SENTINEL!r} is "
        "reserved by orcapod for unnamed Series storage. "
        "Rename the Series before hashing it."
    )

Also added test_reserved_name_raises_value_error to test_pandas_handlers.py to verify this path.

[kurodo3]

Fixed in commit 9d6e921. The spec now uses the correct sentinel "__orcapod:unnamed__" in both the python_to_storage description (steps 1–2) and the storage_to_python description (step 2). Also added a step documenting the sentinel rejection guard.

[kurodo3]

Fixed in commit 9d6e921. The Hashing Handlers section of the spec now correctly describes the actual behavior: both handlers convert to pa.Table (with preserve_index=True) and delegate to the configured Arrow hasher (defaulting to StarfixArrowHasher, which produces ContentHash(method="arrow_v0.1", ...)). The outdated SHA-256 / IPC-bytes description has been removed.

[kurodo3]

Review response — all 5 comments addressed (commit 9d6e921)

#	Comment	Action taken
1	Spec sentinel was "__value__" but impl uses "__orcapod:unnamed__"	Updated superpowers/specs/2026-07-01-pandas-native-types-design.md to use the correct sentinel throughout
2	Unused import numpy as np in test_pandas_handlers.py	Removed — numpy is used in test_pandas_type.py for dtype fixtures and was left there
3	PandasSeriesHandler would silently treat a Series named "__orcapod:unnamed__" as unnamed	Added guard: raises ValueError ("reserved") immediately after the isinstance check; added test_reserved_name_raises_value_error
4	Spec described hashing as SHA-256 of IPC bytes; impl uses Arrow hasher	Rewrote the Hashing Handlers section to document the actual pa.Table → StarfixArrowHasher → ContentHash(method="arrow_v0.1") path
5	LogicalPandasSeries.python_to_storage would silently corrupt a Series named with the sentinel	Added identical guard before the col_name computation; added test_reserved_name_raises_value_error to test_pandas_type.py

All 71 pandas-related tests pass (including the 2 new sentinel-rejection tests). Full suite: 3972 passed, 56 skipped, 6 xfailed.

[eywalker]

let's not make this name anything that's orcapod specific -- something like "pandas_series_unnamed" would have been sufficient

[kurodo3]

Fixed in commit 714647b. Changed the sentinel to "__pandas_series_unnamed__" throughout — _SERIES_UNNAMED_SENTINEL in pandas_type.py, _UNNAMED_SENTINEL in PandasSeriesHandler, all docstrings, error messages, and test assertions.

[eywalker]

I don't want this to grab anything by default -- arrow hasher must be explicitly passed in

[kurodo3]

Fixed in commit 714647b. Removed _get_arrow_hasher() and the get_default_context() fallback from both handlers. arrow_hasher is now a required constructor parameter — no default, no global state lookup. handle() uses self._arrow_hasher.hash_table(table) directly.

[eywalker]

same as above -- do NOT grab default arrow hasher -- this must be explicitly passed in the constructor

[kurodo3]

Fixed in commit 714647b — same as above. arrow_hasher is now required at construction; the lazy fallback is gone.

[eywalker]

is this function invoked anywhere? Aren't we using context loader to take care of setting up default type handlers?

[kurodo3]

Good catch. The function IS called in tests and by BuiltinPythonTypeHandlerRegistry, but you're right that the context loader (via v0.1.json) handles the pandas handlers independently.

Fixed in commit 714647b: removed the pandas handler registrations from register_builtin_python_type_handlers. Instead, a new PandasHandlerRegistrar class is added — it receives both registry and arrow_hasher explicitly and registers the handlers as a side-effect of construction. It's placed in v0.1.json AFTER arrow_hasher so both refs are already resolved in the loader's ref_lut. This also cleanly breaks the circular forward-reference constraint that prevented using {"_ref": "arrow_hasher"} inside the python_type_handler_registry handlers block.

[eywalker]

this must pass in arrow hasher explicitly

[kurodo3]

Fixed in commit 714647b. The pandas handlers are no longer in the python_type_handler_registry.handlers list at all. Instead, a new top-level entry after arrow_hasher explicitly passes both refs:

"pandas_type_handlers": {
    "_class": "...PandasHandlerRegistrar",
    "_config": {
        "registry":     {"_ref": "python_type_handler_registry"},
        "arrow_hasher": {"_ref": "arrow_hasher"}
    }
}

By the time the loader reaches this entry, both python_type_handler_registry and arrow_hasher are already in ref_lut, so the refs resolve correctly. The context schema was also updated to allow this new key.

[kurodo3]

Review response — eywalker round (commit 714647b)

#	Comment	Action
1	Sentinel shouldn't be orcapod-specific	Renamed to "__pandas_series_unnamed__" everywhere (both logical type and handler)
3	Is register_builtin_python_type_handlers needed?	Removed pandas handler registrations from it. Added PandasHandlerRegistrar class — a lightweight helper that receives registry + arrow_hasher and calls register() as a construction side-effect
4	PandasDataFrameHandler: no lazy default-context fallback	Removed _get_arrow_hasher() method; arrow_hasher is now a required constructor param
5	PandasSeriesHandler: same	Same fix
6	v0.1.json must pass arrow_hasher explicitly	Removed pandas handlers from python_type_handler_registry.handlers. Added pandas_type_handlers entry after arrow_hasher in the JSON, using {"_ref": "arrow_hasher"} — both refs are in ref_lut by that point, so no forward-reference error. Context schema updated to allow the new key

Architectural note on why pandas_type_handlers is a separate entry: The context loader builds top-level components in JSON order. python_type_handler_registry must precede semantic_hasher, which must precede arrow_hasher. Placing pandas_type_handlers after all three breaks the circular dependency cleanly without touching the loader.

Full suite: 3974 passed, 56 skipped, 6 xfailed — no regressions.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.