add morton_index datatype user docs (issue #64)

[espg]

Closes #64. Refs #48, #51.

What

A new user-facing page, docs/morton_index_datatype.md, documenting the morton_index datatype skin: the pandas ExtensionArray (MortonIndexArray / MortonIndexDtype, registered as "morton_index") and the matching pyarrow ExtensionType.

It covers:

• Optional-dependency story — numpy stays the only hard runtime dep; pandas / pyarrow are extras (mortie[pandas], mortie[pyarrow]). The exact ImportError messages raised by the lazy __getattr__ import guards are quoted so a numpy-only user knows what they'll see.
• pandas ExtensionArray — construction (from_nested, from_latlon, from_words, pd.Series(..., dtype="morton_index")), the packed-uint64 storage and how the raw constructor expects packed words (not decimal ids), the decimal-Morton repr and MortonIndexScalar (displays 11, int()s to the packed word), order-aware accessors (.orders()/.order(), .base_cells()/.base_cell(), .is_fixed_order(), .coarsen(k)), Z-order comparisons/argsort, and the NA / empty-sentinel semantics.
• pyarrow ExtensionType — from_morton_index / to_morton_index, null↔sentinel round-trip, parquet + table.to_pandas() landing back as morton_index, and pa.array(arr) via the C-Data interface. Points at docs/arrow_interchange.md for the pyarrow-free C Data surface.
• numpy ↔ pandas ↔ arrow round-trip table.
• Forward pointer to the deferred, NEP-42-style operator grammar: 1.x supports comparisons/ordering and the domain accessors but not arithmetic operators (raw arithmetic on a packed word is meaningless); the richer operator grammar is out of scope for 1.x and won't change the meaning of anything documented. Grounded against the code (mortie/morton_index.py docstrings: "no arithmetic operators are defined").

Also added a one-paragraph cross-link from the "Integration with DataFrames" section of USAGE.md.

No code changes — docs only.

Approach

Grounded every snippet in the real API by building the extension (maturin develop --release) with the test extra (pulls pandas + pyarrow) and executing each example, then pasting the actual output as inline comments. Two early draft snippets that mis-used decimal ids as packed words (rendering <invalid ...>) were corrected to use real packed words pulled from np.asarray(arr, dtype=np.uint64). The pa.Table.from_pandas(df) path was found to raise ArrowTypeError on a morton_index column (no __arrow_array__ hook) — documented as a known non-path with the explicit-conversion workaround rather than shown as working.

Phases

• Write docs/morton_index_datatype.md with runnable, extras-gated examples; wire a cross-link into USAGE.md.

How it was tested

• maturin develop --release + pip install -e ".[test]" (pandas 3.0.3, pyarrow 25.0.0).
• Every code block executed against the built extension; outputs shown are the real results.
• flake8 mortie --select=E9,F63,F7,F82 — clean.
• pytest -q — 683 passed, 11 skipped, 1 failed. The single failure is the pre-existing test_spec_page.py::TestDecimalParseTieBreak::test_order29_string_parses_to_area_word (issue main is red: test_spec_page tie-break test predates #120's p-suffix rendering #123, tracked in a separate PR); a docs-only change does not touch it.

Questions for review

• Placement: the page lives at docs/morton_index_datatype.md alongside arrow_interchange.md / coverage_methods.md / specification.md, cross-linked from USAGE.md. There is no sphinx/mkdocs toctree in the repo (docs are plain markdown linked from README.md / USAGE.md), so no index wiring beyond the cross-link — confirm that's the intended home.
• The NEP-42 operator grammar is described as "deferred / forward-looking" with no dedicated tracking issue found in the repo (only v1.0 docs: morton_index datatype user documentation #64 mentions it). I kept the forward pointer deliberately non-committal (no version promised). Flag if there's a specific issue/design doc it should link to.

🤖 Generated with Claude Code

https://claude.ai/code/session_015Ziny2mbAruJjuNQTTaHip

---

Generated by Claude Code

[espg]

🤖 from Claude (review)

Fresh adversarial review of this docs-only change, validated by reading mortie/morton_index.py, mortie/arrow.py, mortie/__init__.py, and pyproject.toml (not by running maturin).

Accurate — spot-checked against source and confirmed:

• Every documented method/accessor exists with the claimed behavior: from_nested / from_latlon / from_words, orders() / order() (raises on mixed), base_cells() / base_cell(), is_fixed_order(), coarsen(), to_nested(), decimal_repr() / to_decimal() ("<U32"), hive_path(), isna(), from_morton_index / to_morton_index / morton_index_type, from_arrow, and the MortonIndexScalar display-vs-int() split.
• The two quoted ImportError messages match the __getattr__ import guards verbatim (pandas and pyarrow).
• Pinned numeric outputs I could recompute are exact: the packed words [1152921504606846977, 1441151880758558721, 1729382256910270465, 2017612633061982209], argsort → [1 3 2 0], b == arr → [False False True False], and coarsen(2) mapping 12321/12322/12323/12324 → 123.
• Extras story matches pyproject.toml: numpy is the only hard runtime dep; pandas / pyarrow are optional extras; each example is labelled *Requires mortie[...]* and the "every example below requires the relevant extra" gate is stated up front.
• Extension name mortie.morton_index, the extension<mortie.morton_index<MortonIndexType>> repr, the null↔empty-sentinel round-trip, and the __from_arrow__ / to_pandas_dtype parquet landing all line up with arrow.py.
• The in-page anchor #what-is-and-isnt-supported-in-1x resolves, and all three See also cross-links (arrow_interchange.md, specification.md, coverage_methods.md) exist under docs/.
• The NEP-42 forward pointer is appropriately hedged (deferred / forward-looking, no version promised, "nothing above will change its meaning") — no over-promise; and it correctly notes the backing set ops already exist as moc_and / moc_or / moc_minus.
• No @-mentions; no misuse of #N in the doc body (issue references stay in the PR body where they belong).

Findings: two non-blocking nits, inline. Nothing blocking — issue #64's acceptance points (extras-gated runnable examples, how the dtype wraps the packed-u64 kernel, repr + order accessors, optional-dep story, deferred-grammar pointer without over-promising) are all met.

---

Generated by Claude Code

[espg]

🤖 from Claude (review)

nit (non-blocking): this is the one NA example, and it reaches for the underscore-private _from_sequence, which reads as internal API in a user-facing doc. The public path builds the same array and is what a reader would actually write:

na = pd.array([int(words[0]), pd.NA, int(words[2])], dtype="morton_index")

(pd.array dispatches to _from_sequence internally.) Verified against mortie/morton_index.py: _coerce_words maps pd.NA/None/NaN → the all-zero sentinel, so isna() still round-trips the same way.

---

Generated by Claude Code

[espg]

🤖 from Claude (review)

nit (non-blocking): pinning the exact class ArrowTypeError here is version-sensitive. The behavior was verified on pandas 3.0.3 / pyarrow 25, but the extras pin allows pandas>=2.0 / pyarrow>=14, and older stacks can surface a different Arrow error (ArrowInvalid / ArrowNotImplementedError) for the same missing-__arrow_array__ case. Consider softening to "raises an Arrow error" — the real takeaway is the workaround (from_morton_index / pa.array(series.array)). Confirmed against mortie/morton_index.py that MortonIndexArray deliberately defines __arrow_c_array__ but no __arrow_array__, so the pandas→arrow from_pandas path does indeed fail.

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will not alter performance

✅ 67 untouched benchmarks
⏩ 1 skipped benchmark¹

---

_{Comparing claude/64-morton-index-datatype-docs (1b479ce) with main (f773873)}

Footnotes

1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

[espg]

🤖 from Claude (review)

Adversarial review of the morton_index datatype docs (issue #64). I built the Rust extension and executed every runnable snippet on the PR branch against a real mortie[pandas,pyarrow] env.

Verified accurate (every claim held):

• All pandas outputs reproduce exactly as commented: from_nested/from_latlon reprs, the packed-word round-trip (from_words ↔ np.asarray(..., uint64)), pd.Series(words, dtype="morton_index"), MortonIndexScalar decimal-vs-int() split, decimal_repr/to_decimal, the order accessors, the mixed-order ValueError text, coarsen, argsort/== Z-order, and the <NA> sentinel/isna behavior.
• All pyarrow outputs reproduce exactly: the extension<mortie.morton_index<MortonIndexType>> type repr, from_/to_morton_index, null↔sentinel round-trip, the parquet + table.to_pandas() landing as morton_index, pa.array(arr) via the C-Data path, and the documented pa.Table.from_pandas → ArrowTypeError.
• Both ImportError message blocks are verbatim from morton_index.py / arrow.py; the extras (mortie[pandas], mortie[pyarrow]) match pyproject.toml; numpy stays the only runtime dep; the lazy-__getattr__ optional-dependency story is correctly described.
• The NEP-42 forward pointer is honest — arithmetic is genuinely undefined, the operator grammar is framed as deferred/non-1.x without over-promising, and the accessor methods are correctly named as the stable 1.x contract (with the existing moc_* functions cited).
• Cross-doc links (arrow_interchange.md, specification.md, coverage_methods.md) and the internal anchor #what-is-and-isnt-supported-in-1x all resolve.

Findings: none blocking. Two optional polish notes inline (an undemonstrated .hive_path() in the Supported list; a minor np.asarray dtype-idiom inconsistency). Extras-gating is clearly stated per section. The page is terse and accurate.

---

Generated by Claude Code

[espg]

🤖 from Claude (review)

.hive_path() is listed here as a supported 1.x accessor but is the only one on this list never demonstrated or explained anywhere on the page (.orders()/.order(), .base_cells()/.base_cell(), .is_fixed_order(), .coarsen(k), .to_nested() all get an example or a mention). It's a real method (MortonIndexArray.hive_path(root="", suffix=".zarr") in mortie/morton_index.py), so the claim is accurate — but a reader hits a dead reference. Optional/non-blocking: either add a one-line example, or drop it from the enumeration to keep the page total. Not a defect in what's written.

---

Generated by Claude Code

[espg]

🤖 from Claude (review)

Very minor consistency nit (non-blocking): earlier the page deliberately teaches np.asarray(arr, dtype=np.uint64) as the idiom for pulling the packed uint64 storage back out (Construction section, and again on line 210 w = np.asarray(arr)). Here the mixed example uses the un-dtyped np.asarray(arr)/np.asarray(pts). Both work — I verified the whole snippet runs and reproduces order=mixed exactly — because element iteration yields MortonIndexScalar (a np.uint64 subclass), so numpy infers uint64. But relying on inferred dtype here reads as slightly inconsistent with the storage-extraction idiom the page just established. Consider np.asarray(arr, dtype=np.uint64) for uniformity.

---

Generated by Claude Code

[espg]

I'm trying to understand this (under section):

print(arr.base_cells()) # [0 0 0 0]

can you explain? Why is repr returning healpix base cells instead of morton base cells in this example?

[espg]

🤖 from Claude

Folded the four non-blocking self-review nits into docs/morton_index_datatype.md (commit 0a81afb). All snippet outputs below are runtime-verified against a fresh maturin develop --release build (numpy/pandas/pyarrow):

1. NA example — replaced the underscore-private MortonIndexArray._from_sequence(..., dtype=arr.dtype) with the public idiom pd.array([int(words[0]), pd.NA, int(words[2])], dtype="morton_index"). pd.array dispatches to _from_sequence internally; repr and isna() round-trip identically (MortonIndexArray([11, <NA>, 13], len=3, order=mixed) / [False True False]). Surrounding prose didn't name _from_sequence, so no text change needed.

2. Arrow error — softened the pinned exact class ArrowTypeError to "raises an Arrow error" (older pandas/pyarrow stacks may raise ArrowInvalid / ArrowNotImplementedError). The workaround (from_morton_index / pa.array(series.array)) stays as the takeaway. No code comment pinned the class, so nothing else to generalize.

3. .hive_path() — was listed in the 1.x supported-accessors enumeration but never demonstrated. Added a terse one-line example in the order-aware-accessors section: arr.hive_path() → ['1/1/11.zarr', '1/2/12.zarr', '1/3/13.zarr', '1/4/14.zarr'] (verified), with a pointer to the hive-path convention in specification.md.

4. Mixed example — changed the un-dtyped np.asarray(arr) / np.asarray(pts) to np.asarray(arr, dtype=np.uint64) / np.asarray(pts, dtype=np.uint64), consistent with the storage-extraction idiom taught earlier on the page. Expected-output comment unchanged and still holds (same values).

Docs-only; no Rust/Python source touched. Marking waiting for review.

---

Generated by Claude Code

[espg]

🤖 from Claude

Good question — and the earlier run should have addressed it instead of only folding the self-review nits. There's no bug, but the example was genuinely confusing. Explanation, then the doc fix (1b479ce):

arr.base_cells() returns the native HEALPix base pixel (0–11, uint8) — the exact index mort2norm, cdshealpix, and healpy use. It delegates straight to the HEALPix decode, so it is not the leading digit of the decimal repr.

The confusion is the mismatch with the decimal-Morton repr, which renders the base cell in its signed form (spec §2): base pixels 0–5 lead with 1–6, and southern base pixels 6–11 lead with -1 to -6. In the example, from_nested([0, 1, 2, 3], depth=1) are all children of HEALPix base pixel 0 (at depth 1, nside=2, so nested ids 0–3 all satisfy id // nside² == 0). Base pixel 0 renders as decimal-leading 1, giving repr [11, 12, 13, 14] — while base_cells() correctly reports [0, 0, 0, 0].

Verified against the build, spanning the range so both forms are visible:

b = MortonIndexArray.from_nested([0, 4, 40, 44], depth=1)  # base pixels 0, 1, 10, 11
print(b.base_cells())    # [ 0  1 10 11]                 native HEALPix base pixel (0–11)
print(b.decimal_repr())  # ['11', '21', '-51', '-61']    signed decimal-Morton form

So the all-base-0 example paired [11,12,13,14] against [0,0,0,0] with no explanation of the offset. The commit adds exactly that clarifying paragraph plus this multi-base illustration to the "Order-aware accessors" section.

One thing worth your call, not changed here: .base_cells() (native 0–11) and the decimal repr's base digit (1–6 / -1..-6) are deliberately different conventions — the accessor matches the HEALPix ecosystem, the repr matches the decimal-Morton string grammar. If you'd rather the accessor mirror the decimal form for consistency, that's a public-API behavior change (it would ripple into zagg and the freeze), so I'd raise it separately rather than fold it in — say the word.

---

Generated by Claude Code