fix(gfql): graceful numpy fallback when cupy cannot JIT (NVRTC-less CUDA)

[lmeyerov]

The NVRTC-less-cudf follow-up from the benchmark campaign. cudf imports and runs precompiled libcudf ops on a host whose CUDA install lacks NVRTC, but cupy's kernel compiler — which the GFQL index/count kernels' bincount/searchsorted require — raises RuntimeError: CuPy failed to load libnvrtc.so.12 at first use, deep inside a query.

array_namespace now probes one JIT-compiled cupy op per process (cached) and falls back to numpy host arrays for the cudf engine, col_to_array aligned to the same probe: identical values, host-side compute, no crash — decline-not-serve. Diagnosed empirically on a real NVRTC-less box (cudf constructs/merges/groupbys pass; cupy bincount/searchsorted raise); with the fix, the three previously-crashing cudf suite tests pass there. Pins: forced-fallback backend + array types, probe run-once caching. No new casts, no new Any (ratchets clean).

For owner review.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi

[lmeyerov]

DGX container matrix (cd149a4f, NVRTC present so the device path stays engaged): 8,370 passed, 53 skipped, 19 xfailed — no regression from the probe; the fallback arms are covered by the forced-fallback pins, exercised for real on the NVRTC-less dev box.

🤖 Generated with Claude Code

[lmeyerov]

Scoping audit done — you were right that the first patch treated a symptom. Reworked:

Diagnosis (measured on the NVRTC-less box): essentially ALL cupy compute fails there — arithmetic, comparisons, trig, astype, sort/search/bincount; only allocation and cub-backed sum survive — while cudf's dataframe engine passes everything (construct/filter/merge/groupby/string-eq/isin/to_pandas). The distinction is architectural, which answers the 'don't we need cupy for all cudf?' question: cudf executes through precompiled libcudf kernels; cupy JIT-compiles elementwise kernels via NVRTC at first use. cudf touches NVRTC only on numba-UDF paths our engine avoids. So an NVRTC-less install keeps the whole cudf engine correct and fast, and only the cupy array sidecars need to degrade.

Solution (library-level, not module-local): lazy_cupy_import() in graphistry/utils/lazy_import.py — available iff cupy can compute (one cached probe, reason preserved) — replacing the private probe. Consumer audit:

site	verdict
gfql/index/engine_arrays	consumes the gate (private probe deleted)
layout/mercator	fixed — except ImportError could never catch the compute-time RuntimeError; now falls back to NumPy with the real reason logged
layout/ring/util	fixed — was an unconditional crash on the cudf path; now routes to its existing numpy branch
layout/circle	safe — allocation/mem-info APIs only (work NVRTC-less)
umap_utils, ai_utils	safe — only touch cupy arrays produced by cuml (a working stack implied)
compute/cluster, Engine.s_sqrt/s_arange	same hazard class, cuml-coupled/caller-contract — flagged follow-up
cudf numba-JIT expression paths (round, temporal)	pre-existing on master: raise GFQLTypeError on NVRTC-less boxes instead of honest NIE — flagged follow-up

Verification: mypy + hygiene clean; gate + fallback pins refit to the library gate and green on the NVRTC-less box; layout suites 410 green locally; the 27 local conformance failures reproduce identically on pristine master (pre-existing box artifact, green in the DGX container). Fresh container matrix queued.

🤖 Generated with Claude Code

[lmeyerov]

Container matrix on the gate rework (4d2f4549, now including the layout suites): 8,785 passed, 1 failed — the failure is test_gib_cudf (group-in-a-box layout), a suite first added to this container run by the rework itself, so it has no prior in-container baseline; a master-baseline check is queued behind the benchmark lanes currently holding the box. Will report whether it's pre-existing or mine before calling this done.

🤖 Generated with Claude Code

[lmeyerov]

Generalization follow-through, two more hazard-class members found and fixed (both repro'd on the NVRTC-less box):

1. Scalar fns (floor/ceil/round tails) applied numpy ufuncs to cudf Series — silently dispatching into cupy kernel compilation — because the legacy Series.floor()/.ceil() native branch died when cudf 26.02 removed those methods. New _unary_ufunc_on_series routes through host numpy only when cupy cannot compute; healthy stacks keep the on-device path.
2. Temporal date/time predicates called Series.dt.date/.dt.time — both also removed in cudf 26.02 (pure API drift, not NVRTC). Day-truncated datetimes now pair with a midnight Timestamp, time-of-day timedeltas with a Timedelta — dtype-driven pairing, identical comparison semantics on every engine.

Receipts: the local NVRTC-less conformance matrix goes 27 failed → 270 passed / 0 failed. mypy + hygiene clean. Container matrix re-queue waits behind the benchmark lane chain currently holding the dgx box (also still owing the test_gib_cudf master-baseline check from the previous comment).

🤖 Generated with Claude Code

[lmeyerov]

Encapsulation per review: cudf_runtime_caps() is now the ONE question for typical cudf-path code — bundles cudf importability (precompiled engine) and cupy compute capability (JIT sidecar), with modules and reasons; every consumer site switched, lazy_cupy_import demoted to the low-level probe. NVRTC-less conformance matrix stays 270 passed / 0 failed on the full stack.

🤖 Generated with Claude Code