Skip to content

feat(gfql)!: AUTO treats polars as a first-class compute engine (+ index gate agreement, typed degree facts) - #1873

Merged
lmeyerov merged 21 commits into
masterfrom
feat/gfql-engine-resolve-modern
Aug 13, 2026
Merged

feat(gfql)!: AUTO treats polars as a first-class compute engine (+ index gate agreement, typed degree facts)#1873
lmeyerov merged 21 commits into
masterfrom
feat/gfql-engine-resolve-modern

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Default-policy change — needs explicit owner sign-off even when green.

resolve_engine(AUTO) on polars frames now returns POLARS. The legacy semantics — polars is an input format, coerce to pandas — survive under an explicit name, resolve_input_engine, used by the surfaces that genuinely compute in pandas (layouts, hypergraph, ComputeMixin's degrees/materialize/topological). Migrating any of those to native polars later = flipping one call site.

Why

Three AUTO gates had grown around the legacy mapping (legacy resolver, #1849's query router, #1843's index gate) and they already disagreed — mixed and edges-only frames routed differently per surface, which is the #1767 cliff class: an index built for one engine meets a query routed to another, every hop declines, answers stay correct so nothing fails.

Load-bearing details

  • NIE fallback made explicit: when native polars declines, AUTO serves via pandas by statement, not by accident of the legacy mapping.
  • Transitional policy guard at the single public entry: with a policy attached, AUTO serves via pandas until the polars postload/postchain hook gap is fixed. Commented delete-me-when-fixed; the pins assert the hook contract, not this mechanism (owner: the old policy_disables_the_route pinned a workaround's side-effect as public interface).
  • A latent wrong-answer surfaced: pd.api.types.is_numeric_dtype returns a confident False for polars dtypes — polars Float64 classified as a string in the pushdown planner. Polars dtypes now classify first-class.
  • Doctrine pins flipped to invariants per the never-pin-accidents rule (mixed-frames bridge, hop-bridges-to-pandas, engine_mismatch-decline, nullable-Boolean omission).

Folds in #1843 (index AUTO polars rework)

The two halves are atomically coupled — the index-gate agreement needs modern AUTO to exist, and modern AUTO leaves the old index gate red — so they land as one unit (#1843 closed with pointers). Its content here: resolve_index_engine derives from the frames (narrowing to pandas only when both aren't eager polars — the index can't gather LazyFrame rows) instead of post-correcting pandas on a polars path; typed per-relationship degree facts with identity anchored to the bound edge frame; the cross-engine AUTO agreement matrix (test_auto_engine_agreement.py).

Two more route pins flipped on the way to green: string predicates and bare float comparisons are now rendered by the WHERE pushdown on polars (the broken classifier was why they weren't), so "query raises upstream" and "float reaches the residual lane" were accidents — the pinned contracts are pandas-oracle parity and NaN-normalization-at-ingest.

Perf receipts live in pyg-bench (q8@100k 13.5→3.90 ms, WIN 2.6× vs warmed Kuzu; lane graphbench-100k-degree-20260812).

🤖 Generated with Claude Code

https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi

@lmeyerov lmeyerov mentioned this pull request Aug 12, 2026
Merged
@lmeyerov lmeyerov changed the title feat(gfql)!: AUTO treats polars as a first-class compute engine feat(gfql)!: AUTO treats polars as a first-class compute engine (+ index gate agreement, typed degree facts) Aug 12, 2026
This was referenced Aug 13, 2026
Closed
Merged
Open
Open
Open
Open
Open
Open
Open
Open
Merged
Open
Open
lmeyerov
lmeyerov commented Aug 13, 2026
View reviewed changes
Comment thread graphistry/compute/gfql/lazy/engine/polars/dtypes.py Outdated
from graphistry.compute.typing import PolarsFrame, PolarsT


def is_polars_dtype(dt: object) -> bool:

@lmeyerov lmeyerov Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

object bad

lmeyerov
lmeyerov commented Aug 13, 2026
View reviewed changes
Comment thread graphistry/compute/filter_by_dict.py Outdated
def _is_string_dtype_safe(dtype: Any) -> bool:
from graphistry.compute.gfql.lazy.engine.polars.dtypes import is_polars_dtype, is_stringlike
if is_polars_dtype(dtype):
return is_stringlike(dtype)

@lmeyerov lmeyerov Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move generic helpers out (dtypes.py?)

@lmeyerov
lmeyerov force-pushed the feat/gfql-engine-resolve-modern branch from c1775ca to 0b0a615 Compare August 13, 2026 05:31
@lmeyerov lmeyerov mentioned this pull request Aug 13, 2026
Open
lmeyerov and others added 11 commits August 13, 2026 01:29
@lmeyerov @claude
wip(engine): modern AUTO resolver -- 9 known test deltas remain, see …
fb4e4a8 
…plan.md

resolve_engine: AUTO+polars -> POLARS. resolve_input_engine carries legacy
semantics explicitly (layouts, hypergraph, ComputeMixin pandas-computing
surfaces migrated mechanically -- zero behavior change there). gfql_unified NIE
fallback made explicitly pandas. test_polars.py 16 fails -> 0.

Differential vs master (identical files both sides): exactly 9 migration-caused
failures in 3 groups -- (1) 2x dtype-pushdown deferral pins, (2) 4x 1767-era
'polars index + AUTO not usable' doctrine pins that r1843 flips, (3) 3x
TestAutoEngineDoesNotRoutePolarsNative, of which policy_disables_the_route is a
REAL constraint: #1849's arm excluded policy because of the polars postload/
postchain hook gap, and modern generic resolution would now route polars WITH
policy set, making that gap reachable. Needs a policy guard or the hook fix, not
a test flip.

DO NOT MERGE without owner sign-off: default-policy change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
fix(gfql): polars-aware dtype classifiers -- a latent wrong-answer th…
88e7522 
…e migration exposed

pd.api.types.is_numeric_dtype returns a CONFIDENT False for polars dtypes (no
exception, so the except-fallback never ran) and _is_string_dtype_safe returned
True for everything -- polars Float64 classified as a STRING. Confident-wrong is
not screened by try/except; polars dtypes now classify first-class.

Two pins updated from accident to invariant per the standing rule: 'flag
omitted' was an artifact of the legacy pandas conversion; under modern AUTO the
executor filters polars Boolean natively and the planner must REPORT what the
executor filters. The head(0) hazard demo likewise: the divergence it
demonstrated cannot occur when the probe never converts.

Deltas: group (1) closed, 9 -> 7 (4 are r1843's doctrine flips, 3 policy-hook).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
fix(gfql): transitional policy guard at the single entry; mixed-frame…
36dacd5 
…s pin flipped

POLICY (group 3): with a policy attached, AUTO serves via pandas -- at the ONE
public entry, not scattered across the 8 resolution points. Explicitly
TRANSITIONAL: hooks are the governance surface and must fire exactly once on
whatever engine serves; the polars route has a known postload/postchain hook
gap. The comment says delete-me-when-fixed, and the surviving pins assert the
hook CONTRACT rather than this mechanism, so removing the guard later will not
fight the tests. Per owner: the old test_policy_disables_the_route pinned a
workaround's side-effect as if it were public interface -- same genus as
benchmark hacking.

MIXED FRAMES: modern AUTO follows the edges frame and coerces nodes across;
_coerce_input_formats IS the bridge the old doctrine said did not exist. Pin
flipped from 'must not try' (absence-of-bridge accident) to value + engine
invariants. 173/173 in the conformance file.

Remaining deltas: only the 4 doctrine flips that r1843 owns -- next step is
rebasing r1843 onto this branch and landing as a stack.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
perf(gfql): index AUTO preserves resident polars frames (rework of #1767
5030e80 
, stacked on #1743)

resolve_engine(AUTO) maps polars input frames to PANDAS (legacy input-format
policy), so gfql_index_all()/create_index() with the default engine
coerce-and-REPLACED a user's polars frames with pandas copies: every later
polars-engine query then paid a full-frame pandas->polars re-conversion (O(E)
per call) and the resident pandas index could never fingerprint-match.

New resolve_index_engine(): when the caller said AUTO and BOTH resident frames
are eager polars, the index layer resolves POLARS and indexes the frames in
place (the layer is already engine-polymorphic: numpy sidecar arrays + polars
row-gather). Wired into create_index, _is_resident_index_valid, show_indexes,
and the DDL wire's NODE_PROP reuse check. Explicit engines are unchanged.

Why this is safe NOW and was not in 2026-07: the retracted #1767 shipped this
without query-side AUTO routing, so an AUTO-built polars index met an
AUTO(pandas) query -> every hop declined to the scan floor, a measured
125-534x DEFAULT-path regression. This branch is STACKED ON #1743
(fix/gfql-auto-engine-polars-native), whose gfql() guard routes AUTO on
all-polars-frame graphs to the native polars engine -- so the cliff inverts
into the win: AUTO builds the polars index, AUTO queries route polars, index
serves. Pinned by test_inversion_auto_index_auto_gfql_serves_polars_index
(index_trace() must show path=index engine=polars with NO engine argument
anywhere, plus value parity vs both explicit-engine spellings).

show_indexes() under AUTO now reports the routed truth through the #1841
usable/reason columns: the previously not-usable "polars index + AUTO query"
combination flips to usable=True (test renamed to
test_polars_index_auto_query_usable_the_1841_flip); explicit mismatched-engine
previews keep the #1838 decline wording.

Deliberately narrower than #1743's query gate, self-consistent either way
because create_index coerces the frames it indexes: LazyFrame graphs keep the
legacy pandas build (an index cannot gather from a lazy plan), and edges-only
graphs keep it too (materialize_nodes() does not yet produce polars nodes from
polars edges -- pre-existing gap, also reachable via explicit engine='polars').
Pandas/cuDF graphs, mixed frames, and nodes-only graphs are unchanged, each
pinned.

Verification (local CPU lane, cudf/gpu params env-excluded on this box):
index suite 229 passed; polars cypher conformance + cache coverage lock
177 passed / 3 skipped; lint.sh clean; scoped mypy (api.py, wire.py) clean;
type-hygiene guard OK (no growth).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
@lmeyerov @claude
fix(gfql): col-stats entry point keeps AUTO polars frames too
9948117 
Rebasing #1843 onto current master surfaced a real interaction: the col-stats
entry point (added by the typed-facts campaign AFTER this PR was written) used
plain resolve_engine, and gfql_index_all now ENDS by calling it -- so under AUTO
it coerced the preserved polars frames back to pandas at the last step, breaking
all four inversion pins. Routed through resolve_index_engine like the other
index entry points. 262 index tests pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
refactor(gfql): resolve_index_engine derives from frames; pin AUTO ga…
0f20881 
…te agreement

Review: 'why does resolve_index_engine reason about pandas on a polars path?'
Because it POST-CORRECTED resolve_engine's legacy polars->pandas mapping instead
of deriving from the frames -- a third independent AUTO gate alongside the legacy
resolver and the #1849 query router, and third reimplementations drift. The gate
now reads directly: explicit engines win; AUTO + both frames eager polars ->
POLARS; everything else keeps the base resolution (never touches cudf/dask).

The reviewer's deeper suspicion -- cross-engine pos/neg coverage -- was right.
New test_auto_engine_agreement.py pins the property that actually matters: the
INDEX gate and the QUERY gate must land together per frame combination, because
their DISAGREEMENT is the #1767 failure class (index built for one engine meets
a query routed to another; every hop declines to the scan floor; answers stay
correct so nothing fails). Matrix: pandas/polars/lazy/mixed both structurally
and END TO END (AUTO build then AUTO query, result frames must match the built
engine). Explicit-engine-wins pinned. cudf pinned as source-native with the
polars-gpu question left VISIBLE rather than silently decided either way.

Also: _is_eager_polars typed Optional[DataFrameT] and built on is_polars_df
(which admits LazyFrames -- the one-line reason it exists); the 32-line
resolve_index_engine docstring cut to the two underivable facts (the #1767
cliff history and the do-not-widen constraint), gates pointed at their pins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
fix(gfql): index gate narrows modern AUTO; last doctrine pins flip to…
c479725 
… invariants

On the stack, resolve_engine's base is POLARS for lazy/mixed too, so
resolve_index_engine inverts from escalation to NARROWING: polars-resolved but
not (both frames present and eager) -> pandas, because an index cannot gather
rows from a lazy plan. The self-heal keeps the gates in AGREEMENT (create_index
coerces the frames it indexes, so later AUTO queries route with post-build
frames) -- pinned in test_auto_engine_agreement.

Last two doctrine pins flipped per the never-pin-accidents rule:
- 'hop AUTO still bridges to pandas' -- the bridge WAS the 1767-era accident;
  modern AUTO serves hop natively on polars frames, values pinned.
- 'expected an engine_mismatch decline' -- under agreed gates that scenario
  CANNOT occur (both sides resolve polars); the invariant is an honest
  cost/coverage decline on the AGREED engine, never a mismatch.

Full stack: 0 non-cudf failures across index + conformance + lowering.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
test(gfql): string-predicate and NaN-guard route pins flip to invariants
b36ddc6 
The polars-first dtype classifiers let the WHERE pushdown render string
predicates and bare float comparisons natively, so two pins of the broken
classifier's routing became false: the query-raises pin (the gap is closed,
parity with the pandas oracle is the contract) and the float-residual spy
(a bare comparison no longer reaches the residual lane; abs() keeps the
NaN-guard test on the lane it exercises).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
test(gfql): q9 shape lock -- endpoint filters decline dense, fused co…
f4708f2 
…unt serves

The precomputed degree product counts ALL paths through a midpoint, so under
WHERE filters on the endpoints consulting it is a wrong answer, not a slow
one; and the decline must not cascade past the fused two_hop_count arm into
a scan. Board receipt: q9@100k rides the fused arm at 36.25ms, WIN 2.31x vs
Kuzu (pyg-bench degboard-100k).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
fix(gfql)!: review-wave fixes -- policy guard keys on resolve_engine;…
02cf5a4 
… input surfaces migrated

Two blockers from the stack review, both pinned:
- A denying policy could be silently bypassed on MIXED-frame graphs under
  AUTO: the transitional guard keyed on frame shapes while routing keys on
  resolve_engine. The guard now uses resolve_engine as its predicate;
  mixed-frame and edges-only deny pins added.
- Unmigrated pandas-computing surfaces regressed under modern AUTO
  (dbscan raised bare AssertionError; chain_remote/python_remote
  hard-errored; circle/fa2 layouts). All five now use resolve_input_engine,
  with a per-module convention pin.

Also: polars string classification keys on the dtype's type name (repr
whitelist missed parameterized Enum), with both-sides classification pins;
gfql_index_col_stats AUTO-preservation pin; accident-pins restated as
contracts (1767 hop test no longer pins the cost gate's fixture-size
verdict; routing-boundary class renamed to what it now asserts; residual
parity test drops its internal-function spy); comment diet per the
editorial standard (perf magnitudes -> pyg-bench pointers, falsified
legacy-AUTO comments corrected); CHANGELOG entries for the resolver flip,
classifier fix, and policy guard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
test(gfql): dbscan resolution pin tolerates a backend-less lane
d27058a 
The regression was a bare AssertionError from the engine check; a lane with
neither cuml nor sklearn raises the documented dependency ValueError, which
the pin now accepts as the other legitimate outcome.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
lmeyerov and others added 10 commits August 13, 2026 01:30
@lmeyerov @claude
test(gfql): wave-2 convergence -- discriminating col-stats lazy arm, …
603f2ed 
…nodes-only deny pin

The eager col-stats pin could not distinguish the index gate from plain
resolve_engine (they agree on eager fixtures); the lazy arm can -- plain
resolve_engine crashes a lazy col-stats build, the gate narrows and answers.
Nodes-only polars graphs get their own deny pin (the one frame combination
the old frame-shape guard missed that the edges-only pin did not cover).
Dead resolve_engine import dropped from ComputeMixin.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
lint: drop unused importorskip binding
42dbbcc 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
refactor(gfql): _is_eager_polars composes the canonical predicates
5e47980 
is_polars_df (the SSOT type guard) + dtypes.is_lazy (TypeIs) instead of a
stringly type-name check -- the exact reimplementation pattern is_polars_df
exists to end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
refactor(gfql): polars dtype detection consolidates onto dtypes.py pr…
98c3309 
…edicates

is_polars_dtype joins the canonical polars-dtype vocabulary module (the
metaclass puts classes and instances under one module, so the frame-guard
sniff transfers), and the string classification delegates to is_stringlike
instead of re-deriving its whitelist. The numeric branch stays on polars'
own dtype.is_numeric() -- Decimal is in scope for this planner where
dtypes.is_numeric deliberately excludes it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
types(gfql): is_polars_dtype is a TypeGuard[PolarsDType], predicates …
15a1ad0 
…take the honest union

Owner review: 'object' input is deliberate (the DType alias is Any, which
would UNcheck the gate), but bool discarded the proof -- now TypeGuard, the
is_polars_df pattern. PolarsDType joins the canonical typing module as the
instance-or-class union polars circulates; the guard's narrowing immediately
surfaced that the typed predicates declared instances only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
refactor(gfql): cross-engine dtype classifiers live in the dtype-cont…
52ffd1e 
…ract module

filter_by_dict.py carried the pushdown planner's dtype classification; it
moves to dtypes.py (one home for the cross-type contract, public names),
with thin deferred delegations left behind -- the polars package __init__
transitively imports filter_by_dict, so a module-level import cycles.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
types(gfql): py3.8-compatible PolarsDType (typing.Type) + localized c…
e663114 
…lass-arm ignore

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
test(gfql): fallback-arm coverage for the relocated dtype classifiers
22421af 
The polars lane's per-file floor caught the move (dtypes.py 79% vs 92%):
the cross-engine classifiers' fallback contracts were unexercised. Pins:
polars-module dtype without the dtype API -> substring arm; raising pandas
classifier -> kind/text arms; str()-raising object -> classifies as nothing,
never an exception.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
fix(gfql): string-dtype fallback recognizes the pandas-3 'str' dtype …
df0b490 
…name

The py3.14 lanes run pandas whose default string dtype reprs as 'str', which
the text fallback missed (exact match added; 'struct' pinned non-string).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov @claude
test(gfql): fallback-arm pin selected by the polars lane's -k filter
d83b581 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov
lmeyerov force-pushed the feat/gfql-engine-resolve-modern branch from f16810c to d83b581 Compare August 13, 2026 08:30
@lmeyerov
lmeyerov merged commit f1380cd into master Aug 13, 2026
77 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lmeyerov