Repository navigation
RFC: A columnar validation path built on Arrow, ADBC, and DuckDB #12101
Replies: 3 comments 1 reply
|
I think this RFC is quite relevant because I’m exploring a native Hugging Face Datasets datasource for GX, and I think there is a useful connection between them (#11987) I believe 3 parts of this RFC seem directly relevant:
The initial HF datasource can be implemented against the existing WDYT? cc: @joshua-stauffer |
|
Yes, i think the replay contract would directly target sources like |
|
Thanks for the RFC @dataders! This is a significant proposal that would change GX architecture at multiple layers. Its strength is the creative re-imagining of GX’s core components. That’s also a liability, and I’ll say upfront that the scope here is too large to accept as is. Within the RFC there are a number of strong ideas that can be evaluated independently and accepted or rejected on their own merits, so that’s the approach I’ll take for this review. Let’s anchor on the user experience we’re improving. I’ve summarized the user outcomes you identified into two groups:
I would add a third which you didn’t name explicitly under “User outcomes”, but is worth mentioning:
Here are the individual proposals I read in this RFC, each mapped to the user experience it impacts:
Corrections and counterpointsI fact checked the claims in this RFC for accuracy. Claims related to Arrow, DuckDB, and ADBC checked out cleanly, but several claims about GX itself are inaccurate, and a few framings invite a counterpoint even where no fact is wrong.
The summary claims that GX implements and maintains “Expectation x execution engine x backend”, and contrasts it with a goal future state of “Expectation -> typed plan -> execution policy”. This overstates it; GX does not maintain a combination for each expectation, execution engine, and backend. Each metric is implemented once per ExecutionEngine — three primary engines today, plus two narrow per-backend subclasses (SQLite, Redshift) that override a handful of metrics. There are dialect-specific forks within the SQLAlchemy implementations, but they are the exception: only four of the 61 metric files with a SQLAlchemy implementation fork inline on dialect, and most dialect handling routes through two shared helpers. The proposal wouldn’t actually reduce this surface: remote and distributed execution still need a per-backend compiler for the typed plan, so the backend axis returns wherever validation doesn’t run in DuckDB. The prototype makes the same point in code: it adds a DuckDB implementation of every metric it supports, registered through the same decorators and registry as the primary engines. That's another engine column, not a collapse of the matrix.
On any horizon we can plan for, the opposite happens: the proposal adds another ExecutionEngine, a typed IR, source adapters, a replay layer, and an execution policy, and removes nothing. The OpaqueLegacyMetricNode escape hatch keeps the legacy path alive for as long as a single custom metric depends on it. Query-based metrics like the UnexpectedRowsExpectation and custom metrics mean OpaqueLegacyMetricNode will never go away; we’ll live with two versions of the system in perpetuity.
Overstated. Pandas file assets already expose reader-level projection — ParquetAsset accepts a columns argument that is passed through to the reader — so users can avoid materializing unused columns today. What GX lacks is projection derived automatically from the suite, plus row-group pruning. Those are real improvements, but they’re smaller than GX must materialize everything, and don’t need to be tied to an architectural refactor.
This is listed as a current recurring cost, but current GX has no one-shot streams to replay. The SQLAlchemy engine executes aggregates in the database and fetches a single fused row per domain; pandas batches are random access. One-shot streams only enter the picture once this proposal’s extraction path introduces them. Replay is a requirement of the new architecture, not a defect of the existing one — and the same goes for extraction planning, since today’s engines never extract the relation for metric computation.
Today’s strategy is limited, but it’s perfectly predictable: a pandas batch lives in RAM; SQL and Spark batches keep essentially nothing local. What this RFC actually offers is a new capability, bounded local execution with spill.
“Can” is doing a lot of work here. A new source on an existing engine touches only the datasource layer — that’s the shape of the Hugging Face proposal in #11987 — and a new SQL backend rides the shared SQLAlchemy implementation plus targeted dialect forks. The full four-layer cost shows up when introducing a new compute paradigm, which is exactly what this RFC does: the prototype’s first PR touches 138 files. I think the interesting question is "What compute paradigms can we add which maximize coverage over data sources users want to connect to?". The RFC correctly identifies both DuckDB and Arrow as potential targets that would expand GX's current reach, but the architectural changes aren't necessary to realize that value.
Half of this already exists. ValidationGraph deduplicates shared metrics across a suite at resolution time, and resolve_metric_bundle already fuses aggregate metrics into a single query per domain on both the SQLAlchemy and Spark engines. On existing engines, the incremental value of the logical plan is explain output plus capability negotiation — worthwhile, but a much smaller claim than a foundation for suite-level optimization, and again can be implemented within the existing architecture.
For the dominant warehouse workload there are no row objects to eliminate. Aggregate metrics execute in the warehouse as one fused query per domain and fetch back a single row. Rows are only materialized for evidence — bounded by partial_unexpected_count, default 20, and capped at 200 rows on the SQL and Spark engines even under COMPLETE — and for UnexpectedRowsExpectation. Extracting the relation over Arrow moves strictly more bytes than the current aggregate path; the RFC concedes as much in its SQLAlchemy paragraph, and the user-outcomes section should reflect it.
There’s an unstated flip side: under the Phase 2 hybrid policy, whether a suite runs in Snowflake or in DuckDB becomes a planner decision, and the two disagree on regex flavor, collation, and timezone handling. Today results are deterministic per configuration, because the datasource fixes the engine. Consistency across sources is being bought with a new placement-dependence across runs — a semantics regression unless the parity gates are airtight.
The Spark-specific implementations already exist and already work — that’s SparkDFExecutionEngine. A plan-to-Spark compiler is a rewrite of functioning support, framed as avoiding work that’s already done. It may still be worth doing for uniformity, but it should be accounted for as net-new work delivering near-zero new user capability. The prototype vs. the proposalI also read the three prototype PRs against the RFC’s architectural narrative. The prototype is real, substantial work, and it makes this discussion concrete — but on several central claims it is evidence against the narrative rather than for it.
In the prototype, Expectations do not compile into the plan. They flow through the existing registry and ValidationGraph unchanged, and the ValidationPlan is assembled after the fact from the per-engine metric partials that machinery produces (ValidationPlan.from_computations, inside the DuckDB engine’s resolve_metric_bundle). The plan sits below the graph as a typed replacement for SQL strings — not between the suite and the engines, where every architecture diagram in this RFC places it. That’s a useful improvement, but it’s a different architecture than the one proposed.
There is no compatibility lowering anywhere in the prototype stack. metric_execution_plan.py has no path from the plan back to MetricConfiguration or ValidationGraph, and no existing engine consumes the plan. The slice this RFC asks us to land first is the one piece of the design with no prototype evidence behind it.
The preflight is plan_suite_projection, which decides projection safety by checking expectation types against a hardcoded frozenset of eighteen type-name strings. That is string matching on names — the same mechanism this RFC lists as a deficiency of today’s ValidationGraph (“based heavily on metric-name strings”).
The direct cross-engine comparison is one 82-line test function, test_typed_duckdb_expectations_match_pandas_and_sqlalchemy; beyond it, the prototype enrolls a DuckDB datasource in the existing canonical-expectations battery, which exercises DuckDB against shared fixtures but never compares one engine’s output against another’s. You hedge the coverage correctly later (“does not establish full expectation parity”), but it’s worth noting as unproven. Evaluating the proposalsTo size the impact of each proposal, I checked it against the demand we can document — prior issues, discussions, and Discourse threads — and against the corrections above. 1. Typed, versioned IR. Nothing in our request history asks for what this layer provides on its own — it’s enabling infrastructure, and its two headline optimizations, suite-level dependency analysis and aggregate fusion, already exist in GX (ValidationGraph and resolve_metric_bundle). Its real consumers are proposals 2 and 7, and the prototype doesn’t yet demonstrate the Expectation-to-plan direction (see above). The design is attractive, but I don't yet understand the problem that this feature alone would solve. 2. Explain/analyze before validation. The transparency need is real but shows up obliquely: #8518 is at bottom a request to see and control the queries GX issues to BigQuery — the reporter had to pull the SQL GX generated themselves to show why it scanned roughly 250x more data than necessary. My guess is that this is high value for a few users, but most of it is deliverable today by logging the generated SQL and chosen policies, without a versioned IR. Modest impact, and a cheap version should come first. 3. DuckDB ExecutionEngine. The standout, with the longest paper trail: #10738 asks for DuckDB support directly and #6932 asks how to use DuckDB with GX; #11579 shows users already forcing duckdb-engine through the SQLAlchemy engine and hitting edge cases; #12089 asks for SQL expectations over in-memory dataframes, as did a Discourse user; and the performance pain it addresses is documented — #6290 describes pandas running out of memory on CSVs of up to a few hundred million rows, with a Spark fallback taking 50 minutes to run 140 expectations over a 100-million-row sample. High end-user impact, and — as the prototype itself demonstrates — deliverable standalone inside the existing engine architecture, with #12084 already derisking it. Query metrics, the specific #12089 ask, should be in scope from the start. 4. Arrow-native batch data. This is the “connect to more data” workhorse. A steady, years-long drumbeat asks for columnar dataframe support: Polars (#5503, #8023, #9649, #10144, and a WIP PolarsExecutionEngine in #5914), Narwhals (#10702), Dask (#5816), and Hugging Face datasets (#11987), which are Arrow-backed natively. An Arrow contract at the Batch seam turns each of these from full ExecutionEngine implementations (touching 100+ files) into thin adapters. After the corrections, the performance half of its pitch is the weaker half — the warehouse path has no row conversions to save — so I’d justify this on reach, not speed. Moderate impact, high leverage for community contribution. 5. Arrow replay policy. Demand-adjacent rather than demanded. Its concrete near-term case is streaming sources we currently have to scope out — HF IterableDataset was excluded from #11987’s v1 precisely because metrics assume multi-pass access — and the long-tail streaming asks (#4849, #7337, Discourse) mostly want continuous validation, which replay doesn’t provide; it gives bounded-window validation of a stream. Per the corrections, it fixes no current defect — it’s a requirement the Arrow path creates. I see this mostly as a good example of what Arrow support unlocks. We may want to eventually implement it, but only after clear community signal. 6. ADBC driver adoption. No documented demand at all — no issue, discussion, or Discourse thread I can find mentions ADBC — and every source it would connect already has a supported path. With the “fewer Python object conversions” claim corrected, the remaining case is large evidence sets: the most-repeated complaint in our support history is truncated failing-row results (#11745, where COMPLETE is capped by a hardcoded limit, and #11527, among roughly twenty reports), and columnar transport is the right substrate for moving complete failing-row sets — but those caps are a product decision, not a transport bottleneck, so ADBC doesn’t unlock this by itself. Treat ADBC as a per-source implementation choice behind proposal 4’s contract, qualified when a concrete source needs it, rather than a program to adopt. 7. Execution placement policy. The most speculative item. The documented warehouse-cost pain points the other direction: #8518 wants GX to generate batch SQL that lets BigQuery prune partitions — better queries in the warehouse, not data moved out of it. The extract-once-replay-locally win applies to repeated validation of bounded relations, which the RFC itself concedes is workload-dependent, and today’s placement is already right for the common case, since aggregates run where the data lives. Placement choice is also exactly what puts “consistent validation semantics” at risk (see above). Defer until the DuckDB engine exists and real workloads demonstrate the need. PositionI’m leaning towards rejecting this RFC as proposed, barring a groundswell of community support. I think the architectural redesign is interesting but too expensive, and the user value delivered is real but too little to justify the high cost. We should first try to deliver the goal user experience within GX architecture, and only break from it when either a) there’s no way to do it within GX’s current design, or b) the cost of working within the current design is greater than the cost of migration. The typed IR, ADBC as a program, stream replay, and the execution placement work contribute an enormous amount of scope to this proposal while (in my estimation) adding relatively little to delivered user value. There are three proposals within this RFC that have real community demand behind them, and I’d like to see each move forward as its own focused RFC:
The thread will remain open at least another week so other GX users have a chance to weigh in directly. Thanks for going deep on this @dataders, looking forward to hearing your thoughts! And if you're up for it, a narrowed RFC for a DuckDB ExecutionEngine would be very welcome. |
|
thanks again for this proposal, @dataders! I'm closing this rfc as declined for the reasons stating above. That being said, I do think this is a really interesting proposal, and I'm looking forward to see the impact it has on GX going forward, as some of these ideas make their way into the architecture via other proposals. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Summary
Great Expectations (GX) should add an opt-in columnar validation path built around four small, separable contracts:
ValidationPlandescribing what an Expectation needs;The first supported sources would be Snowflake, local Parquet and CSV files, and in-memory Arrow-compatible data. Existing pandas, Spark, and SQLAlchemy execution paths would remain available during migration. Existing Expectation suites and validation-result schemas would not change.
The long-term goal is not merely to add another datasource. It is to reduce the number of combinations GX must implement and maintain:
The first landing slice is the engine-neutral logical plan plus compatibility lowering to current GX engines; it does not require Arrow, ADBC, or DuckDB.
This RFC asks whether GX should adopt these internal boundaries and continue the prototype toward an experimental, parity-gated datasource.
Prototype stack:
The prototype is evidence for discussion, not production-ready support.
A short primer
Apache Arrow
Arrow is a standard columnar representation for typed data. “Columnar” means values from a column are stored together instead of turning every result row into a collection of Python objects. Analytical engines can often exchange or scan Arrow data with fewer conversions and allocations.
Arrow is not a database, query engine, or network protocol. An Arrow-native path also does not guarantee that no copy occurs. A database driver still decodes network responses into buffers, and persisting or transforming a stream can allocate or write data. The useful promise is narrower: GX can avoid unnecessary Python row objects and pandas materialization, and can measure copies at explicit boundaries.
ADBC
Arrow Database Connectivity (ADBC) is a standard API for connecting to a database, executing a query, and receiving Arrow data. It serves a role similar to JDBC or ODBC, but its shared result boundary is Arrow rather than rows.
ADBC is not a SQL dialect and does not decide which computations should run in a warehouse. Each source still needs SQL generation, authentication, type qualification, cancellation, and operational testing.
DuckDB
DuckDB is an embedded analytical SQL engine. It can scan Parquet, CSV, and Arrow data directly, executes queries with vectorized operators, supports memory limits and disk spill, and runs inside the GX process without a separate server.
This proposal uses DuckDB as the first common execution target so GX does not need a new Python implementation of every metric for every source.
Arrow IPC and replay
Database result streams are commonly forward-only: once consumed, they cannot simply be scanned again. A GX validation suite may need several metric queries over the same batch.
Arrow IPC is a standardized file/stream representation for Arrow data. GX can consume a source stream once, write it to a controlled temporary IPC file, memory-map that file, and let DuckDB scan it repeatedly. IPC has a write cost and is not always faster. It is one policy option when bounded memory or replayability matters.
Typed ValidationPlan
A
ValidationPlanis an internal, serializable description of validation intent: required columns, row domains, typed predicates, aggregates, success conditions, and requested diagnostics. It separates “what does this Expectation mean?” from “how does pandas, Snowflake, Spark, or DuckDB happen to implement it?”It also gives GX a place to explain an execution before running it, reject unsupported operations early, combine compatible metrics, and choose safe source pushdown.
Relationship to GX's existing validation planning
GX already has important planning machinery. This RFC does not claim that validation execution is currently unplanned.
ValidationDefinitionbinds a suite to a batch definition.ExpectationSuiteandExpectationConfigurationdescribe user intent.MetricConfigurationdescribes individual metric requests and their domains/value arguments.ValidationGraphbuilds the dependency graph required to resolve those metrics.ValidationGraphis therefore the conceptual starting point for this proposal, not something to discard or duplicate indefinitely.The migration deliberately separates three independently risky changes:
These changes can be reviewed, tested, and adopted independently. A decision to improve GX's logical planning does not require approval of ADBC; adopting an Arrow transport does not require every validation to execute in DuckDB.
The distinction is that today's
ValidationGraphis:ExecutionEngine;IDDictkwargs;Those properties are reasonable consequences of how GX evolved. Existing metric providers encode years of behavior and extension points. The proposal is to formalize an engine-neutral layer above that machinery, then migrate incrementally rather than replace it in one step.
The first migration step does not require Arrow, ADBC, or DuckDB
The first independently useful deliverable should be an internal
LogicalValidationPlanthat can lower back into the current runtime unchanged:flowchart LR SUITE[Expectation suite] --> LOGICAL[Engine-neutral LogicalValidationPlan] LOGICAL --> LEGACY[Compatibility lowering] LEGACY --> GRAPH[MetricConfiguration + ValidationGraph] GRAPH --> CURRENT[Current SQLAlchemy / pandas / Spark engines]This first step would formalize typed domains, expressions where available, metric dependencies, success conditions, evidence requirements, required columns, and shared row conditions. It would also support opaque legacy nodes for custom or not-yet-typed metrics.
Existing execution behavior and validation-result schemas would remain unchanged. Arrow transport, ADBC drivers, replay policy, and DuckDB compilation would arrive only after semantic parity and compatibility lowering are demonstrated.
Later physical targets can consume the same logical plan:
flowchart LR LOGICAL[LogicalValidationPlan] --> POLICY{Physical planning} POLICY --> EXISTING[Existing ValidationGraph compatibility path] POLICY --> DUCK[DuckDB physical plan] POLICY --> WAREHOUSE[Warehouse physical plan] POLICY --> SPARK[Spark physical plan] LOGICAL --> EXTRACT[Safe extraction requirements] EXTRACT --> ADBC[ADBC adapter] EXTRACT --> VENDOR[Vendor Arrow adapter] EXTRACT --> FILES[Local file scan]In that design:
This makes Arrow, ADBC, and DuckDB replaceable implementations behind stable architectural seams. It also creates value before any of them are introduced: deterministic explain output, capability negotiation, suite-level dependency analysis, and a foundation for query fusion across existing engines.
Compatibility with custom Expectations
The logical plan must not require every existing or custom metric to become typed immediately. It should support both:
An opaque node must preserve current behavior and report why optimization was unavailable. This escape hatch allows incremental adoption without breaking GX's extension ecosystem.
Problem
GX has mature Expectation semantics, but datasource access, SQL construction, data movement, metric execution, and result diagnostics are coupled across several execution engines.
That creates recurring costs:
SQLAlchemy itself is not inherently row-wise. It is a SQL toolkit and DBAPI abstraction, and GX's SQLAlchemy engines can efficiently execute aggregate SQL in a warehouse. The limitation is that GX's shared contracts do not represent Arrow-native results or replay independently of SQLAlchemy and pandas.
User outcomes
Faster, lower-memory local validation
DuckDB can scan only the required columns and row groups from Parquet and apply CSV projection/filtering without first constructing a full pandas DataFrame.
Fewer Python object conversions
Snowflake or another warehouse can return Arrow batches directly. GX can preserve columnar data through replay and DuckDB scanning instead of building Python row objects and then a DataFrame.
Predictable resource behavior
GX can select or expose three policies:
Each policy can report rows, batches, bytes, spill, temporary storage, cleanup, and fallback reason.
Consistent validation semantics
The same typed plan and DuckDB compiler can validate data from Snowflake, Parquet, CSV, or in-memory Arrow. Backend-specific behavior becomes an explicit capability or parity decision instead of an accidental implementation detail.
Explainable execution
Before validation, GX could show:
Potentially lower warehouse usage
A filtered relation can be extracted once and reused for many local metrics instead of repeatedly querying the warehouse. This is workload-dependent, not a universal cost reduction. Large scans that produce tiny aggregates should normally remain in the warehouse.
Proposed architecture
flowchart LR SUITE[Expectation suite] --> PLAN[Typed ValidationPlan] PLAN --> POLICY{Execution policy} POLICY -->|local or bounded extract| REQUEST[Safe extraction request] REQUEST --> SOURCE[Source adapter] SOURCE --> ARROW[Arrow RecordBatch stream] ARROW --> REPLAY{Replay policy} REPLAY --> MEM[Arrow memory] REPLAY --> IPC[Arrow IPC + memory map] REPLAY --> TABLE[DuckDB temporary table] MEM --> DUCK[DuckDB executor] IPC --> DUCK TABLE --> DUCK POLICY -->|large scan / small result| REMOTE[Qualified remote executor] DUCK --> RESULT[GX ValidationResult] REMOTE --> RESULTThe initial Snowflake prototype takes the local/bounded-extract path. A production design must retain remote execution for workloads where moving the relation would be slower, more expensive, or prohibited.
Contract 1: validation semantics
The typed plan contains domains, typed expressions, metrics, success assertions, and evidence requests. It must be JSON-round-trippable and versioned internally. Existing Expectations compile into it; users do not rewrite suites.
Contract 2: source capabilities
A source advertises whether it can provide schema, projection, structured row filtering, limits, query cancellation, partitioned reads, and Arrow streaming. The planner requests only capabilities that preserve validation semantics.
An Expectation's failure predicate must never be used as an extraction filter. Doing so would change denominators, unexpected percentages, and diagnostics. A row condition may be pushed down only when it defines the shared validation domain.
Contract 3: replay and ownership
The object that makes a one-shot stream replayable owns its Arrow buffers, memory maps, temporary files, DuckDB registrations, and cleanup. Limits and cancellation apply while the stream is consumed, not after a full dataset has already been accumulated.
Contract 4: execution
DuckDB is the first common target for local execution. A separate remote-executor capability can later compile safe plan fragments for a warehouse or distributed engine. Transport and execution remain independent: using ADBC does not require pulling every row locally.
How the prototype extends to other sources
The important abstraction is
ArrowBatchSource, not “every database must use the same driver.” ADBC is preferred when its driver clears GX's correctness and operational gates. A vendor Arrow API or distributed executor can implement the same contract when it is a better fit.Current ADBC documentation lists drivers for BigQuery, Databricks, Snowflake, PostgreSQL, Flight SQL, DataFusion, and other systems. Driver availability is not enough to declare GX support: every driver still needs pinned artifacts, platform CI, authentication coverage, cancellation tests, schema fidelity tests, and workload benchmarks.
BigQuery
BigQuery can fit the same architecture in two ways:
The Storage API may expose multiple read streams, so the source contract should eventually permit partitioned Arrow streams rather than requiring one Python reader. Projection and row restrictions can be pushed into the read session when semantics permit.
For a large table and a small aggregate result, the planner should compile the metric fragment to BigQuery instead of reading the table locally. The decision must consider bytes billed, estimated result size, data location, nested/repeated fields, and requested unexpected-row evidence.
Databricks
Databricks can also have an ADBC implementation and a vendor parity path. The Databricks SQL Connector exposes Arrow fetch methods, and CloudFetch can retrieve large results directly from cloud object storage when PyArrow support is installed.
The adapter would normalize either transport into owned Arrow batches. A remote compiler could keep scan-heavy aggregates in a SQL warehouse. Qualification must cover Unity Catalog identifiers and permissions, OAuth/service principals, SQL warehouse lifecycle, CloudFetch networking, Delta-specific types, cancellation, and complex/nested values.
Apache Spark
Spark is not merely another remote SQL cursor. Its value is distributed execution, and collecting a large Spark DataFrame into the GX process would defeat that value.
Spark 4 provides
DataFrame.toArrow(), but Spark's documentation warns that it collects the complete result into driver memory and should only be used for small results. Therefore the intended Spark architecture is:DuckDB remains useful for small bounded Spark samples, local debugging, or processing already-materialized Arrow results. It should not replace Spark for an unbounded distributed scan.
This implies a second execution target after the DuckDB plan is proven: a Spark compiler for the same typed plan, not a Spark-specific reimplementation of each Expectation.
Additional databases
ADBC drivers and Flight SQL can reduce the transport work needed for PostgreSQL-compatible systems, ClickHouse, DataFusion, and other analytical databases. Each new source would still be opt-in and parity-gated. The architecture makes expansion cheaper; it does not turn driver presence into automatic support.
Execution policy
No single placement is optimal. A later planner should choose between local and remote execution using source capabilities and conservative rules:
Phase 1 does not need a sophisticated cost-based optimizer. It needs safe explicit policies, useful estimates, and an explain output. Automatic placement should remain experimental until telemetry and benchmarks support it.
Prototype evidence
The current stack demonstrates:
RecordBatchReaderretrieval without pandas or SQLAlchemy in the new path;This evidence does not establish full expectation parity, production packaging, multi-platform reliability, external-browser automation, or large-result performance.
Migration plan
Phase 0: agree on internal contracts
Phase 1: productionize the Snowflake/local vertical slice
Phase 2: add conservative hybrid execution
Phase 3: prove a second warehouse
Phase 4: prove a distributed executor
Phase 5: migration and simplification
Correctness and release gates
No adapter or executor becomes recommended until it passes:
Security and privacy
Alternatives considered
Continue only with pandas and SQLAlchemy
This has the lowest migration risk and preserves mature backend coverage. It does not provide a common Arrow boundary, explicit replay, or one plan that multiple engines can compile.
Use vendor connectors directly
Vendor APIs are valuable parity controls and may be the best adapter for a particular source. Using only vendor-specific contracts would duplicate connection, schema, streaming, ownership, and observability logic across GX.
Standardize on Ibis
Ibis is a strong relational expression system and could help compile source-side queries. It does not by itself define GX success conditions, diagnostic evidence, stream ownership, or replay. The prototype does not require it; this remains a separate design choice.
Use DataFusion or Substrait as the plan
DataFusion logical plans and Substrait could become useful physical-plan or interchange targets. Neither directly represents GX-specific success assertions and result diagnostics. The initial typed plan should stay small and domain-specific while retaining a future lowering boundary.
Always execute in DuckDB
Simple, but wrong for large warehouse/distributed scans, data-residency restrictions, and small aggregate results. DuckDB should be the common local target, not mandatory placement for every workload.
Risks
Non-goals
Questions for reviewers
ArrowBatchSourceremain private until two independent source implementations ship?Decision requested
Approve continued development of an additive, experimental columnar validation path based on:
ValidationPlan;Acceptance would approve the architectural direction and bounded productionization work, not make the prototype a default or declare every listed datasource supported.
References
DataFrame.toArrowDisclosure
No affiliation with ADBC Driver Foundry, Columnar, DuckDB Labs, Snowflake, Databricks, Google Cloud, or the Apache Arrow project is being asserted in this proposal. Driver stewardship and distribution are described so reviewers can evaluate the dependency boundary; GX would independently qualify any adopted artifact.
All reactions