scanColumn wrappers forward WHERE and the applied-hint flags (LLP 0098) - #304
Conversation
Adopt the squirreling 0.15 ScanColumnResults contract through the kernel's scanColumn wrappers so filtered single-column aggregates stay streaming: icebird prunes files and pushes the predicate to hyparquet instead of the engine materializing every row. - scan-column.js: normalizeScanColumn shim for legacy bare-iterable sources - union-source.js: where forwarded per partition; appliedWhere is the AND across partitions; limit/offset never coexist with a forwarded where - sql.js withHeapBudget: flags pass through untouched - ai-gateway withSchemaColumns: strips a predicate on a declared but physically absent column, reports appliedWhere false - pins: icebird 0.8.14, squirreling 0.15.0 - LLP 0098 records the decision; LLP 0055 points to it The walkthrough_to_first_query smoke failure predates this change (fails identically on master with the old pins).
Summarysquirreling 0.15.0 extended the Correctness - verified against the engine and source
Tests
Conventions
RiskLow, with one merge-timing constraint: master breaks if icebird 0.8.14 is installed without these wrappers (old code crashes on the new return shape). The pin bump and wrapper changes land atomically here, so merge promptly and don't let a partial dep bump land ahead of it. Requested changesNone. The one item raised during review (document the eager-probe tradeoff and the defer-IO-to- |
Filtered aggregates (e.g.
COUNT(*) WHERE ...) fell off squirreling's streaming scanColumn fast path because none of the wrappers forwarded thewherepredicate. The engine then materialized every row and filtered per-row: 30-47s on 489k rows, and stacked filtered subqueries crashed the remote daemon (gateway 504, daemon 502).squirreling 0.15.0 extended the scanColumn contract with
whereand a flaggedScanColumnResultsreturn shape, and icebird 0.8.14 implements pushdown into the parquet read. This PR adopts that contract in hypaware's wrapper stack:normalizeScanColumnshim (src/core/query/scan-column.js) adapts legacy bare-iterable sources to the flagged shape with honest flags.unionSourcesforwardswhereper partition when the partition can apply it, mergesappliedWhereas the AND across partitions, and never combines limit/offset with an unresolved where.withSchemaColumnsstrips predicates naming declared-but-physically-absent columns (parquet throws on missing filter columns) and reportsappliedWhere: falseso the engine re-filters.withHeapBudgetpasses options and flags through untouched.Measured locally on an 18.5k-row cache: filtered COUNT 0.375s (vs 30-47s on the old path), three stacked filtered subquery COUNTs 0.616s.
The pre-existing
walkthrough_to_first_querysmoke failure (ai_gateway_messages never lands in the cache) predates this change, verified by A/B with old pins.