Skip to content

scanColumn wrappers forward WHERE and the applied-hint flags (LLP 0098) - #304

Merged
platypii merged 2 commits into
masterfrom
scancolumn-where-pushdown
Jul 11, 2026
Merged

scanColumn wrappers forward WHERE and the applied-hint flags (LLP 0098)#304
platypii merged 2 commits into
masterfrom
scancolumn-where-pushdown

Conversation

@platypii

Copy link
Copy Markdown
Contributor

Filtered aggregates (e.g. COUNT(*) WHERE ...) fell off squirreling's streaming scanColumn fast path because none of the wrappers forwarded the where predicate. The engine then materialized every row and filtered per-row: 30-47s on 489k rows, and stacked filtered subqueries crashed the remote daemon (gateway 504, daemon 502).

squirreling 0.15.0 extended the scanColumn contract with where and a flagged ScanColumnResults return shape, and icebird 0.8.14 implements pushdown into the parquet read. This PR adopts that contract in hypaware's wrapper stack:

  • New shared normalizeScanColumn shim (src/core/query/scan-column.js) adapts legacy bare-iterable sources to the flagged shape with honest flags.
  • unionSources forwards where per partition when the partition can apply it, merges appliedWhere as the AND across partitions, and never combines limit/offset with an unresolved where.
  • ai-gateway withSchemaColumns strips predicates naming declared-but-physically-absent columns (parquet throws on missing filter columns) and reports appliedWhere: false so the engine re-filters.
  • withHeapBudget passes options and flags through untouched.
  • Pins bumped to icebird 0.8.14 and squirreling 0.15.0. Note master breaks if icebird 0.8.14 is installed without this change (old wrappers crash on the new return shape), so this should merge promptly.
  • Design rationale in LLP 0098; tests cover union flag merging, real-parquet pushdown, and flag propagation through the budget wrapper.

Measured locally on an 18.5k-row cache: filtered COUNT 0.375s (vs 30-47s on the old path), three stacked filtered subquery COUNTs 0.616s.

The pre-existing walkthrough_to_first_query smoke failure (ai_gateway_messages never lands in the cache) predates this change, verified by A/B with old pins.

platypii added 2 commits July 11, 2026 15:17
Adopt the squirreling 0.15 ScanColumnResults contract through the kernel's
scanColumn wrappers so filtered single-column aggregates stay streaming:
icebird prunes files and pushes the predicate to hyparquet instead of the
engine materializing every row.

- scan-column.js: normalizeScanColumn shim for legacy bare-iterable sources
- union-source.js: where forwarded per partition; appliedWhere is the AND
  across partitions; limit/offset never coexist with a forwarded where
- sql.js withHeapBudget: flags pass through untouched
- ai-gateway withSchemaColumns: strips a predicate on a declared but
  physically absent column, reports appliedWhere false
- pins: icebird 0.8.14, squirreling 0.15.0
- LLP 0098 records the decision; LLP 0055 points to it

The walkthrough_to_first_query smoke failure predates this change (fails
identically on master with the old pins).
@platypii

Copy link
Copy Markdown
Contributor Author

Summary

squirreling 0.15.0 extended the scanColumn contract with a where predicate and a flagged ScanColumnResults return shape; icebird 0.8.14 implements pushdown into the parquet read. This PR adopts that contract across hypaware's wrapper stack so filtered aggregates (COUNT(*) WHERE ...) stay on the streaming fast path instead of falling back to per-row materialization - the path that took 30-47s on 489k rows and crashed the remote daemon under stacked filtered subqueries. Measured result: 0.375s for a filtered COUNT, 0.616s for three stacked filtered subqueries.

Correctness - verified against the engine and source

  • Flag propagation is sound end to end. The union's eager probe reads appliedWhere before consuming chunks; icebird computes both flags synchronously from already-loaded manifest entries (icebergDataSource.js:227-250), so the merged AND is always valid. normalizeScanColumn mirrors squirreling's own boundary shim exactly.
  • All four wrapper layers are covered: storage (storage.js:337, verbatim passthrough - correctly untouched), unionSources, withHeapBudget, withSchemaColumns. No other non-test site iterates the old bare-iterable shape.
  • The slice-safety invariant holds everywhere. limit/offset never coexist with an unresolved where: the union reports appliedLimitOffset: false in its where-branch, and withSchemaColumns strips limit/offset along with a predicate on a declared-but-absent column. This matches the engine's own conservatism (execute.js:286-288).
  • Degradation is graceful, never catastrophic. When the union reports appliedWhere: false (schema-drifted partition, legacy source), the engine reuses the probed column stream and re-filters per value (aggregates.js:334-335 -> filterRows) - the middle tier, never the row-materialization path this PR exists to kill.
  • The eager probe is the right design. With a where, all partitions are probed up front - required because merged flags must be synchronous. The cost is O(partitions) of manifest CPU, no IO; the lazy alternative (pessimistic flags) would trade that for per-value re-filtering scaling with row count. The tradeoff and the untyped convention it rests on (scanColumn() cheap, IO deferred to chunks()) are now recorded in LLP 0098's Consequences (bbb5b02).

Tests

  • All 32 tests in the three touched files pass. Coverage hits the right seams: union flag merging (all-applied, drifted-partition, legacy-partition), real-parquet pushdown including the absent-filter-column strip, and the budget wrapper via a deliberately lying source - the only observable probe for a dropped flag, since re-filtering matched values is idempotent. Good instinct.
  • The two npm test failures are environmental (gpg: signing failed: No secret key in untouched git-hook tests), not from this diff.

Conventions

  • No semicolons, no em dashes, @import type imports at top - all consistent.
  • @ref LLP 0098#wrapper-duties / #union-flags anchors resolve; refs are honest and non-mechanical. The retained @ref LLP 0055 on withSchemaColumns still applies. LLP 0055 correctly cross-links forward to 0098.

Risk

Low, with one merge-timing constraint: master breaks if icebird 0.8.14 is installed without these wrappers (old code crashes on the new return shape). The pin bump and wrapper changes land atomically here, so merge promptly and don't let a partial dep bump land ahead of it.

Requested changes

None. The one item raised during review (document the eager-probe tradeoff and the defer-IO-to-chunks() expectation for plugin authors) is already addressed in bbb5b02.

@platypii
platypii merged commit 6bd85c4 into master Jul 11, 2026
4 checks passed
@platypii
platypii deleted the scancolumn-where-pushdown branch July 11, 2026 22:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant