Skip to content

v0.36.2 — a BOOLEAN column is a predicate on its own

Choose a tag to compare

@rustyconover rustyconover released this 20 Sep 21:13
· 172 commits to main since this release

Fix: a BOOLEAN column is a predicate on its own

WHERE flag and WHERE NOT flag were rejected at decode with
FilterDeserializationError: predicate root must resolve to BOOLEAN. DuckDB
pushes such a predicate down as a bare column_ref rather than rewriting it to
flag = true, and the schema admits that — coreExpression lists columnRef
first — so the decoder was stricter than its own published contract, and the
idiomatic form of the commonest boolean predicate there is failed for every
worker on filter encoding v2.

Three things, each independently wrong:

  • FilterParser._is_boolean classified by node kind instead of resolved
    type.
    The two agree for every node whose type is structurally fixed — a
    comparison is BOOLEAN whatever its operands — which is why the enumeration
    looked sufficient. It is not: column_ref, field_ref, cast and
    arithmetic are BOOLEAN or not depending on a type resolved from elsewhere,
    and all four were refused. The parser had already resolved the column
    against the bind output schema and stored the type on the node, and
    expression_type() — written in the same commit, used one branch away to
    type-check field_ref — reads it correctly. It now calls that.

  • A bare boolean column rendered no SQL. Left as a generic v2 expression it
    became a V2ExpressionFilter, which evaluates but does not render, so a
    SQL-backed worker silently lost pushdown on it — and on a conjunction
    containing one, for the whole conjunction. _v2_filter_view now projects
    flag and NOT flag onto the legacy equality leaf. The rewrite is exact
    under three-valued logic: WHERE flag keeps only TRUE and so does
    flag = true; WHERE NOT flag keeps only FALSE and so does flag = false.
    An unresolved or non-boolean column is left alone rather than guessed at.

  • to_sql() raised a bare ValueError: Unknown filter type. It now raises
    FilterSQLUnsupportedError naming the node and pointing at apply(). It
    still raises rather than skipping, deliberately: DuckDB does not re-apply
    a predicate it pushed into a table function — measured, not assumed, at 3255
    rows returned where the predicate admits 1874 — so a quietly dropped filter
    is a wrong answer, not a slow one.

Tests

Nothing covered this, and the obvious place could not have. The conformance
corpus had no bare boolean case, and the extension's filter_pushdown/booleans.test
— added specifically to close a boolean coverage gap — writes every predicate as
b = true, b != true or b IN (...), never WHERE b.

  • bool_filter_echo (new fixture): a table function with a nullable BOOLEAN
    column, echoing the SQL rendering of what was pushed. filter_echo has no
    boolean column and the table-in-out echo path is never handed a bare boolean
    column as a pushed predicate, so cases written there pass whether or not the
    worker can decode one. It takes a table function with a boolean column to see
    the shape at all.
  • structural.boolean-column-predicate in the filter-v2 conformance corpus,
    for a bare column_ref root and for not over one. That is the cross-port
    contract: every port's decoder is free to have the same gap, and none had a
    case that would say so.
  • Unit tests for both forms, for a conjunction containing one, for the SQL they
    must render, for the refusal of an unresolved or non-boolean column, and for
    the no-silent-drop guarantee.

Companion tests land in Query-farm/vgi as
test/sql/integration/filter_pushdown/boolean_column_predicate.test.