Skip to content

feat: cover dictionary-encoded bind parameter columns - #256

Merged
lidavidm merged 3 commits into
adbc-drivers:mainfrom
fornwall:test/bind-dictionary
Jul 22, 2026
Merged

feat: cover dictionary-encoded bind parameter columns#256
lidavidm merged 3 commits into
adbc-drivers:mainfrom
fornwall:test/bind-dictionary

Conversation

@fornwall

@fornwall fornwall commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Add a test for binds a dictionary-encoded column (what pandas often produce).

Dictionary encoding is an encoding of the same logical values, not a different logical type (Arrow columnar format, Dictionary-encoded Layout), so a driver that binds plain string columns should accept the dictionary-encoded equivalent, decoding it if the database has no native counterpart:

https://arrow.apache.org/docs/format/Columnar.html#dictionary-encoded-layout

Verified against existing drivers: sqlite, postgres, mysql and flightsql (for the example server it needed a fix: apache/arrow-go#936).

Context: Missed this when implementing a new driver, so having this test would have helped.

Add a test for binds a dictionary-encoded string column (what pandas
often produce).

Dictionary encoding is an encoding of the same logical values, not a
different logical type (Arrow columnar format, Dictionary-encoded
Layout), so a driver that binds plain string columns should accept the
dictionary-encoded equivalent, decoding it if the database has no
native counterpart:

> https://arrow.apache.org/docs/format/Columnar.html#dictionary-encoded-layout

Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net>
@fornwall
fornwall force-pushed the test/bind-dictionary branch from 5f79835 to e980017 Compare July 14, 2026 09:53
@fornwall fornwall changed the title test(statement): cover dictionary-encoded bind parameter columns feat: cover dictionary-encoded bind parameter columns Jul 14, 2026

@lidavidm lidavidm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be just a .txtcase or derive from an existing .txtcase (e.g. see the ingest tests where ingest/string is used with some Python code to test other things while allowing some flexibility to override the test data)

@fornwall

Copy link
Copy Markdown
Contributor Author

I think this should be just a .txtcase or derive from an existing .txtcase (e.g. see the ingest tests where ingest/string is used with some Python code to test other things while allowing some flexibility to override the test data)

👍 Made the transition to .txtcase in abe8932, adding the necessary dictionary support to the test machinery. I didn't see a clean fit for deriving from an existing .txtcase, but let me know if you think otherwise.

@fornwall
fornwall requested a review from lidavidm July 15, 2026 07:34
@lidavidm

Copy link
Copy Markdown
Contributor

Sorry, I should've explained more...This is what I mean:

def test_replace_schema(
self,
driver: model.DriverQuirks,
conn: adbc_driver_manager.dbapi.Connection,
query: Query,
) -> None:
subquery = query.query
assert isinstance(subquery, model.IngestQuery)
data = subquery.input()
expected = subquery.expected()
# Create a table in the default schema
default_data = data
# Create different data for the secondary schema
data = data.slice(0, 2)
data2 = data.slice(0, 1)

It's "parametrized" but the parameterization only ever uses a single test case. The test itself then transforms the data before doing something with it. That way it effectively inherits whatever customization was done for that test case without requiring a separate test case.

@fornwall

Copy link
Copy Markdown
Contributor Author

Thanks! I transitioned to parameterization in 2ac53ff - does that match what you meant?

I found that support for dictionary encoded types other than strings is lacking in at least the sqlite/postgres driver (looking into a PR for that), so for now I'm just checking string & large_string.

@lidavidm
lidavidm merged commit 1c20ca4 into adbc-drivers:main Jul 22, 2026
3 checks passed
fornwall added a commit to fornwall/adbc-spanner that referenced this pull request Jul 26, 2026
…0ca4b (UP-9) (#394)

Moves both validation-suite dependency families to their current upstream
`main`, and takes the two behaviour changes that came with them.

**apache/arrow-adbc `198f39a9` -> `3b3f123a`** (all three crates plus the C++
suite's ARROW_ADBC_TAG, previously `65957bba`, in lockstep). The one behaviour
change in range is apache/arrow-adbc#4510, `InfoCode::Other(u32)`: the FFI
exporter used to reject the *whole* `GetInfo` call with "Unknown info code"
before the driver ran, so an XDBC-range ([500, 1000)) or vendor-specific
(>= 10000) code could never be requested even though "fetch all" could return
one. It now forwards them, leaving "ignore unrecognized codes" to the driver as
adbc.h states. No driver change was needed — `info::build` already filters
explicit requests to `REPORTED` and omits the rest — but the contract was
unobservable through the C ABI until now, so pin it from both sides: a unit test
at the trait level and an assertion through the driver manager in
`conformance_via_driver_manager`. Closes UP-9 in REVIEW.md.

apache/arrow-adbc#4534 (the other PR in range worth checking) was already
adopted in #343: SqlPrepareUpdate/Stream are un-excluded and gate-enforced with
RewriteSql entries pinning the new ORDER BY defaults. `c/validation` has had no
commits since, so the tag bump adds no cases, and none of the four survivors in
EXCLUDED are reachable by RewriteSql — three are Arrow type-mapping gaps
(Duration/Interval/UInt64) and one is an arrow-rs FFI stream limitation
(SqlQueryCancel cannot emit ECANCELED).

**adbc-drivers/validation `dbc6857f` -> `1c20ca4b`**, which carries
adbc-drivers/validation#256: `test_query_bind_dictionary` re-binds the
`type/bind/string` and `type/bind/large_string` cases dictionary-encoded — the
layout a pandas categorical produces. Both pass unchanged and are gate-enforced
(not in the skip baseline, which needed no edit): dictionary encoding is an
encoding of the same logical values, not a different logical type, and
`bind::cell_value` already decodes the key at each row back through the same
mapping as the plain column.

Verified: `cargo fmt`/`clippy -D warnings` clean; `with-emulator.sh cargo test`
green (321 unit + 53 integration + 44 mock + 6 resilience + 1 doctest);
`run-adbc-validation.sh` 89 passed / 7 skipped with the gate, expected-failure
and stale guards all OK; `run-foundry-validation.sh` 188 passed / 65 skipped
with FOUNDRY_VALIDATION_REQUIRE_PASSES and the skip-baseline guard on.

Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants