feat: cover dictionary-encoded bind parameter columns - #256
Conversation
Add a test for binds a dictionary-encoded string column (what pandas often produce). Dictionary encoding is an encoding of the same logical values, not a different logical type (Arrow columnar format, Dictionary-encoded Layout), so a driver that binds plain string columns should accept the dictionary-encoded equivalent, decoding it if the database has no native counterpart: > https://arrow.apache.org/docs/format/Columnar.html#dictionary-encoded-layout Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net>
5f79835 to
e980017
Compare
👍 Made the transition to .txtcase in abe8932, adding the necessary dictionary support to the test machinery. I didn't see a clean fit for deriving from an existing .txtcase, but let me know if you think otherwise. |
|
Sorry, I should've explained more...This is what I mean: validation/adbc_drivers_validation/tests/ingest.py Lines 628 to 642 in 94f8332 It's "parametrized" but the parameterization only ever uses a single test case. The test itself then transforms the data before doing something with it. That way it effectively inherits whatever customization was done for that test case without requiring a separate test case. |
|
Thanks! I transitioned to parameterization in 2ac53ff - does that match what you meant? I found that support for dictionary encoded types other than strings is lacking in at least the sqlite/postgres driver (looking into a PR for that), so for now I'm just checking |
…0ca4b (UP-9) (#394) Moves both validation-suite dependency families to their current upstream `main`, and takes the two behaviour changes that came with them. **apache/arrow-adbc `198f39a9` -> `3b3f123a`** (all three crates plus the C++ suite's ARROW_ADBC_TAG, previously `65957bba`, in lockstep). The one behaviour change in range is apache/arrow-adbc#4510, `InfoCode::Other(u32)`: the FFI exporter used to reject the *whole* `GetInfo` call with "Unknown info code" before the driver ran, so an XDBC-range ([500, 1000)) or vendor-specific (>= 10000) code could never be requested even though "fetch all" could return one. It now forwards them, leaving "ignore unrecognized codes" to the driver as adbc.h states. No driver change was needed — `info::build` already filters explicit requests to `REPORTED` and omits the rest — but the contract was unobservable through the C ABI until now, so pin it from both sides: a unit test at the trait level and an assertion through the driver manager in `conformance_via_driver_manager`. Closes UP-9 in REVIEW.md. apache/arrow-adbc#4534 (the other PR in range worth checking) was already adopted in #343: SqlPrepareUpdate/Stream are un-excluded and gate-enforced with RewriteSql entries pinning the new ORDER BY defaults. `c/validation` has had no commits since, so the tag bump adds no cases, and none of the four survivors in EXCLUDED are reachable by RewriteSql — three are Arrow type-mapping gaps (Duration/Interval/UInt64) and one is an arrow-rs FFI stream limitation (SqlQueryCancel cannot emit ECANCELED). **adbc-drivers/validation `dbc6857f` -> `1c20ca4b`**, which carries adbc-drivers/validation#256: `test_query_bind_dictionary` re-binds the `type/bind/string` and `type/bind/large_string` cases dictionary-encoded — the layout a pandas categorical produces. Both pass unchanged and are gate-enforced (not in the skip baseline, which needed no edit): dictionary encoding is an encoding of the same logical values, not a different logical type, and `bind::cell_value` already decodes the key at each row back through the same mapping as the plain column. Verified: `cargo fmt`/`clippy -D warnings` clean; `with-emulator.sh cargo test` green (321 unit + 53 integration + 44 mock + 6 resilience + 1 doctest); `run-adbc-validation.sh` 89 passed / 7 skipped with the gate, expected-failure and stale guards all OK; `run-foundry-validation.sh` 188 passed / 65 skipped with FOUNDRY_VALIDATION_REQUIRE_PASSES and the skip-baseline guard on. Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net>
Add a test for binds a dictionary-encoded column (what pandas often produce).
Dictionary encoding is an encoding of the same logical values, not a different logical type (Arrow columnar format, Dictionary-encoded Layout), so a driver that binds plain string columns should accept the dictionary-encoded equivalent, decoding it if the database has no native counterpart:
Verified against existing drivers: sqlite, postgres, mysql and flightsql (for the example server it needed a fix: apache/arrow-go#936).
Context: Missed this when implementing a new driver, so having this test would have helped.