Skip to content

perf: fast-path column decoders for non-null data + execute/2 for DDL#154

Closed
hugobarauna wants to merge 2 commits into
livebook-dev:mainfrom
hugobarauna:dux-perf
Closed

perf: fast-path column decoders for non-null data + execute/2 for DDL#154
hugobarauna wants to merge 2 commits into
livebook-dev:mainfrom
hugobarauna:dux-perf

Conversation

@hugobarauna
Copy link
Copy Markdown
Member

@hugobarauna hugobarauna commented Mar 27, 2026

Context

This PR is the result of an AI-driven performance optimization experiment on Dux (a DuckDB dataframe library) using Claude Code. The code was not human-reviewed beyond passing Dux's test suite. The intent is to show potential optimization approaches — feel free to close this PR or cherry-pick individual ideas.

Companion Dux PR: elixir-dux/dux#44

Changes

1. Fast-path column decoders when no nulls are present

DuckDB always sends validity bitmaps even when all values are valid (all 0xFF bytes). The current decoders check bitmap_valid? and call through a decoder closure per element — for 400k rows × 5 columns, that's 2M unnecessary function calls.

This adds an all_valid?/1 check per column (O(1) binary comparison) and dispatches to plain decoders that skip both the bitmap check and the closure call:

# Before (per element): bitmap check + closure call
value = if bitmap_valid?(validity, index, offset), do: decoder.(int)

# After (per column): one check, then plain binary traversal
if all_valid?(validity), do: decode_s64_plain(binary)

Fast paths added for: s8-s64, u8-u64, f16-f64, string/binary (32+64 bit offsets), and date32.

Impact: ~15-20% improvement on Adbc.Result.to_map for typical non-null data.

2. Adbc.Connection.execute/2 for DDL statements

Adds a public function that uses command dispatch (no stream setup/teardown) for statements that return no data (CREATE VIEW, DROP TABLE, SET, etc.):

Adbc.Connection.execute(conn, "CREATE TEMPORARY VIEW my_view AS (SELECT ...)")

This avoids the stream → stream_results → unlock cycle that query/2 goes through. Saves ~50-100μs per call.

3. View cleanup in GC handler (⚠️ Dux-specific, open to alternatives)

Modified handle_command({:delete_on_gc, table_name}) to check for a __dux_v_ name prefix and use DROP VIEW IF EXISTS instead of DROP TABLE IF EXISTS. This is needed because Dux's performance optimization creates temp views instead of temp tables, but the existing GC mechanism only drops tables.

I'm open to alternative approaches here. Some options:

  • A separate adbc_delete_view_on_gc_new NIF function
  • An additional parameter to adbc_delete_on_gc_new specifying the drop command
  • A generic adbc_delete_on_gc_new/3 that accepts a custom SQL template

The prefix-based approach is the least invasive but couples ADBC to a Dux naming convention.

4. Single-batch fast path for to_map/1

Added a pattern match for the common single-batch case that skips Enum.zip_with + Enum.flat_map.

Benchmark impact (measured in Dux context)

These ADBC changes contributed to closing the gap between Dux and Explorer on to_rows():

Operation Before ADBC changes After Explorer
filter to_rows 1.5x slower 1.0x (parity) baseline
mutate to_rows 1.5x slower 0.95x (faster) baseline

🤖 Generated with Claude Code

Hugo and others added 2 commits March 27, 2026 11:19
When all values in a column are valid (no nulls), skip the per-element
bitmap_valid? check and decoder closure call. DuckDB always sends validity
bitmaps even for non-null data, so detect all-valid via binary comparison.

Fast paths for: s8-s64, u8-u64, f16-f64, string/binary (32+64 offsets), date32.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two changes to support Dux's view-based compute optimization:

1. Add Adbc.Connection.execute/2 — command dispatch (no stream) for DDL
   statements that return no data. Faster than query/2 for CREATE VIEW.

2. Fix delete_on_gc handler to DROP VIEW for __dux_v_ prefixed names.
   Previously all GC cleanup used DROP TABLE, which is a no-op on views.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@josevalim
Copy link
Copy Markdown
Contributor

Thank you! I pushed each of those as individual commits. For the validity one, I opted-in to check the null_count on the C side, which should be quite more efficient (assuming it is populated correctly).

@josevalim josevalim closed this Mar 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants