String columns come back as `Utf8` even when cardinality is tiny — is that expected?

[PavloPolovyi]

Hey!

We've been using both adbc_snowflake and databricks-adbc to pull data, and we noticed that string columns always come back as plain Utf8, no matter how repetitive the values are. Even when a 1M-row column has only 10 distinct values, we get a flat Utf8 array rather than Dictionary<Int32, Utf8>.

A few questions:

1. Is this the intended behavior? We weren't sure whether the warehouse ships strings as dictionary-encoded on the wire and the driver flattens them, or whether they always come in flat from the source.
2. Is there an option we're missing? Something to either preserve dictionary encoding (if it exists upstream), or otherwise get Dictionary<Int32, Utf8> back from the driver?
3. If there isn't, would you be open to adding one? Something like a cardinality threshold — columns where the distinct-to-row-count ratio is below some limit get cast to Dictionary<Int32, Utf8> before reaching the consumer.

It matters to us because we serialize the data to Arrow IPC and send it over a network. Dictionary-encoded strings are a few times smaller on the wire and the decoder on the other end has a fast path for that type. Today we walk each RecordBatch ourselves and cast qualifying Utf8 columns to dictionaries — it works, but it feels like the kind of thing that should live in the driver, especially since every consumer doing IPC over a network probably wants the same thing.

Happy to share more detail if useful. Thanks!

[CurtHagenlocher]

For both Snowflake and Databricks, the serialized Arrow data is produced by the server and simply passed through by the driver.

[zeroshade]

As far as I'm aware, in both the Snowflake and Databricks cases, it's coming back flat from the source. If there's an option to get them to use Dictionary encoding, I'm not currently aware of it. Neither driver has logic that would flatten it out if the source delivered it in a dictionary encoded format.

[skalkin]

I am also interested in this question - in our experiments with Snowflake and Databricks, ADBC drivers performed on par with the existing JDBC drivers, even though the tests were designed to favor the columnar-first approach (a mix of numerical columns, and string columns with repetitive long strings - something a columnar engine should be able to very efficiently compress and work with).

I keep reading about people getting 30x speedup when switching to ADBC, but I'm a bit puzzled at this point...

[zeroshade]

@skalkin how many rows/columns were you dealing with in your tests? As far as I'm aware, neither Snowflake nor Databricks are using the Arrow compression. Also, both of them are already using Arrow for their transport in the JDBC drivers so the big difference between ADBC or JDBC in those cases is going to be solely the fact that ADBC avoids the transposition into rows that JDBC does. As a result, for many cases the exact performance benefits will depend on the number of rows (hundreds of thousands/millions) and number of columns and on what specifically you're doing with the data afterwards.

For example, if you're taking the data and feeding it into a dataframe, writing it out to a parquet file, or building charts/visualizations (i.e. things that already convert to a columnar representation) you'll see more benefit than if you're just printing it out.

The query also matters when testing as a particularly expensive query may end up dwarfing the transport I/O. Can you share more information about what your experiments were testing? Specifically for Snowflake, we've tested using the default TPC-H sample dataset that it provides and saw that we start getting statistically significant performance benefits at around half a million rows. Again, mostly because they are already using Arrow for the transport in the ODBC/JDBC drivers so it's just the cost of the transpose/conversion. To be fair though, we've mostly tested against ODBC not JDBC.

[PavloPolovyi]

@zeroshade We are working together with @skalkin.

What we tested - all simple SELECT * FROM …, no joins or aggregations, so transport is the main cost:

• 1K / 100K / 1M rows × 10 columns of mixed types (int, double, bool, timestamp, varchar)
• 1M rows × 10 string columns at varying cardinality - 10, 50, 100, 500, 1K, 5K, 10K, 50K, 100K, and 1M distinct values

One note on setup: both sides ship chunks of 100K rows to the next stage of our pipeline - on JDBC via setFetchSize(100000), on the ADBC path our service accumulates RecordBatches into matching 100K-row chunks before forwarding. Same chunk count, same chunk size on the wire, neither side buffers the whole result. So the comparison should be fair.

We keep the data columnar from start to finish - each RecordBatch is sent as Arrow IPC over a WebSocket, then decoded into a DataFrame on the other side for visualization. This is the "consumer is already columnar" case you mentioned, and on the smaller datasets (1K, 100K rows), Snowflake ADBC is ~20–30% faster than our JDBC-based path.

Where we don't see the win, two cases:

1. 1M rows, mixed types. Snowflake is roughly tied with JDBC. Databricks ADBC is clearly slower - most of the time is spent waiting on the ADBC reader. Our best guess is that Databricks's CloudFetch path (results delivered through S3) is the bottleneck when running across regions, while JDBC streams directly via Thrift. But that's a separate conversation.

2. 1M rows, strings - the topic of this discussion. On the JDBC side, our Java client dict-encodes strings before sending, so the wire bytes are all 2–3× smaller. With plain UTF-8 from ADBC, each of those steps does more work. Our own workaround that checks each column and casts it to a Dictionary gets Snowflake to ~1.6× JDBC on this test (down from ~3.2× without it) - much better, but still not even.

[zeroshade]

1M rows, mixed types. Snowflake is roughly tied with JDBC. Databricks ADBC is clearly slower - most of the time is spent waiting on the ADBC reader. Our best guess is that Databricks's CloudFetch path (results delivered through S3) is the bottleneck when running across regions, while JDBC streams directly via Thrift. But that's a separate conversation.

This is definitely something we should look into, but yea it's a separate conversation. CC @lidavidm

1M rows, strings - the topic of this discussion. On the JDBC side, our Java client dict-encodes strings before sending, so the wire bytes are all 2–3× smaller. With plain UTF-8 from ADBC, each of those steps does more work. Our own workaround that checks each column and casts it to a Dictionary gets Snowflake to ~1.6× JDBC on this test (down from ~3.2× without it) - much better, but still not even.

It's curious that JDBC would be faster in this case since for Snowflake it would have to convert from Arrow -> Rows for JDBC before you do your dict-encoding which then would convert back to Arrow. Though I guess it would depend on how you implemented the dictionary casting/conversion (I'm not familiar enough with the Java implementation to know how good it is there). If the dictionary encoding is improving the performance that much, can you confirm where the bottleneck is and if it's the ADBC interaction or somewhere else?

It would definitely be an improvement if we could convince Snowflake/Databricks to implement dictionary encoding on the server side though. I'll see if I can do anything there.

[lidavidm]

The Databricks Go driver is too early in development. I think I saw it wasn't even enabling parallel downloads in CloudFetch. Also if Arrow-Java + JNI bindings are being used, I think Arrow-Java also introduces some issues around strings in particular...data gets copied into a separate buffer (Text type) before it then gets re-encoded as a String so you're paying extra copies

[PavloPolovyi]

If the dictionary encoding is improving the performance that much, can you confirm where the bottleneck is and if it's the ADBC interaction or somewhere else?

The ADBC interaction is not the bottleneck. Warehouse fetch takes about the same on both paths (~3 s). After fetch, the limiting factor is the overall size of the data going through our pipeline. The JDBC path converts the data into our in-house columnar format on the way out and dict-encodes strings as part of that conversion, so its payload ends up around 3× smaller (~40 MB vs ~120 MB on this query). Less data through the rest of the pipeline, faster end-to-end.

About "JDBC has to convert Arrow → rows → back to columnar" - true on paper, and it's exactly why ADBC wins on every workload where the wire representation is the same on both paths (though only 20 - 30%). The string-heavy case is special because the JDBC path's column rebuild gives us dict-encoding essentially for free. The ADBC path correctly skips the rebuild, but that also means dict-encoding doesn't fall out naturally from the pipeline - we had to add a dedicated pass for it. On the JDBC path, the column-build step happens to be a natural place to categorize strings, and the cost is small relative to the size win.

Server-side dictionary encoding from Snowflake/Databricks would skip our user-side cast entirely and shrink the fetch payload too. Really appreciate you offering to push on that.

[PavloPolovyi]

The Databricks Go driver is too early in development. I think I saw it wasn't even enabling parallel downloads in CloudFetch. Also if Arrow-Java + JNI bindings are being used, I think Arrow-Java also introduces some issues around strings in particular...data gets copied into a separate buffer (Text type) before it then gets re-encoded as a String so you're paying extra copies

Thanks for the comment. On our end, the CloudFetch parallel downloads do seem to kick in. Tuning databricks.cloudfetch.max_chunks_in_memory=16 and databricks.cloudfetch.link_prefetch_window=128 gave us measurable improvement, so the parallelism path appears to be active in the version we tested (the Rust databricks-adbc crate at the current main revision). The gap to JDBC is still real on our end (~2.5× slower on 1M-row mixed-type queries on Databricks).
On Arrow-Java + JNI, we're not on that path directly. Our Java service reads the JDBC ResultSet via the standard getObject() API and builds its own internal columnar format from there, with no Arrow-Java in our code.

[lidavidm]

Ah, I was thinking of the Go driver...that said the Rust crate is also rather early and hasn't had an actual release yet.

How are you using the Rust driver if there's no JNI and no Arrow-Java? Sorry, I think I'm misunderstanding something about the comparison being made. Ah wait, do you have Rust service and you have a Java bridge somewhere to use the JDBC driver? Or you're comparing separate Rust and Java services?

[PavloPolovyi]

We have separate services, Rust (for ADBC) and Java (JDBC).

[lidavidm]

Ah, ok. So for Java/JDBC, we have: Arrow data from Databricks/Snowflake => converted to rows for JDBC => converted to Arrow and dict-encoded => sent over WebSocket; for Rust/ADBC we have: Arrow data from Databricks/Snowflake => dict-encoded => sent over WebSocket? I wonder if there's some other bottleneck (especially in the Snowflake case), since it seems it should do at most the same amount of work...