Arrow Row Format #2677
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I think this crate has pretty good stories for operating on individual columns, either by downcasting to a concrete type, or invoking a
The stories for multi-column operations are substantially weaker, with patchy support for common multi-column operations such as sorts, groupings, aggregations, reassembly, etc... We have some pieces such as
Describe the solution you'd like
Having a first-class row representation will not only allow us to implement more performant versions of existing kernels such as lexsort, but also provide a pretty compelling primitive to downstreams with which to implement more advanced operations such as streaming merges, joins, aggregates, etc... There is also precedent, with the C++ arrow library providing its own row format.
Describe alternatives you've considered
We could extend the row format in DataFusion, however, this would limit its benefits to DataFusion. I think a row-oriented representation is such a fundamental primitive that it makes sense for inclusion in arrow-rs, so that it can be both used in its kernels and by downstreams that don't make use of DataFusion.
The text was updated successfully, but these errors were encountered:
This will still not allow operations on data with multiple schema, same as with RecordBatch, if that is what you're asking for?
That being said, in the case of rows with different variants, nulls will be inserted by the JSON reader for the columns found in other variants but not present in the current record. The data is effectively unified to a single schema.
This trades off memory efficiency for the ability to efficiently process data in a columnar fashion, without per-value dynamic dispatch.
This is likely acceptable, however, partitioning the data so that different schema aren't interleaved will lead to better performance and memory efficiency. The row format could help with implementing this, but does not alter the nature of arrow schema