Skip to content

[otal-arrow-rust] Adaptive array builders optimize dictionary upgrade #536

Open
@albertlockett

Description

@albertlockett

In #473 we added basic functionality for adaptive array builder but currently when upgrading the dictionary, we copy all the values.

Ideally we'd just be able to get access to the underlying key builder of the dictionary builder, finish/cast it to the new index type, and then create a new dictionary builder with the same values/internal state + the new key builder. As far as I know there's not currently a way to do this in arrow-rs.

TODO

  • confirm with arrow-rs community if this is possible
  • open any necessary issues in arrow-rs for this capability
  • implement changes in arrow-rs
  • use optimized implementation of dictionary builder conversion in dictionary upgrade.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestperformancerustPull requests that update Rust code

Type

No type

Projects

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions