Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unblocking [RUST-INT][TPCH]: Add a call to .combine_chunks() when creating a Table from arrow #700

Merged
merged 1 commit into from Mar 15, 2023

Conversation

jaychia
Copy link
Contributor

@jaychia jaychia commented Mar 15, 2023

  • When we create Tables from PyArrow tables, we can only handle one batch at the moment
  • This causes some issues when running our TPC-H unit tests, ostensibly because the parquet files define multiple Parquet row groups
  • This PR adds a temporary fix to unblock running our new Rust code on TPC-H by calling to .combine_chunks() before creating the table so that we only have one batch

@jaychia jaychia changed the title Add a call to .combine_chunks() when creating a Table from arrow Hack for [RUST-INT][TPCH]: Add a call to .combine_chunks() when creating a Table from arrow Mar 15, 2023
@jaychia jaychia changed the title Hack for [RUST-INT][TPCH]: Add a call to .combine_chunks() when creating a Table from arrow Unblocking [RUST-INT][TPCH]: Add a call to .combine_chunks() when creating a Table from arrow Mar 15, 2023
@jaychia jaychia merged commit 095a12f into main Mar 15, 2023
6 checks passed
@jaychia jaychia deleted the jay/hack-Table-from-arrow-multi-chunks branch March 15, 2023 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant