[Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning #8307
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…keep state about how much of a chunk has been processed already
…duce an empty (or null) chunk, to signal depleted
I had a look at the |
|
Thanks! |
krlmlr
pushed a commit
to krlmlr/duckdb-r
that referenced
this pull request
Sep 2, 2023
Merge pull request duckdb/duckdb#8307 from Tishj/chunk_scan_state [Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning Merge pull request duckdb/duckdb#8089 from pdet/basepython_tz Python TIMESTAMPTZ support Merge pull request duckdb/duckdb#8052 from Mytherin/evenlessci [CI] More CI reduction and clean-up Merge pull request duckdb/duckdb#7701 from Mause/bugfix/python-union-description Add union to test_all_types, and arrow and json R/W
krlmlr
pushed a commit
to krlmlr/duckdb-r
that referenced
this pull request
Sep 2, 2023
Merge pull request duckdb/duckdb#8307 from Tishj/chunk_scan_state [Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning Merge pull request duckdb/duckdb#8089 from pdet/basepython_tz Python TIMESTAMPTZ support Merge pull request duckdb/duckdb#8052 from Mytherin/evenlessci [CI] More CI reduction and clean-up Merge pull request duckdb/duckdb#7701 from Mause/bugfix/python-union-description Add union to test_all_types, and arrow and json R/W
krlmlr
pushed a commit
to krlmlr/duckdb-r
that referenced
this pull request
Sep 2, 2023
Merge pull request duckdb/duckdb#8307 from Tishj/chunk_scan_state [Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning Merge pull request duckdb/duckdb#8089 from pdet/basepython_tz Python TIMESTAMPTZ support Merge pull request duckdb/duckdb#8052 from Mytherin/evenlessci [CI] More CI reduction and clean-up Merge pull request duckdb/duckdb#7701 from Mause/bugfix/python-union-description Add union to test_all_types, and arrow and json R/W
krlmlr
pushed a commit
to krlmlr/duckdb-r
that referenced
this pull request
Sep 2, 2023
- Merge pull request duckdb/duckdb#8307 from Tishj/chunk_scan_state: [Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning - Merge pull request duckdb/duckdb#8089 from pdet/basepython_tz: Python TIMESTAMPTZ support - Merge pull request duckdb/duckdb#8052 from Mytherin/evenlessci: [CI] More CI reduction and clean-up - Merge pull request duckdb/duckdb#7701 from Mause/bugfix/python-union-description: Add union to test_all_types, and arrow and json R/W - Merge pull request duckdb/duckdb#8497 from samansmink/pending-execute-result-api-change: Add PendingExecutionResult::ALL_TASKS_BLOCKED
krlmlr
pushed a commit
to duckdb/duckdb-r
that referenced
this pull request
Sep 5, 2023
- Merge pull request duckdb/duckdb#8307 from Tishj/chunk_scan_state: [Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning - Merge pull request duckdb/duckdb#8089 from pdet/basepython_tz: Python TIMESTAMPTZ support - Merge pull request duckdb/duckdb#8052 from Mytherin/evenlessci: [CI] More CI reduction and clean-up - Merge pull request duckdb/duckdb#7701 from Mause/bugfix/python-union-description: Add union to test_all_types, and arrow and json R/W - Merge pull request duckdb/duckdb#8497 from samansmink/pending-execute-result-api-change: Add PendingExecutionResult::ALL_TASKS_BLOCKED
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds an abstract ChunkScanState class, which can be used when scanning from a source that produces data chunks, and the chunks could potentially only be partially processed.
Previously we used a
CurrentChunkstruct located in the QueryResult to keep this state, this was only used in theArrowUtil::FetchChunkmethod.We now move to using a
ChunkScanStatein there instead, which allows us to replace the source that these chunks come from.This is currently only implemented for one source, the
QueryResult.Previously we used
CurrentChunkwhich was being kept in the QueryResult, this PR retires that struct.Motivation for this PR is work on the upcoming pyarrow result collector.
In that PR we will need to scan from a ColumnDataCollection in
ArrowUtil::FetchChunk, and we will need similar state to this.