You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add GetFragmentsAsync() and GetFragmentsAsyncImpl() functions to the generic Dataset interface, which allows to produce fragments in a streamed fashion.
This is one of the prerequisites for making FileSystemDataset to support lazy fragment processing, which, in turn, can be used to start scan operations without waiting for the entire dataset to be discovered.
To aid the transition process of moving to async implementation in Dataset/AsyncScanner code, a default implementation for GetFragmentsAsyncImpl() should be provided (yielding a VectorGenerator over the fragments vector, which is stored by every implementation of Dataset interface at the moment).
Add
GetFragmentsAsync()
andGetFragmentsAsyncImpl()
functions to the genericDataset
interface, which allows to produce fragments in a streamed fashion.This is one of the prerequisites for making
FileSystemDataset
to support lazy fragment processing, which, in turn, can be used to start scan operations without waiting for the entire dataset to be discovered.To aid the transition process of moving to async implementation in
Dataset
/AsyncScanner
code, a default implementation forGetFragmentsAsyncImpl()
should be provided (yielding a VectorGenerator over the fragments vector, which is stored by every implementation of Dataset interface at the moment).Reporter: Pavel Solodovnikov / @ManManson
Assignee: Pavel Solodovnikov / @ManManson
PRs and other links:
Note: This issue was originally created as ARROW-17318. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: