New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[GLib][Ruby][Dataset] Remove bindings for internal classes #27634
Comments
Weston Pace / @westonpace: |
Kouhei Sutou / @kou: |
Kouhei Sutou / @kou: |
Weston Pace / @westonpace: In the future though it might be better to change Scanner::Scan to return a RecordBatchIterator instead of ScanTaskIterator. Otherwise we're putting the burden on the user to manage how many files are loaded concurrently. I think that should either be hidden completely or controlled by some option (MaxConcurrentFiles or, better yet, MaxAllocatedBytes) |
Ben Kietzman / @bkietz: If that's not acceptable, I'd think that ARROW-7001 should prefer to add |
Neal Richardson / @nealrichardson: What I really want is to be able to essentially pass a function/lambda to something like ToTable or ToBatches and have that function be applied to every record batch in the stream. I don't want to manage consuming the ScanTasks/RecordBatchIterators, I'd prefer to have the C++ library handle that. (In my current hacky use of ScanTasks, it's actually prohibitively slow because it has to consume the iterators single-threaded.) |
Ben Kietzman / @bkietz: |
Kouhei Sutou / @kou: I agree with removing GLib bindings of them. |
Kouhei Sutou / @kou: |
GLib and ruby include bindings for internal classes such as ScanOptions, ScanContext, InMemoryScanTask, ScanTask, ... These are probably unnecessary and should be removed to present a simpler interface less prone to breakage under refactoring of the wrapped classes https://github.com/apache/arrow/pull/9532/checks?check_run_id=1974229719#step:8:2071
Reporter: Ben Kietzman / @bkietz
Assignee: Kouhei Sutou / @kou
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-11782. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: