-
Notifications
You must be signed in to change notification settings - Fork 4.1k
Description
Currently, the Substrait implementation uses a consuming_sink node as the end-point to capture the results of an executed query. To further enhance the ability to use the wide variety of SinkNodes in the streaming execution engine, it is better to evaluate other options to provide usability for various use cases.
Some of the use cases include consuming a Substrait query plan in R and Python APIs. And this extends beyond the Arrow APIs where third-party tools willing to use Substrait can be benefitted from these. At the moment there is active work on providing R/Python APIs for Substrait consumption (https://issues.apache.org/jira/browse/ARROW-15779).
A potential modification to provide usability is to take in a SinkNodeFactory-like interface instead of just using the ConsumingSink (current implementation). Such a modification would provide flexibility in developing a wide array of applications.
This issue is open for discussion.
Reporter: Vibhatha Lakmal Abeykoon / @vibhatha
Note: This issue was originally created as ARROW-16036. Please see the migration documentation for further details.