Skip to content

[C++] Substrait SinkNode Modification for usability #31455

@asfimport

Description

@asfimport

Currently, the Substrait implementation uses a consuming_sink node as the end-point to capture the results of an executed query. To further enhance the ability to use the wide variety of SinkNodes in the streaming execution engine, it is better to evaluate other options to provide usability for various use cases. 

Some of the use cases include consuming a Substrait query plan in R and Python APIs. And this extends beyond the Arrow APIs where third-party tools willing to use Substrait can be benefitted from these. At the moment there is active work on providing R/Python APIs for Substrait consumption (https://issues.apache.org/jira/browse/ARROW-15779). 
A potential modification to provide usability is to take in a SinkNodeFactory-like interface instead of just using the ConsumingSink (current implementation). Such a modification would provide flexibility in developing a wide array of applications. 
This issue is open for discussion. 

Reporter: Vibhatha Lakmal Abeykoon / @vibhatha

Note: This issue was originally created as ARROW-16036. Please see the migration documentation for further details.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions