Add datafusion-substrait crate #4543
Conversation
Get first test passing
* use substrait 0.1 * use datafusion 8.0
Added fetch to consumer Added limit to producer Added unit tests for limit Added roundtrip_fill_none() for testing when None input can be converted to 0 Update src/consumer.rs Co-authored-by: Andy Grove <andygrove73@gmail.com> Co-authored-by: Andy Grove <andygrove73@gmail.com>
Add consumer Add producer and test Modified error string
* Add plan and function extension support * Removed unwraps
* Add consumer, producer and tests for aggregate relation Change function extension registration from absolute to relative anchor (reference) Remove operator to/from reference * Fixed function registration bug * Add test * Addressed PR comments
* Changed field reference from masked reference to direct reference * Handle unsupported case (struct with child)
Fixed aggregate function register bug
Add test case
* Implement CASE WHEN * Add more case to test * Addressed comments
* feat: support explicit catalog/schema names in ReadRel Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: use re-exported expr crate Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
|
@waynexia @nseekhao @JanKaul I need to go through the ASF IP clearance process to have this code moved into this repo. Could you please follow the instructions at https://www.apache.org/licenses/contributor-agreements.html to submit an ICLA (Individual Contributor Licence Agreement) if you do not already have one on file? Thanks in advance for helping with this! |
|
Sorry for the delay 🥲 I just got feedback from the secretary. I'll fix the issue and resubmit it today. Update: it's done 🥳 |
|
@waynexia @JanKaul @nseekhao @Dandandan Thanks everyone for filing ICLAs. As part of the IP clearance process, I have to remind active committers that they are responsible for ensuring that a Corporate CLA is recorded if such is required to authorize their contributions under their individual CLA. I have filled out the necessary "paperwork" and will file this in the next day or two and start a vote on accepting this donation. Thanks for your patience. |
|
Thanks for handling this @andygrove |
|
IP clearance form: https://incubator.apache.org/ip-clearance/arrow-datafusion-substrait.html |
|
The Apache Arrow PMC has voted to accept this contribution: https://lists.apache.org/thread/qcbdt8y2vkwvwkjmmljn4jzvggzb2fkk A lazy consensus incubator vote has now started: https://lists.apache.org/thread/m5q4qxr32xzhotljv6z7mg3dofl46rv6 |
andygrove
changed the title
|
The incubator vote passed. Given that the PMC has already voted to accept this donation I plan on merging this PR later today. |
There was a problem hiding this comment.
It is great to see this finally land @andygrove -- thank you!
|
Benchmark runs are scheduled for baseline = eb19a67 and contender = 0d27fcb. 0d27fcb is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Closes #4536
Rationale for this change
Move development of datafusion-substrait into main DataFusion repo.
What changes are included in this PR?
datafusion/substraitfolder, containing contents ofdatafusion-substraitrepo, preserving history.Are these changes tested?
Tests are included.
Are there any user-facing changes?
No