-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[C++] Substrait join results in all zeroes on the righthand side of the join #34484
Comments
Your plan is good. It seems we are doing way too much testing with in-memory tables 😰 Since you are actually scanning it is using a scan node. The scan node is obnoxiously (and incorrectly) appending the augmented fields This is a bug in the Acero's Substrait handling for scan. |
CC @vibhatha if you have time to take a look. For now we should just hide the augmented fields entirely from Substrait. At the moment I'm thinking the simplest way to do this might be to place a project node after any scan node. The project node can then hide these fields. Then we can wait to fix it properly until we have the new scan node. I think a more "proper" fix would be to key on the ReadRel's |
take |
Sure, @westonpace I will work on this. |
The flip side of that is that we really should implement a proper TableProvider instead of writing temporary Parquet files! Glad that it at least exposed this before it was used somewhere else 😬 |
I've identified a smaller reproduction case for this bug -- a single read relation that consumes a local parquet file. This issue does not occur for named tables. Here's a test that properly catches the issue:
|
It feels to me that augmented fields should never leave the read rel. If the contents of the augmented fields are interesting for the rest of the plan then an expression could be used in the read relation to reference those fields (thus preserving them for future consumption). With that design the processing schema additionally includes the augmented fields but the output schema includes only the normal fields. |
I'm not entirely sure I follow. What is the "processing schema". Mentally I think of the augmented fields as fields that are always present in every file. In other words, if you create a file with one column X then you have 5 field (X plus the four augmented fields). The base schema of the file has 5 fields and you can pick which ones to include or not in the projection. I think the issue at hand here might be better described as the fact that the producer and consumer disagree about what fields are present in the file. |
### Rationale for this change Augmented fields interfere with the schema passing between nodes. When enabled they cause names/schema mismatching at the end of the plan. ### What changes are included in this PR? Adds an option to disable augmented fields (defaulting to adding them), connects it everywhere it is called, and disables it in ReadRel conversion. ### Are these changes tested? Yes. ### Are there any user-facing changes? There are no API related changes however this will allow Substrait plans that consume local files to work without requiring a project/emit relation after the read relation to remove the unexpected fields. * GitHub Issue: #34484 Authored-by: David Sisson <EpsilonPrime@users.noreply.github.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
Issue resolved by pull request 41583 |
Here's how I would implement augmented fields in Substrait: There are some number of extra fields that can be defined on a consumer that aren't strictly part of the data. These can be referenced by accessing hidden fields known as "augmented" fields. To access them a Substrait plan can reference these fields with a new FieldReference reference_type augmented_reference. Since there is no enum or registry for augmented types we access them by name.
Augmented fields are only valid in the read relation (and the short chain of non-join relations (including FilterRel and most importantly ProjectRel) thereafter unless we want to define a new steps out option). If these fields need to be accessed after that point a ProjectRel should expose these fields to be preserved as normal fields for the rest of the computation. A consumer should reject plans that ask for augmented fields that are not supported. |
…apache#41583) ### Rationale for this change Augmented fields interfere with the schema passing between nodes. When enabled they cause names/schema mismatching at the end of the plan. ### What changes are included in this PR? Adds an option to disable augmented fields (defaulting to adding them), connects it everywhere it is called, and disables it in ReadRel conversion. ### Are these changes tested? Yes. ### Are there any user-facing changes? There are no API related changes however this will allow Substrait plans that consume local files to work without requiring a project/emit relation after the read relation to remove the unexpected fields. * GitHub Issue: apache#34484 Authored-by: David Sisson <EpsilonPrime@users.noreply.github.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
…apache#41583) ### Rationale for this change Augmented fields interfere with the schema passing between nodes. When enabled they cause names/schema mismatching at the end of the plan. ### What changes are included in this PR? Adds an option to disable augmented fields (defaulting to adding them), connects it everywhere it is called, and disables it in ReadRel conversion. ### Are these changes tested? Yes. ### Are there any user-facing changes? There are no API related changes however this will allow Substrait plans that consume local files to work without requiring a project/emit relation after the read relation to remove the unexpected fields. * GitHub Issue: apache#34484 Authored-by: David Sisson <EpsilonPrime@users.noreply.github.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
Describe the bug, including details regarding any error messages, version, and platform.
Very possible that there's something wrong with my plan here! Reproducer via the R bindings:
Created on 2023-03-07 with reprex v2.0.2
(The substrait producer interface I'm using to generate this):
Created on 2023-03-07 with reprex v2.0.2
Component(s)
C++
The text was updated successfully, but these errors were encountered: