feat: add SubqueryAliasRel #648

Blizzara · 2024-06-10T15:13:30Z

Add SubqueryAliasRel that can be used to name/(re)qualify a downstream relation.

As Substrait mostly only referes to columns by their index, there is no inherent need for table name/qualifiers within Substrait. However, some consumers, e.g. DataFusion, require column names to be either unique or qualified for joins, which is troublesome w/o the possibility to qualify relations.

Blizzara · 2024-06-10T15:16:23Z

In my usecase, this is mainly a problem with VirtualTables in DataFusion. An alternative would be to add a name into the VirtualTable (or ReadRel) directly. However, having this SubqueryAliasRel would be more general.

FWIW, both Spark and DataFusion have some version of SubqueryAlias: Spark, DataFusion

EpsilonPrime · 2024-06-10T19:24:45Z

For Substrait, the answer might be to automatically name/number the tables in order to provide that distinction. Ignoring how the tables are numbered for a bit, just being able to say column X from table Y should be enough to solve the naming problem without actually naming.

A number of ways to number are possible including just referring to the location in the plan (plan id in Spark), registering the read relations in some discovery order, or using a URN for the local file/virtual table so that table references could be coalesced despite being read multiple times.

Technically I would not call this a subquery alias we aren't aliasing the subtree but instead the fields at a point in the subtree.

EpsilonPrime · 2024-06-10T19:25:58Z

proto/substrait/algebra.proto

+message SubqueryAliasRel {
+  RelCommon common = 1;
+  Rel input = 2;
+  repeated string names = 3;


If we have structured content would these names be in depth first form as described elsewhere? If so, that comment should be copied here as well.

This would be aliaising the subquery, not the field names - so this is more similar to the names on a NamedTable rather than the depth-first order used for fields.

Based on that it sounds like this is a list of names that all refer to the input relation.

Blizzara · 2024-06-10T19:38:54Z

Technically I would not call this a subquery alias we aren't aliasing the subtree but instead the fields at a point in the subtree.

The proposal here was to alias the subtree specifically, ie. do the same as what Spark's and DataFusion's SubqueryAlias does.

For Substrait, the answer might be to automatically name/number the tables in order to provide that distinction. Ignoring how the tables are numbered for a bit, just being able to say column X from table Y should be enough to solve the naming problem without actually naming.

Yes, Substrait itself does not require this addition - the Substrait plan is valid w/o it, as columns are referred by indices. However it is not possible (or at least I couldn't figure out how, yet) to do lossless roundtrips from DataFusion -> Substrait -> DataFusion for some cases without this. Like for example:
SELECT * FROM (VALUES (1, 'a')) AS t1(c1,c2) JOIN (VALUES (1, 'b')) AS t2(c3,c4) ON t1.c1 == t2.c3
or
SELECT p1.a p1_a, p2.a p2_a FROM data p1, data p2

I don't know how much value that argument holds, but I've found the roundtrip tests super useful when developing Substrait functionality for DataFusion and Spark - makes it much easier to confirm correctness.

A number of ways to number are possible including just referring to the location in the plan (plan id in Spark), registering the read relations in some discovery order, or using a URN for the local file/virtual table so that table references could be coalesced despite being read multiple times.

Yeah, there are ways. But it will make the roundtrip testing more difficult (since now the Substrait consumer has different logic for naming things than the original producer of the query plan), and may make debugging more difficult for users if the tables are not identifiable by human-written ids but only locations or generated ids. Again, not sure how much weight that argument holds - depends on what the priorities are, I guess :)

Blizzara · 2024-06-10T19:49:13Z

Actually, when I say "roundtrip", it's a bit more than that - w/o this, I think it's also impossible to maintain output schema for the roundtrip, as the columns after X -> Substrait -> X conversions would have different qualifiers than at the beginning. That is also not crucial for correctness, but helps with "quality of life", I think.

EpsilonPrime · 2024-06-10T22:04:54Z

Actually, when I say "roundtrip", it's a bit more than that - w/o this, I think it's also impossible to maintain output schema for the roundtrip, as the columns after X -> Substrait -> X conversions would have different qualifiers than at the beginning. That is also not crucial for correctness, but helps with "quality of life", I think.

Side note: If you were to do roundtrip testing starting with Substrait (Substrait -> X -> Substrait) you'd get nearly the same guarantees but without needing to ensure that the names were preserved.

Blizzara · 2024-06-11T07:31:28Z

Side note: If you were to do roundtrip testing starting with Substrait (Substrait -> X -> Substrait) you'd get nearly the same guarantees but without needing to ensure that the names were preserved.

True. But that feels less natural and requires one to use something else to generate the original substrait (or write it manually). There are cases where it makes sense, to ensure interoperability, but I'm not sure if it's feasible to be the main way for testing.

Also, it doesn't help with the human understandability/debuggability of the plans created from Substrait, which I think would/will be important.

westonpace

First of all, thank you for taking this on. Many people have asked for some kind of capability like this. I think it's important.

Is this meant to be in included together with #649 or are they mutually exclusive? If they are meant to be included together then why couldn't you apply a rename by specifying the rel_names field on the subquery relation?

Next question. Are we working with subquery, the expression, or are we working with subquery, the physical operator? Or, in datafusion terms, are you hoping to rename Expr::ScalarSubquery or LogicalPlan::Subquery.

Judging from the existence of LogicalPlan::SubqueryAlias I am assuming you are working with the latter. However, substrate has no concept of a "subquery physical operator". Can you describe a little bit how that operator works? Can a LogicalPlan::Subquery output more than a single column? If so, does this relation rename all of the output columns?

westonpace · 2024-06-11T13:42:10Z

proto/substrait/algebra.proto

@@ -144,6 +144,14 @@ message ReadRel {
  }
 }

+// (Re)qualify column names in a relation. If the system does not need qualified names, this can be ignored.
+message SubqueryAliasRel {


Why is this unique to subqueries? Wouldn't you want to be able to insert an alias at any point in a plan?

That's just the name Spark and DataFusion uses - it can be added anywhere in the plan

westonpace · 2024-06-11T13:46:42Z

proto/substrait/algebra.proto

@@ -144,6 +144,14 @@ message ReadRel {
  }
 }

+// (Re)qualify column names in a relation. If the system does not need qualified names, this can be ignored.
+message SubqueryAliasRel {


When adding a new relation you should add it to the Rel one of or else it will not be usable.

I did: https://github.com/substrait-io/substrait/pull/648/files#diff-60807bab1bfb567415f3703dba7bb84eb0ef09e691e1abd7019501f015eb9e2eR445 :)

westonpace · 2024-06-11T14:08:12Z

Side note: If you were to do roundtrip testing starting with Substrait (Substrait -> X -> Substrait) you'd get nearly the same guarantees but without needing to ensure that the names were preserved.

I do think the X -> Substrait -> X path is an important one. I think there are going to be many cases where people are using Substrait internally and they feel comfortable relying the fact that the consumer / producer are part of the "same solution" and can thus, for example, rely on the same extensions. I think this would mean that information that survives X -> Substrait -> X is valid even if it wouldn't survive X -> Substrait -> Y -> Substrait -> X (because Y wouldn't know how to interpret the field).

Blizzara

First of all, thank you for taking this on. Many people have asked for some kind of capability like this. I think it's important.

Is this meant to be in included together with #649 or are they mutually exclusive? If they are meant to be included together then why couldn't you apply a rename by specifying the rel_names field on the subquery relation?

Mutually exclusive - both independently provides the needed functionality, just in different ways. I added the other one for discussion to decide which one makes more sense (if either) :)

Next question. Are we working with subquery, the expression, or are we working with subquery, the physical operator? Or, in datafusion terms, are you hoping to rename Expr::ScalarSubquery or LogicalPlan::Subquery.

Latter - this would be turned into a LogicalPlan::SubqueryAlias (or its equivalent in Spark)

Judging from the existence of LogicalPlan::SubqueryAlias I am assuming you are working with the latter. However, substrate has no concept of a "subquery physical operator". Can you describe a little bit how that operator works? Can a LogicalPlan::Subquery output more than a single column? If so, does this relation rename all of the output columns?

Yeah, the "subquery" is maybe a misnomer here. The LogicalPlan::SubqueryAlias outputs the same columns as its input, but with the qualifiers of the names changed to the new qualifier/alias provided in the SubqueryAlias.

So as an example, if at the beginning of the query you have a table called "tbl" with columns "col1" and "col2", then you can refer to those columns either with unqualified names ("col1", "col2") or qualified names ("tbl.col1", "tbl.col2"). If you then wrap that plan in a SubqueryAlias(alias = "something_else"), you can then refer to the columns using the new qualified names ("something_else.col1", "something_else.col2") (or still the unqualified names). Where this gets important is with joins, where one may need the qualified names to distinguish between columns coming from left or right.

Blizzara · 2024-06-11T14:06:34Z

proto/substrait/algebra.proto

@@ -144,6 +144,14 @@ message ReadRel {
  }
 }

+// (Re)qualify column names in a relation. If the system does not need qualified names, this can be ignored.
+message SubqueryAliasRel {


I did: https://github.com/substrait-io/substrait/pull/648/files#diff-60807bab1bfb567415f3703dba7bb84eb0ef09e691e1abd7019501f015eb9e2eR445 :)

Blizzara · 2024-06-11T14:15:58Z

proto/substrait/algebra.proto

@@ -144,6 +144,14 @@ message ReadRel {
  }
 }

+// (Re)qualify column names in a relation. If the system does not need qualified names, this can be ignored.
+message SubqueryAliasRel {


That's just the name Spark and DataFusion uses - it can be added anywhere in the plan

westonpace · 2024-06-12T02:54:59Z

Mutually exclusive - both independently provides the needed functionality, just in different ways. I added the other one for discussion to decide which one makes more sense (if either) :)

If we go with the other approach (metadata on RelCommon) do we still need some kind of "no-op" node that can be used to alias a relation tree? Or would you envision that we just map DF Subquery -> SubqueryAlias to a single node (subquery) that has alias metadata?

Also, are you wanting to add a node for the "subquery physical operator"? (e.g. LogicalPlan::Subquery). That would be a different PR, I'm just curious.

Blizzara · 2024-06-12T08:29:43Z

If we go with the other approach (metadata on RelCommon) do we still need some kind of "no-op" node that can be used to alias a relation tree? Or would you envision that we just map DF Subquery -> SubqueryAlias to a single node (subquery) that has alias metadata?

In that case I don't think there'd be need to add "no-op" node. Producers could map their "SubqueryAlias" by adding metadata to the preceding Rel node, and consumers could read the metadata (if they want to) and add a "SubqueryAlias" to wrap the Rel.

I kinda like that approach since it makes it clear the metadata/alias is optional, and consumers/producers don't need to care about it at all unless they want to. Contrast to this SubqueryAliasRel where consumers would need to start handling the node (even if just to pass through the input rel).

Also, are you wanting to add a node for the "subquery physical operator"? (e.g. LogicalPlan::Subquery). That would be a different PR, I'm just curious.

I haven't seen the need for that, yet at least, so not planning on it currently.

vbarua · 2024-06-12T23:26:09Z

I kinda like that approach since it makes it clear the metadata/alias is optional, and consumers/producers don't need to care about it at all unless they want to. Contrast to this SubqueryAliasRel where consumers would need to start handling the node (even if just to pass through the input rel).

I was going to say something similar, but you said it better than I could myself. This is the reason I'm in favour of the solution you're proposing in #649.

Blizzara · 2024-06-19T09:54:23Z

Closing in favor of #649 - that approach seems like the better path forward overall, so let's focus discussion there!

Same goal as in #648 - to support what Spark's and DataFusion's SubqueryAlias relation does. As Substrait mostly only referes to columns by their index, there is no inherent need for table name/qualifiers within Substrait. However, some consumers, e.g. DataFusion, require column names to be either unique or qualified for joins, which is troublesome w/o the possibility to qualify relations. Also, for debugging failed plans and for roundtrip testing of X -> Substrait -> X conversions, it would be convenient to have proper, human-readable names to refer to. Closes #571

feat: add SubqueryAliasRel

4b93136

Blizzara requested review from jacques-n, cpcloud, westonpace, EpsilonPrime and vbarua as code owners June 10, 2024 15:13

Blizzara mentioned this pull request Jun 10, 2024

Projects require unique expressions names error in substrait producer/consumer apache/datafusion#10815

Closed

fix: rename to fit snake case

526019a

EpsilonPrime reviewed Jun 10, 2024

View reviewed changes

Blizzara mentioned this pull request Jun 11, 2024

feat: allow naming/aliasing relations #649

Merged

westonpace reviewed Jun 11, 2024

View reviewed changes

Blizzara commented Jun 11, 2024

View reviewed changes

Blizzara closed this Jun 19, 2024

Blizzara deleted the avo/add-subquery-alias-rel branch July 8, 2024 09:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add SubqueryAliasRel #648

feat: add SubqueryAliasRel #648

Blizzara commented Jun 10, 2024

Blizzara commented Jun 10, 2024

EpsilonPrime commented Jun 10, 2024

EpsilonPrime Jun 10, 2024

Blizzara Jun 10, 2024

EpsilonPrime Jun 10, 2024

Blizzara commented Jun 10, 2024

Blizzara commented Jun 10, 2024

EpsilonPrime commented Jun 10, 2024

Blizzara commented Jun 11, 2024

westonpace left a comment

westonpace Jun 11, 2024

Blizzara Jun 11, 2024

westonpace Jun 11, 2024

Blizzara Jun 11, 2024

westonpace commented Jun 11, 2024

Blizzara left a comment

Blizzara Jun 11, 2024

Blizzara Jun 11, 2024

westonpace commented Jun 12, 2024

Blizzara commented Jun 12, 2024

vbarua commented Jun 12, 2024

Blizzara commented Jun 19, 2024

feat: add SubqueryAliasRel #648

feat: add SubqueryAliasRel #648

Conversation

Blizzara commented Jun 10, 2024

Blizzara commented Jun 10, 2024

EpsilonPrime commented Jun 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Blizzara commented Jun 10, 2024

Blizzara commented Jun 10, 2024

EpsilonPrime commented Jun 10, 2024

Blizzara commented Jun 11, 2024

westonpace left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

westonpace commented Jun 11, 2024

Blizzara left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

westonpace commented Jun 12, 2024

Blizzara commented Jun 12, 2024

vbarua commented Jun 12, 2024

Blizzara commented Jun 19, 2024