[SPARK-49566][SQL] Add SQL pipe syntax for the SET operator #48940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closed

dtenedor wants to merge 6 commits into apache:master from dtenedor:pipe-syntax-set

Contributor

dtenedor commented Nov 23, 2024 •

edited

Loading

What changes were proposed in this pull request?

This PR adds SQL pipe syntax support for SET operator.

This operator removes one or more existing column from the input table and replaces each one with a new computed column whose value is equal to evaluating the specified expression.

This is equivalent to SELECT * EXCEPT (name), <newExpressions> AS name in the SQL compiler. It is provided as a convenience feature and some functionality overlap exists with lateral column aliases.

For example:

-- Setting with an expression.
values (0, 'pqr', 2), (3, 'tuv', 5) as tab(a, b, c)
|> set c = a + length(b);

0, 'pqr', 3
3, 'tuv', 6

Why are the changes needed?

The SQL pipe operator syntax will let users compose queries in a more flexible fashion.

Does this PR introduce any user-facing change?

Yes, see above.

How was this patch tested?

This PR adds a few unit test cases, but mostly relies on golden file test coverage. I did this to make sure the answers are correct as this feature is implemented and also so we can look at the analyzer output plans to ensure they look right as well.

Was this patch authored or co-authored using generative AI tooling?

No


          commit

6cf23b2

github-actions bot added the SQL label

dtenedor changed the title ~~[WIP][SPARK-49566][SQL] Add SQL pipe syntax for the SET operator~~ [SPARK-49566][SQL] Add SQL pipe syntax for the SET operator

dtenedor marked this pull request as ready for review

November 23, 2024 04:03

Contributor Author

dtenedor commented Nov 23, 2024

cc @cloud-fan @gengliangwang here is the |> SET operator :)

cloud-fan reviewed

View reviewed changes

sql/core/src/test/resources/sql-tests/results/pipe-operators.sql.out Outdated Show resolved Hide resolved

cloud-fan reviewed

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala Outdated Show resolved Hide resolved


          respond to code review comments

5bdc971

respond to code review comments

respond to code review comments

respond to code review comments

dtenedor commented

View reviewed changes

Contributor Author

dtenedor left a comment

Thanks @cloud-fan for your review! Please take another look.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala Outdated Show resolved Hide resolved

sql/core/src/test/resources/sql-tests/results/pipe-operators.sql.out Outdated Show resolved Hide resolved

dtenedor requested a review from cloud-fan

November 25, 2024 19:01

cloud-fan reviewed

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala Outdated Show resolved Hide resolved

cloud-fan reviewed

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala Show resolved Hide resolved

dtenedor added 2 commits

November 26, 2024 10:33


          respond to code review comments

42ad558


          respond to code review comments

b050a61

dtenedor commented

View reviewed changes

Contributor Author

dtenedor left a comment

Thanks again @cloud-fan for your reviews!!

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala Show resolved Hide resolved

gengliangwang reviewed

View reviewed changes

sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4 Show resolved Hide resolved

gengliangwang reviewed

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala Outdated Show resolved Hide resolved


          respond to code review comments

de05d9d

dtenedor commented

View reviewed changes

Contributor Author

dtenedor left a comment

Thanks @gengliangwang for your review!

sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4 Show resolved Hide resolved

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala Outdated Show resolved Hide resolved

dtenedor requested review from cloud-fan and gengliangwang

November 27, 2024 00:28

cloud-fan reviewed

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/pipe-operators.sql Show resolved Hide resolved


          respond to code review comments

041727c

respond to code review comments

dtenedor commented

View reviewed changes

Contributor Author

dtenedor left a comment

Thanks @cloud-fan for your review again.

sql/core/src/test/resources/sql-tests/inputs/pipe-operators.sql Show resolved Hide resolved

dtenedor requested a review from cloud-fan

November 27, 2024 19:15

cloud-fan reviewed

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala

-              case class UnresolvedStarExcept(target: Option[Seq[String]], excepts: Seq[Seq[String]])
+              case class UnresolvedStarExceptOrReplace(
+                  target: Option[Seq[String]],
+                  excepts: Seq[Seq[String]],

Contributor

cloud-fan Nov 28, 2024

The excepts here can be nested columns, so technically we can support nested columns in the SET clause. This can be a followup work as an improvement.

Contributor Author

dtenedor Nov 28, 2024

@cloud-fan yes, and also we could support SELECT * REPLACE(<column_name>, <new_expression>) as well (some other SQL engines support this). We can keep these features in mind for adding to Spark in the future.

cloud-fan approved these changes

View reviewed changes

Contributor

cloud-fan commented Nov 28, 2024

thanks, merging to master!

cloud-fan closed this in

68be1da

Contributor

cloud-fan commented Nov 28, 2024

@HyukjinKwon @allisonwang-db shall we implement df.withColumn with this way?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

SQL