-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-12719][SQL] SQL generation support for Generate #11768
Conversation
Test build #53337 has finished for PR 11768 at commit
|
// This only happens when we put UDTF in project list and there is no FROM clause. Because we | ||
// always generate LATERAL VIEW for `Generate`, here we use a trick to put a dummy sub-query | ||
// after FROM clause, so that we can generate a valid LATERAL VIEW SQL string. | ||
s"(SELECT 1) ${SQLBuilder.newSubqueryName}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you put a concrete example at here?
@cloud-fan I resolved the conflict and the commit at here. You can cherry-pick this commit to your original branch (and keep using that) or you can push update directly to my branch. |
SQLBuilder.newSubqueryName, | ||
"AS", | ||
columnAliases | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This build call has many arguments. It will be good to put an example at here.
@cloud-fan Overall it looks good. I have left a few comments. |
Test build #53339 has finished for PR 11768 at commit
|
Closing it. |
What changes were proposed in this pull request?
This PR adds SQL generation support for Generate operator. It always converts Generate operator into LATERAL VIEW format as there are many limitations to put UDTF in project list.
This PR is based on #11658, please see the last commit to review the real changes.
Thanks @dilipbiswal for his initial work! Takes over #11596
How was this patch tested?
new tests in LogicalPlanToSQLSuite