[WIP][SPARK-24497][SQL] Support recursive SQL query #29210

peter-toth · 2020-07-23T18:44:07Z

What changes were proposed in this pull request?

This PR adds recursive query feature to Spark SQL.

A recursive query is defined using the WITH RECURSIVE keywords and referring the name of the common table expression within the query.
The implementation complies with SQL standard and follows similar rules to other relational databases:

A query is made of an anchor followed by a recursive term.
The anchor terms doesn't contain self reference and it is used to initialize the query.
The recursive term contains a self reference and it is used to expand the current set of rows with new ones.
The anchor and recursive terms must be joined with each other by UNION or UNION ALL operators.
New rows can only be derived from the newly added rows of the previous iteration (or from the initial set of rows of anchor terms). This limitation implies that recursive references can't be used with some of the joins, aggregations or subqueries.

Please see cte-recursive.sql and with.sql for some examples.

Please note that this PR focuses on the minimal working implementation which means:

SQL recursion is actually loop where the current iteration is computed based on the previous one's result and when an iteration returns no rows the loop is over. The final result is the union of all iteration results. This means that caching intermediate results could speed up the process, but caching was removed from this PR to reduce complexity and can be added back in a follow-up PR.
A common way to stop SQL recursion is using the LIMIT operator to stop computing more than the required number of rows. LIMIT support was removed from this PR to reduce complexity and can be added back in a follow-up PR.
Some relational databases are more relaxed in terms how many anchor and recursive terms can be in a recursion. This PR allows the most simple case and allows only 1-1 of them. A follow-up PR can target to relax this limitation.

Why are the changes needed?

Recursive query is an ANSI SQL feature that is useful to process hierarchical data.

Does this PR introduce any user-facing change?

Yes, adds recursive query feature.

How was this patch tested?

Added new UTs and tests in cte-recursion.sql and with.sql.

peter-toth · 2020-07-23T18:56:52Z

This is the static version of #23531 which means this PR doesn't do adaptive replanning in each recursive iteration, but the advantage is that the implementation comes with simplified code. Common relational DB implementations don't do replanning either.

@maropu this is very close to what you suggested before, if you have some time please review.

@maryannxue I think we can add adaptive support in a follow-up PR if needed.

cc @cloud-fan @viirya @gatorsmile

SparkQA · 2020-07-23T23:32:54Z

Test build #126431 has finished for PR 29210 at commit 016b952.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedRecursiveReference(cteName: String, accumulated: Boolean) extends LeafNode
case class RecursiveRelation(
case class RecursiveReference(
abstract class UnionBase extends LogicalPlan
case class Union(children: Seq[LogicalPlan]) extends UnionBase
case class With(
case class RecursiveRelationExec(
case class RecursiveReferenceExec(

maropu · 2020-07-24T00:30:03Z

The simpler design as a first step looks fine to me. Anyone preferring the adaptive one for this? I think we need to choose which approach to take first. @maryannxue @viirya @gatorsmile @viirya

…497-recursive-sql-static

SparkQA · 2020-07-24T14:39:14Z

Test build #126490 has finished for PR 29210 at commit 3646400.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnschedulableTaskSetAdded(stageId: Int, stageAttemptId: Int)
case class UnschedulableTaskSetRemoved(stageId: Int, stageAttemptId: Int)
case class SparkListenerUnschedulableTaskSetAdded(
case class SparkListenerUnschedulableTaskSetRemoved(
case class Union(

manuzhang · 2020-08-21T09:40:49Z

sql/core/src/test/resources/sql-tests/inputs/cte-recursion.sql

+
+CREATE TEMPORARY VIEW t AS SELECT * FROM VALUES 0, 1, 2 AS t(id);
+
+-- fails due to recursion isn't allowed with RECURSIVE keyword


without RECURSIVE ?

Thanks @manuzhang. Fixed.

SparkQA · 2020-09-01T14:36:47Z

Test build #128152 has finished for PR 29210 at commit 1a2826e.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

peter-toth · 2020-09-01T19:45:56Z

retest this please

SparkQA · 2020-09-02T00:44:12Z

Test build #128162 has finished for PR 29210 at commit 1a2826e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

github-actions · 2020-12-11T00:48:45Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

SparkQA · 2021-02-04T17:07:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39470/

SparkQA · 2021-02-04T17:26:09Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39470/

SparkQA · 2021-02-04T19:53:01Z

Test build #134884 has finished for PR 29210 at commit 794300c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-02-04T20:17:44Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39473/

SparkQA · 2021-02-04T20:51:43Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39473/

SparkQA · 2021-02-05T00:50:23Z

Test build #134890 has finished for PR 29210 at commit 1e364d3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-02-05T12:53:34Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39516/

SparkQA · 2021-02-05T13:22:18Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39516/

SparkQA · 2021-02-05T16:49:45Z

Test build #134933 has finished for PR 29210 at commit 8f79020.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

github-actions · 2021-05-17T00:10:12Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

peter-toth · 2021-05-17T07:47:19Z

I'm happy to update this PR if anyone is willing to review it. Just let me know...

phancey · 2021-07-19T09:28:36Z

I would like to see recursive queries - why did this not go through? Is there an alternative?

timschwab · 2021-10-14T20:26:21Z

@peter-toth

Pinging this to ensure it doesn't fall through the cracks. I'm currently hitting the common use case of wanting to create a materialized path from an adjacency list, and not having recursive queries built into Spark makes this more painful and less performant.

sixdimensionalarray · 2022-07-19T17:08:33Z

Another +1, ran into a use case where recursive queries were needed and alternate technology was utilized to solve because it was not available in Spark SQL.

peter-toth · 2022-07-25T11:57:04Z

Thanks for the feedback. I will try to rebase this PR on the latest master (Spark 3.4) in a few weeks.

peter-toth · 2023-01-17T14:20:11Z

Sorry guys, this is unlikely to land in Spark 3.4, maybe in 3.5...

wangyum · 2023-01-18T13:27:56Z

@peter-toth Could you rebase this PR on the master branch. I have removed the Stale tag.

peter-toth · 2023-01-18T14:16:25Z

@peter-toth Could you rebase this PR on the master branch. I have removed the Stale tag.

@wangyum, unfortunately this is a very old PR and a lots of changes are needed to make it work on the latest Spark.
I've started rebasing it and fixing it last year, but it requires quite some time and I haven't finished yet.

Let me close the PR for now and reopen once I have a working solution.

peter-toth · 2023-04-19T13:54:37Z

This PR is rebased on latest master here: #40744

[SPARK-24497][SQL] Support recursive SQL query

016b952

probot-autolabeler bot added DOCS SQL labels Jul 23, 2020

peter-toth mentioned this pull request Jul 23, 2020

[WIP][SPARK-24497][SQL] Support recursive SQL query with adaptive replanning #23531

Closed

Merge commit '84efa04c57a95ee2d289fa699036f1aa44840f72' into SPARK-24…

3646400

…497-recursive-sql-static

peter-toth changed the title ~~[WIP][SPARK-24497][SQL] Support recursive SQL query~~ [SPARK-24497][SQL] Support recursive SQL query Jul 26, 2020

manuzhang reviewed Aug 21, 2020

View reviewed changes

peter-toth added 2 commits September 1, 2020 16:32

Merge branch 'master' into SPARK-24497-recursive-sql-static

db0f28f

fix comment

1a2826e

github-actions bot added the Stale label Dec 11, 2020

gatorsmile removed the Stale label Dec 11, 2020

Merge branch 'master' into SPARK-24497-recursive-sql-static

794300c

fix expected error messages, fix table.* syntax

1e364d3

fix Scala 2.13 compilation

8f79020

github-actions bot added the Stale label May 17, 2021

github-actions bot closed this May 18, 2021

peter-toth mentioned this pull request Jul 31, 2021

[SPARK-34079][SQL] Merge non-correlated scalar subqueries #32298

Closed

wangyum removed the Stale label Jan 18, 2023

wangyum reopened this Jan 18, 2023

peter-toth changed the title ~~[SPARK-24497][SQL] Support recursive SQL query~~ [WIP][SPARK-24497][SQL] Support recursive SQL query Jan 18, 2023

peter-toth closed this Jan 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP][SPARK-24497][SQL] Support recursive SQL query #29210

[WIP][SPARK-24497][SQL] Support recursive SQL query #29210

peter-toth commented Jul 23, 2020

peter-toth commented Jul 23, 2020 •

edited

Loading

SparkQA commented Jul 23, 2020

maropu commented Jul 24, 2020

SparkQA commented Jul 24, 2020

manuzhang Aug 21, 2020

peter-toth Sep 1, 2020

SparkQA commented Sep 1, 2020

peter-toth commented Sep 1, 2020

SparkQA commented Sep 2, 2020

github-actions bot commented Dec 11, 2020

SparkQA commented Feb 4, 2021

SparkQA commented Feb 4, 2021

SparkQA commented Feb 4, 2021

SparkQA commented Feb 4, 2021

SparkQA commented Feb 4, 2021

SparkQA commented Feb 5, 2021

SparkQA commented Feb 5, 2021

SparkQA commented Feb 5, 2021

SparkQA commented Feb 5, 2021

github-actions bot commented May 17, 2021

peter-toth commented May 17, 2021 •

edited

Loading

phancey commented Jul 19, 2021

timschwab commented Oct 14, 2021

sixdimensionalarray commented Jul 19, 2022 •

edited

Loading

peter-toth commented Jul 25, 2022

peter-toth commented Jan 17, 2023

wangyum commented Jan 18, 2023

peter-toth commented Jan 18, 2023

peter-toth commented Apr 19, 2023


		CREATE TEMPORARY VIEW t AS SELECT * FROM VALUES 0, 1, 2 AS t(id);

		-- fails due to recursion isn't allowed with RECURSIVE keyword

[WIP][SPARK-24497][SQL] Support recursive SQL query #29210

[WIP][SPARK-24497][SQL] Support recursive SQL query #29210

Conversation

peter-toth commented Jul 23, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

peter-toth commented Jul 23, 2020 • edited Loading

SparkQA commented Jul 23, 2020

maropu commented Jul 24, 2020

SparkQA commented Jul 24, 2020

manuzhang Aug 21, 2020

Choose a reason for hiding this comment

peter-toth Sep 1, 2020

Choose a reason for hiding this comment

SparkQA commented Sep 1, 2020

peter-toth commented Sep 1, 2020

SparkQA commented Sep 2, 2020

github-actions bot commented Dec 11, 2020

SparkQA commented Feb 4, 2021

SparkQA commented Feb 4, 2021

SparkQA commented Feb 4, 2021

SparkQA commented Feb 4, 2021

SparkQA commented Feb 4, 2021

SparkQA commented Feb 5, 2021

SparkQA commented Feb 5, 2021

SparkQA commented Feb 5, 2021

SparkQA commented Feb 5, 2021

github-actions bot commented May 17, 2021

peter-toth commented May 17, 2021 • edited Loading

phancey commented Jul 19, 2021

timschwab commented Oct 14, 2021

sixdimensionalarray commented Jul 19, 2022 • edited Loading

peter-toth commented Jul 25, 2022

peter-toth commented Jan 17, 2023

wangyum commented Jan 18, 2023

peter-toth commented Jan 18, 2023

peter-toth commented Apr 19, 2023

peter-toth commented Jul 23, 2020 •

edited

Loading

peter-toth commented May 17, 2021 •

edited

Loading

sixdimensionalarray commented Jul 19, 2022 •

edited

Loading