-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-13924][SQL] officially support multi-insert #11754
Conversation
cc @hvanhovell |
Test build #53274 has finished for PR 11754 at commit
|
Cool! LGTM |
Merging in master. Thanks. |
## What changes were proposed in this pull request? There is a feature of hive SQL called multi-insert. For example: ``` FROM src INSERT OVERWRITE TABLE dest1 SELECT key + 1 INSERT OVERWRITE TABLE dest2 SELECT key WHERE key > 2 INSERT OVERWRITE TABLE dest3 SELECT col EXPLODE(arr) exp AS col ... ``` We partially support it currently, with some limitations: 1) WHERE can't reference columns produced by LATERAL VIEW. 2) It's not executed eagerly, i.e. `sql("...multi-insert clause...")` won't take place right away like other commands, e.g. CREATE TABLE. This PR removes these limitations and make us fully support multi-insert. ## How was this patch tested? new tests in `SQLQuerySuite` Author: Wenchen Fan <wenchen@databricks.com> Closes apache#11754 from cloud-fan/lateral-view.
@@ -1776,4 +1776,33 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton { | |||
|FROM (SELECT '{"f1": "value1", "f2": 12}' json, 'hello' as str) test | |||
""".stripMargin), Row("value1", "12", BigDecimal("3.14"), "hello")) | |||
} | |||
|
|||
test("multi-insert with lateral view") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cloud-fan how come a test case is added only in the Hive module?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
at that time, we haven't had a clear plan to remove hive dependency, and multi-insert
is considered as a hive only feature. Although the parse rule is in catalyst module, I thought we will separate it into hive module.
What changes were proposed in this pull request?
There is a feature of hive SQL called multi-insert. For example:
We partially support it currently, with some limitations: 1) WHERE can't reference columns produced by LATERAL VIEW. 2) It's not executed eagerly, i.e.
sql("...multi-insert clause...")
won't take place right away like other commands, e.g. CREATE TABLE.This PR removes these limitations and make us fully support multi-insert.
How was this patch tested?
new tests in
SQLQuerySuite