Skip to content

Commit

Permalink
[SPARK-25833][SQL][DOCS] Update migration guide for Hive view compati…
Browse files Browse the repository at this point in the history
…bility

## What changes were proposed in this pull request?
Both Spark and Hive support views. However in some cases views created by Hive are not readable by Spark. For example, if column aliases are not specified in view definition queries, both Spark and Hive will generate alias names, but in different ways. In order for Spark to be able to read views created by Hive, users should explicitly specify column aliases in view definition queries.

Given that it's not uncommon that Hive and Spark are used together in enterprise data warehouse, this PR aims to explicitly describe this compatibility issue to help users troubleshoot this issue easily.

## How was this patch tested?
Docs are manually generated and checked locally.

```
SKIP_API=1 jekyll serve
```

Closes #22868 from seancxmao/SPARK-25833.

Authored-by: seancxmao <seancxmao@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
seancxmao authored and dongjoon-hyun committed Oct 31, 2018
1 parent 9cf9a83 commit 49bea5a
Showing 1 changed file with 15 additions and 0 deletions.
15 changes: 15 additions & 0 deletions docs/sql-migration-guide-hive-compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,21 @@ Spark SQL supports the vast majority of Hive features, such as:
* Explain
* Partitioned tables including dynamic partition insertion
* View
* If column aliases are not specified in view definition queries, both Spark and Hive will
generate alias names, but in different ways. In order for Spark to be able to read views created
by Hive, users should explicitly specify column aliases in view definition queries. As an
example, Spark cannot read `v1` created as below by Hive.

```
CREATE VIEW v1 AS SELECT * FROM (SELECT c + 1 FROM (SELECT 1 c) t1) t2;
```

Instead, you should create `v1` as below with column aliases explicitly specified.

```
CREATE VIEW v1 AS SELECT * FROM (SELECT c + 1 AS inc_c FROM (SELECT 1 c) t1) t2;
```

* All Hive DDL Functions, including:
* `CREATE TABLE`
* `CREATE TABLE AS SELECT`
Expand Down

0 comments on commit 49bea5a

Please sign in to comment.