New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-17903][SQL] MetastoreRelation should talk to external catalog instead of hive client #15460

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
6 participants
@cloud-fan
Contributor

cloud-fan commented Oct 13, 2016

What changes were proposed in this pull request?

HiveExternalCatalog should be the only interface to talk to the hive metastore. In MetastoreRelation we can just use ExternalCatalog instead of HiveClient to interact with hive metastore, and add missing API in ExternalCatalog.

How was this patch tested?

existing tests.

@cloud-fan

This comment has been minimized.

Show comment
Hide comment
@cloud-fan
Contributor

cloud-fan commented Oct 13, 2016

@SparkQA

This comment has been minimized.

Show comment
Hide comment
@SparkQA

SparkQA Oct 13, 2016

Test build #66882 has finished for PR 15460 at commit 79ec4fe.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

SparkQA commented Oct 13, 2016

Test build #66882 has finished for PR 15460 at commit 79ec4fe.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.
@SparkQA

This comment has been minimized.

Show comment
Hide comment
@SparkQA

SparkQA Oct 13, 2016

Test build #66893 has finished for PR 15460 at commit f3ba6e0.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

SparkQA commented Oct 13, 2016

Test build #66893 has finished for PR 15460 at commit f3ba6e0.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@rxin

This comment has been minimized.

Show comment
Hide comment
@rxin

rxin Oct 13, 2016

Contributor

cc @ericl

Contributor

rxin commented Oct 13, 2016

cc @ericl

@ericl

This comment has been minimized.

Show comment
Hide comment
@ericl

ericl Oct 13, 2016

Contributor

LGTM

Contributor

ericl commented Oct 13, 2016

LGTM

@rxin

This comment has been minimized.

Show comment
Hide comment
@rxin

rxin Oct 13, 2016

Contributor

A bunch of mysterious failures ...

Contributor

rxin commented Oct 13, 2016

A bunch of mysterious failures ...

@ericl

This comment has been minimized.

Show comment
Hide comment
@ericl

ericl Oct 13, 2016

Contributor

LGTM if tests pass :)

Contributor

ericl commented Oct 13, 2016

LGTM if tests pass :)

@yhuai

This comment has been minimized.

Show comment
Hide comment
@yhuai

yhuai Oct 13, 2016

Contributor

Caused by: sbt.ForkMain$ForkError: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'default' not found in database 'srcpart'; Seems we swapped the db and table name somehow.

Contributor

yhuai commented Oct 13, 2016

Caused by: sbt.ForkMain$ForkError: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'default' not found in database 'srcpart'; Seems we swapped the db and table name somehow.

@@ -636,6 +637,13 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
client.getPartitions(db, table, partialSpec)
}
override def listPartitionsByFilter(

This comment has been minimized.

@gatorsmile

gatorsmile Oct 14, 2016

Member

Need a test case in HiveExternalCatalogSuite, just like the other APIs?

@gatorsmile

gatorsmile Oct 14, 2016

Member

Need a test case in HiveExternalCatalogSuite, just like the other APIs?

This comment has been minimized.

@cloud-fan

cloud-fan Oct 14, 2016

Contributor

added

@cloud-fan

cloud-fan Oct 14, 2016

Contributor

added

@gatorsmile

This comment has been minimized.

Show comment
Hide comment
@gatorsmile

gatorsmile Oct 14, 2016

Member

LGTM except one comment about test case

Member

gatorsmile commented Oct 14, 2016

LGTM except one comment about test case

@SparkQA

This comment has been minimized.

Show comment
Hide comment
@SparkQA

SparkQA Oct 14, 2016

Test build #66922 has finished for PR 15460 at commit 628fd05.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

SparkQA commented Oct 14, 2016

Test build #66922 has finished for PR 15460 at commit 628fd05.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@SparkQA

This comment has been minimized.

Show comment
Hide comment
@SparkQA

SparkQA Oct 14, 2016

Test build #66948 has finished for PR 15460 at commit 82df987.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

SparkQA commented Oct 14, 2016

Test build #66948 has finished for PR 15460 at commit 82df987.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@asfgit asfgit closed this in 2fb12b0 Oct 14, 2016

@cloud-fan

This comment has been minimized.

Show comment
Hide comment
@cloud-fan

cloud-fan Oct 14, 2016

Contributor

thanks for the review, merging to master!

Contributor

cloud-fan commented Oct 14, 2016

thanks for the review, merging to master!

ThySinner pushed a commit to ThySinner/spark that referenced this pull request Oct 19, 2016

[SPARK-17903][SQL] MetastoreRelation should talk to external catalog …
…instead of hive client

## What changes were proposed in this pull request?

`HiveExternalCatalog` should be the only interface to talk to the hive metastore. In `MetastoreRelation` we can just use `ExternalCatalog` instead of `HiveClient` to interact with hive metastore,  and add missing API in `ExternalCatalog`.

## How was this patch tested?

existing tests.

Author: Wenchen Fan <wenchen@databricks.com>

Closes apache#15460 from cloud-fan/relation.

robert3005 pushed a commit to palantir/spark that referenced this pull request Nov 1, 2016

[SPARK-17903][SQL] MetastoreRelation should talk to external catalog …
…instead of hive client

## What changes were proposed in this pull request?

`HiveExternalCatalog` should be the only interface to talk to the hive metastore. In `MetastoreRelation` we can just use `ExternalCatalog` instead of `HiveClient` to interact with hive metastore,  and add missing API in `ExternalCatalog`.

## How was this patch tested?

existing tests.

Author: Wenchen Fan <wenchen@databricks.com>

Closes apache#15460 from cloud-fan/relation.

uzadude added a commit to uzadude/spark that referenced this pull request Jan 27, 2017

[SPARK-17903][SQL] MetastoreRelation should talk to external catalog …
…instead of hive client

## What changes were proposed in this pull request?

`HiveExternalCatalog` should be the only interface to talk to the hive metastore. In `MetastoreRelation` we can just use `ExternalCatalog` instead of `HiveClient` to interact with hive metastore,  and add missing API in `ExternalCatalog`.

## How was this patch tested?

existing tests.

Author: Wenchen Fan <wenchen@databricks.com>

Closes apache#15460 from cloud-fan/relation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment