[SPARK-18464][SQL] support old table which doesn't store schema in metastore #15900
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #68697 has finished for PR 15900 at commit
|
cloud-fan
force-pushed
the
hive-catalog
branch
from
a75cf30
to
4094a72
Compare
|
Test build #68717 has finished for PR 15900 at commit
|
tejasapatil
reviewed
Nov 16, 2016
| @@ -1023,6 +1023,11 @@ object HiveExternalCatalog { | |||
| // After SPARK-6024, we removed this flag. | |||
| // Although we are not using `spark.sql.sources.schema` any more, we need to still support. | |||
| DataType.fromJson(schema.get).asInstanceOf[StructType] | |||
| } else if (props.filterKeys(_.startsWith(DATASOURCE_SCHEMA_PREFIX)).isEmpty) { | |||
| // If there is no schema information in table properties, it means the schema of this table | |||
| // was empty when saving into metastore, which is possible in older version of Spark. We | |||
There was a problem hiding this comment.
nit: Please mention version number
tejasapatil
reviewed
Nov 16, 2016
| @@ -64,7 +64,9 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log | |||
| val dataSource = | |||
| DataSource( | |||
| sparkSession, | |||
| userSpecifiedSchema = Some(table.schema), | |||
| // In older version of Spark, the table schema can be empty and should be inferred at | |||
There was a problem hiding this comment.
nit: Please mention version number
yhuai
reviewed
Nov 16, 2016
| checkAnswer(spark.table("old"), Row(1, "a")) | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
It will be good to actually create a set of compatibility tests to make sure a new version of Spark can access table metadata created by a older version (starting from Spark 1.3) without problem. Let's create a follow-up jira for this task and do it during the QA period of spark 2.1.
There was a problem hiding this comment.
yhuai
reviewed
Nov 16, 2016
| // If there is no schema information in table properties, it means the schema of this table | ||
| // was empty when saving into metastore, which is possible in older version of Spark. We | ||
| // should respect it. | ||
| new StructType() |
There was a problem hiding this comment.
btw, a clarification question. This function is only needed for data source tables, right?
There was a problem hiding this comment.
no, since we also store schema for hive table, hive table will also call this function. But hive table will never go into this branch, as it always has a schema.(the removal of runtime schema inference happened before we store schema of hive table)
yhuai
reviewed
Nov 16, 2016
| properties = Map( | ||
| HiveExternalCatalog.DATASOURCE_PROVIDER -> "parquet")) | ||
| hiveClient.createTable(tableDesc, ignoreIfExists = false) | ||
| checkAnswer(spark.table("old"), Row(1, "a")) |
There was a problem hiding this comment.
Can we also test describe table and make sure it can provide correct column info?
|
Test build #68745 has finished for PR 15900 at commit
|
|
Merging in master/branch-2.1. |
asfgit
pushed a commit
that referenced
this pull request
Nov 17, 2016
…tastore ## What changes were proposed in this pull request? Before Spark 2.1, users can create an external data source table without schema, and we will infer the table schema at runtime. In Spark 2.1, we decided to infer the schema when the table was created, so that we don't need to infer it again and again at runtime. This is a good improvement, but we should still respect and support old tables which doesn't store table schema in metastore. ## How was this patch tested? regression test. Author: Wenchen Fan <wenchen@databricks.com> Closes #15900 from cloud-fan/hive-catalog. (cherry picked from commit 07b3f04) Signed-off-by: Reynold Xin <rxin@databricks.com>
uzadude
pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
…tastore ## What changes were proposed in this pull request? Before Spark 2.1, users can create an external data source table without schema, and we will infer the table schema at runtime. In Spark 2.1, we decided to infer the schema when the table was created, so that we don't need to infer it again and again at runtime. This is a good improvement, but we should still respect and support old tables which doesn't store table schema in metastore. ## How was this patch tested? regression test. Author: Wenchen Fan <wenchen@databricks.com> Closes apache#15900 from cloud-fan/hive-catalog.
ghost
pushed a commit
to dbtsai/spark
that referenced
this pull request
Aug 15, 2017
…hema in table properties ## What changes were proposed in this pull request? This is a follow-up of apache#15900 , to fix one more bug: When table schema is empty and need to be inferred at runtime, we should not resolve parent plans before the schema has been inferred, or the parent plans will be resolved against an empty schema and may get wrong result for something like `select *` The fix logic is: introduce `UnresolvedCatalogRelation` as a placeholder. Then we replace it with `LogicalRelation` or `HiveTableRelation` during analysis, so that it's guaranteed that we won't resolve parent plans until the schema has been inferred. ## How was this patch tested? regression test Author: Wenchen Fan <wenchen@databricks.com> Closes apache#18907 from cloud-fan/bug.
cloud-fan
added a commit
to cloud-fan/spark
that referenced
this pull request
Aug 16, 2017
…hema in table properties This is a follow-up of apache#15900 , to fix one more bug: When table schema is empty and need to be inferred at runtime, we should not resolve parent plans before the schema has been inferred, or the parent plans will be resolved against an empty schema and may get wrong result for something like `select *` The fix logic is: introduce `UnresolvedCatalogRelation` as a placeholder. Then we replace it with `LogicalRelation` or `HiveTableRelation` during analysis, so that it's guaranteed that we won't resolve parent plans until the schema has been inferred. regression test Author: Wenchen Fan <wenchen@databricks.com> Closes apache#18907 from cloud-fan/bug.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Before Spark 2.1, users can create an external data source table without schema, and we will infer the table schema at runtime. In Spark 2.1, we decided to infer the schema when the table was created, so that we don't need to infer it again and again at runtime.
This is a good improvement, but we should still respect and support old tables which doesn't store table schema in metastore.
How was this patch tested?
regression test.