[SPARK-15959] [SQL] Add the support of hive.metastore.warehouse.dir back#13679
[SPARK-15959] [SQL] Add the support of hive.metastore.warehouse.dir back#13679yhuai wants to merge 4 commits intoapache:masterfrom yhuai:hiveWarehouseDir
Conversation
|
cc @rxin @gatorsmile @andrewor14 for review |
|
Can we put this in sql/core so it is not Hive specific? |
| } | ||
|
|
||
| logInfo(s"Warehouse path is '${tempConf.warehousePath}'.") | ||
| } |
There was a problem hiding this comment.
@rxin How about this version? This block is moved to SharedState from HiveSharedState.
There was a problem hiding this comment.
You know, I like this change. : )
|
Test build #60553 has finished for PR 13679 at commit
|
|
Have a very general question. Based on my understanding, we should introduce our own If my above understanding is right, should we create a more general way for handling all the similar cases? Thank you! |
|
Test build #60558 has finished for PR 13679 at commit
|
|
@gatorsmile What are other confs? |
|
For example, |
|
I am not sure if we need those at all... |
|
LGTM - merging in master/2.0. |
## What changes were proposed in this pull request? This PR adds the support of conf `hive.metastore.warehouse.dir` back. With this patch, the way of setting the warehouse dir is described as follows: * If `spark.sql.warehouse.dir` is set, `hive.metastore.warehouse.dir` will be automatically set to the value of `spark.sql.warehouse.dir`. The warehouse dir is effectively set to the value of `spark.sql.warehouse.dir`. * If `spark.sql.warehouse.dir` is not set but `hive.metastore.warehouse.dir` is set, `spark.sql.warehouse.dir` will be automatically set to the value of `hive.metastore.warehouse.dir`. The warehouse dir is effectively set to the value of `hive.metastore.warehouse.dir`. * If neither `spark.sql.warehouse.dir` nor `hive.metastore.warehouse.dir` is set, `hive.metastore.warehouse.dir` will be automatically set to the default value of `spark.sql.warehouse.dir`. The warehouse dir is effectively set to the default value of `spark.sql.warehouse.dir`. ## How was this patch tested? `set hive.metastore.warehouse.dir` in `HiveSparkSubmitSuite`. JIRA: https://issues.apache.org/jira/browse/SPARK-15959 Author: Yin Huai <yhuai@databricks.com> Closes #13679 from yhuai/hiveWarehouseDir. (cherry picked from commit e1585cc) Signed-off-by: Reynold Xin <rxin@databricks.com>
4 participants
What changes were proposed in this pull request?
This PR adds the support of conf
hive.metastore.warehouse.dirback. With this patch, the way of setting the warehouse dir is described as follows:spark.sql.warehouse.diris set,hive.metastore.warehouse.dirwill be automatically set to the value ofspark.sql.warehouse.dir. The warehouse dir is effectively set to the value ofspark.sql.warehouse.dir.spark.sql.warehouse.diris not set buthive.metastore.warehouse.diris set,spark.sql.warehouse.dirwill be automatically set to the value ofhive.metastore.warehouse.dir. The warehouse dir is effectively set to the value ofhive.metastore.warehouse.dir.spark.sql.warehouse.dirnorhive.metastore.warehouse.diris set,hive.metastore.warehouse.dirwill be automatically set to the default value ofspark.sql.warehouse.dir. The warehouse dir is effectively set to the default value ofspark.sql.warehouse.dir.How was this patch tested?
set hive.metastore.warehouse.dirinHiveSparkSubmitSuite.JIRA: https://issues.apache.org/jira/browse/SPARK-15959