[SPARK-15901] [SQL] [TEST] Verification of CONVERT_METASTORE_ORC and CONVERT_METASTORE_PARQUET#13622
[SPARK-15901] [SQL] [TEST] Verification of CONVERT_METASTORE_ORC and CONVERT_METASTORE_PARQUET#13622gatorsmile wants to merge 2 commits intoapache:masterfrom
Conversation
|
Test build #60358 has finished for PR 13622 at commit
|
|
Test build #60369 has finished for PR 13622 at commit
|
| } | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
It is a bit costy to write a end to end test like this, how about adding an Unit test using SimpleAnalyzer?
There was a problem hiding this comment.
Thank you for your review! @clockfly
I like your suggestion, but the conf CONVERT_METASTORE_ORC and CONVERT_METASTORE_PARQUET are not available in CatalystConf and SimpleCatalystConf. I am hesitant to change the source codes for improving test case coverage.
Actually, I am not sure if we should keep these rules in the Analyzer in the future release. These two rules are not used for resolution. Instead, they are for performance enhancement. Ideally, I think they are Optimizer rules.
There was a problem hiding this comment.
Having Parquet/ORC conversions in the optimizer is also reasonable, but doesn't seem to bring much benefits. Furthermore, it also requires us to refactor the optimizer to support Hive specific rules.
|
LGTM, merging to master and branch-2.0. Thanks! |
…NVERT_METASTORE_PARQUET #### What changes were proposed in this pull request? So far, we do not have test cases for verifying whether the external parameters `HiveUtils .CONVERT_METASTORE_ORC` and `HiveUtils.CONVERT_METASTORE_PARQUET` properly works when users use non-default values. This PR is to add such test cases for avoiding potential regression. #### How was this patch tested? N/A Author: gatorsmile <gatorsmile@gmail.com> Closes #13622 from gatorsmile/addTestCase4parquetOrcConversion. (cherry picked from commit 0992573) Signed-off-by: Cheng Lian <lian@databricks.com>
…NVERT_METASTORE_PARQUET So far, we do not have test cases for verifying whether the external parameters `HiveUtils .CONVERT_METASTORE_ORC` and `HiveUtils.CONVERT_METASTORE_PARQUET` properly works when users use non-default values. This PR is to add such test cases for avoiding potential regression. N/A Author: gatorsmile <gatorsmile@gmail.com> Closes apache#13622 from gatorsmile/addTestCase4parquetOrcConversion.
4 participants
What changes were proposed in this pull request?
So far, we do not have test cases for verifying whether the external parameters
HiveUtils .CONVERT_METASTORE_ORCandHiveUtils.CONVERT_METASTORE_PARQUETproperly works when users use non-default values. This PR is to add such test cases for avoiding potential regression.How was this patch tested?
N/A