[SPARK-20728][SQL][FOLLOWUP] Use an actionable exception message #19903

dongjoon-hyun · 2017-12-06T02:51:01Z

What changes were proposed in this pull request?

This is a follow-up of #19871 to improve an exception message.

How was this patch tested?

Pass the Jenkins.

cloud-fan · 2017-12-06T02:56:53Z

LGTM

HyukjinKwon · 2017-12-06T03:07:36Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala

@@ -602,7 +602,7 @@ object DataSource extends Logging {
                  provider1.startsWith("org.apache.spark.sql.hive.orc")) {
                  throw new AnalysisException(
                    "Hive-based ORC data source must be used with Hive support enabled. " +
-                    "Please use native ORC data source instead")
+                    "Please use native ORC data source instead by `SET spark.sql.orc.impl=native`")


Wait .. @dongjoon-hyun can we just say like by setting 'spark.sql.orc.impl' configuration to 'native'?

HyukjinKwon · 2017-12-06T03:49:24Z

LGTM

dongjoon-hyun · 2017-12-06T03:53:45Z

Thank you, @HyukjinKwon and @cloud-fan .

gatorsmile · 2017-12-06T04:23:37Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala

@@ -602,7 +602,8 @@ object DataSource extends Logging {
                  provider1.startsWith("org.apache.spark.sql.hive.orc")) {
                  throw new AnalysisException(
                    "Hive-based ORC data source must be used with Hive support enabled. " +
-                    "Please use native ORC data source instead")
+                    "Please use native ORC data source instead by setting 'spark.sql.orc.impl' " +
+                    "configuration to 'native'")


Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'

gatorsmile · 2017-12-06T04:25:19Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala

@@ -602,7 +602,8 @@ object DataSource extends Logging {
                  provider1.startsWith("org.apache.spark.sql.hive.orc")) {
                  throw new AnalysisException(
                    "Hive-based ORC data source must be used with Hive support enabled. " +


Hive built-in ORC data source must be used with Hive support enabled.

gatorsmile · 2017-12-06T04:25:37Z

Do you have a test case?

dongjoon-hyun · 2017-12-06T04:37:27Z

Yep. The test case will be updated together.

dongjoon-hyun · 2017-12-06T05:19:05Z

@cloud-fan , @HyukjinKwon , @gatorsmile . The PR is updated again.
Please let me know if I need to update more. Thank you!

SparkQA · 2017-12-06T05:32:54Z

Test build #84521 has finished for PR 19903 at commit d6cdde7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-12-06T05:45:33Z

LGTM pending Jenkins on $84528

dongjoon-hyun · 2017-12-06T05:58:01Z

Thanks, @gatorsmile .

SparkQA · 2017-12-06T06:25:34Z

Test build #84526 has finished for PR 19903 at commit ba1e3ae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-12-06T06:54:04Z

Test build #84528 has finished for PR 19903 at commit d7d268d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-12-06T06:56:04Z

Retest this please.

SparkQA · 2017-12-06T08:05:01Z

Test build #84532 has finished for PR 19903 at commit d7d268d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2017-12-06T08:07:27Z

retest this please

viirya · 2017-12-06T08:18:18Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala

@@ -601,8 +601,9 @@ object DataSource extends Logging {
                if (provider1.toLowerCase(Locale.ROOT) == "orc" ||


As suggested in other PR, we can remove this condition.

Yep. I'll remove this here since it's small.

viirya · 2017-12-06T08:19:15Z

LGTM with one minor comment.

SparkQA · 2017-12-06T10:56:12Z

Test build #84536 has finished for PR 19903 at commit d7d268d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-12-06T11:19:11Z

Test build #84541 has finished for PR 19903 at commit 9809af8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2017-12-06T11:19:42Z

Merged to master.

dongjoon-hyun · 2017-12-06T16:19:02Z

Thank you, @HyukjinKwon , @cloud-fan , @gatorsmile , @viirya !

[SPARK-20728][SQL][FOLLOWUP] Use a actionable exception message

d6cdde7

dongjoon-hyun mentioned this pull request Dec 6, 2017

[SPARK-20728][SQL] Make OrcFileFormat configurable between sql/hive and sql/core #19871

Closed

HyukjinKwon approved these changes Dec 6, 2017

View reviewed changes

HyukjinKwon reviewed Dec 6, 2017

View reviewed changes

Address comments

ba1e3ae

dongjoon-hyun changed the title ~~[SPARK-20728][SQL][FOLLOWUP] Use a actionable exception message~~ [SPARK-20728][SQL][FOLLOWUP] Use an actionable exception message Dec 6, 2017

gatorsmile reviewed Dec 6, 2017

View reviewed changes

Address comments

d7d268d

viirya reviewed Dec 6, 2017

View reviewed changes

Remove invalid condition

9809af8

asfgit closed this in fb6a922 Dec 6, 2017

dongjoon-hyun deleted the orc_exception branch December 6, 2017 16:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-20728][SQL][FOLLOWUP] Use an actionable exception message #19903

[SPARK-20728][SQL][FOLLOWUP] Use an actionable exception message #19903

dongjoon-hyun commented Dec 6, 2017 •

edited

cloud-fan commented Dec 6, 2017

HyukjinKwon Dec 6, 2017

dongjoon-hyun Dec 6, 2017

HyukjinKwon commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017

gatorsmile Dec 6, 2017

dongjoon-hyun Dec 6, 2017

gatorsmile Dec 6, 2017

dongjoon-hyun Dec 6, 2017

gatorsmile commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017

SparkQA commented Dec 6, 2017

gatorsmile commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017

SparkQA commented Dec 6, 2017

SparkQA commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017

SparkQA commented Dec 6, 2017

HyukjinKwon commented Dec 6, 2017

viirya Dec 6, 2017

dongjoon-hyun Dec 6, 2017

viirya commented Dec 6, 2017

SparkQA commented Dec 6, 2017

SparkQA commented Dec 6, 2017

HyukjinKwon commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017

		@@ -601,8 +601,9 @@ object DataSource extends Logging {
		if (provider1.toLowerCase(Locale.ROOT) == "orc" \|\|

[SPARK-20728][SQL][FOLLOWUP] Use an actionable exception message #19903

[SPARK-20728][SQL][FOLLOWUP] Use an actionable exception message #19903

Conversation

dongjoon-hyun commented Dec 6, 2017 • edited

What changes were proposed in this pull request?

How was this patch tested?

cloud-fan commented Dec 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HyukjinKwon commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gatorsmile commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017

SparkQA commented Dec 6, 2017

gatorsmile commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017

SparkQA commented Dec 6, 2017

SparkQA commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017

SparkQA commented Dec 6, 2017

HyukjinKwon commented Dec 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya commented Dec 6, 2017

SparkQA commented Dec 6, 2017

SparkQA commented Dec 6, 2017

HyukjinKwon commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017

dongjoon-hyun commented Dec 6, 2017 •

edited