[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled #1891

sarutak · 2014-08-11T11:57:59Z

No description provided.

… before shutdown hook of o.a.h.f.FileSystem

SparkQA · 2014-08-11T11:59:39Z

QA tests have started for PR 1891. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18308/consoleFull

SparkQA · 2014-08-11T12:00:19Z

QA results for PR 1891:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18308/consoleFull

SparkQA · 2014-08-11T12:49:40Z

QA tests have started for PR 1891. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18309/consoleFull

SparkQA · 2014-08-11T14:03:34Z

QA results for PR 1891:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18309/consoleFull

liancheng · 2014-08-11T16:06:30Z

LGTM, thanks!

andrewor14 · 2014-08-11T18:17:54Z

...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala

@@ -116,13 +118,17 @@ private[hive] object SparkSQLCLIDriver {
    SessionState.start(sessionState)

    // Clean up after we exit
-    Runtime.getRuntime.addShutdownHook(
+    /**
+     This should be executed before shutdown hook of


small nit: this line is missing a * before "This"

SparkQA · 2014-08-11T18:24:44Z

QA tests have started for PR 1891. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18315/consoleFull

SparkQA · 2014-08-11T19:30:11Z

QA results for PR 1891:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18315/consoleFull

liancheng · 2014-08-12T03:07:01Z

Jenkins, retest this please.

liancheng · 2014-08-12T05:20:40Z

Embarrassing... I don't know why Jenkins doesn't respond me, he used to :(

andrewor14 · 2014-08-12T05:47:51Z

test this please, let's see if he likes me :)

SparkQA · 2014-08-12T05:49:51Z

QA tests have started for PR 1891. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18360/consoleFull

SparkQA · 2014-08-12T06:57:41Z

QA results for PR 1891:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18360/consoleFull

liancheng · 2014-08-12T07:44:54Z

The build failure was caused by PySpark. (Let me try again...)

liancheng · 2014-08-12T07:45:03Z

test this please

SparkQA · 2014-08-12T07:49:59Z

QA tests have started for PR 1891. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18365/consoleFull

sarutak · 2014-08-12T08:07:22Z

Thanks a lot for triggering test...

SparkQA · 2014-08-12T08:59:38Z

QA results for PR 1891:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18365/consoleFull

sarutak · 2014-08-12T10:15:03Z

Ah... It's strange. I think there are no test using SparkSQLCLIDriver right?

liancheng · 2014-08-13T09:26:05Z

Build failure was caused by PySpark again :(

liancheng · 2014-08-13T09:26:12Z

retest this please

SparkQA · 2014-08-13T09:29:54Z

QA tests have started for PR 1891. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18437/consoleFull

liancheng · 2014-08-13T09:32:12Z

Oh, actually the build failure was caused by a Spark SQL Python API test case. It seems that we are comparing floating point numbers directly:

Expected:
    [Row(byte1=126, byte2=-127, short1=-32767, short2=32766, int=2147483646, float=2.1)]
Got:
    [Row(byte1=126, byte2=-127, short1=-32767, short2=32766, int=2147483646, float=2.1000000000000001)]

This should be fixed... Sorry @sarutak, we'll come back to this PR later, thanks for the patience.

SparkQA · 2014-08-13T10:38:44Z

QA results for PR 1891:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18437/consoleFull

liancheng · 2014-08-13T15:12:24Z

@sarutak Just FYI, @yhuai is taking care of the floating-point issue. Jenkins was pretty crazy during the last 12 hours. We're trying to calm him down...

marmbrus · 2014-08-13T22:31:21Z

Jenkins, test this please.

SparkQA · 2014-08-13T22:35:05Z

QA tests have started for PR 1891. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18482/consoleFull

SparkQA · 2014-08-14T00:01:57Z

QA results for PR 1891:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18482/consoleFull

marmbrus · 2014-08-14T00:43:33Z

Thanks! I've merged this to master and 1.1

…ogging is enabled Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes #1891 from sarutak/SPARK-2970 and squashes the following commits: 4a2d2fe [Kousuke Saruta] Modified comment style 8bd833c [Kousuke Saruta] Modified style 6c0997c [Kousuke Saruta] Modified the timing of shutdown hook execution. It should be executed before shutdown hook of o.a.h.f.FileSystem (cherry picked from commit 905dc4b) Signed-off-by: Michael Armbrust <michael@databricks.com>

liancheng · 2014-08-15T06:37:30Z

Sorry, didn't realize that ShutdownHookManager is only available in Hadoop 2. Compilation fails when building Spark with Hadoop 1. Filed SPARK-3062 to track this issue.

@marmbrus Maybe we should revert this PR first for safe.

sarutak · 2014-08-15T17:20:41Z

@liancheng @marmbrus So sorry for my fault.
Now I had a compromised solution on #1970.

…n EventLogging is enabled" Revert #1891 due to issues with hadoop 1 compatibility. Author: Michael Armbrust <michael@databricks.com> Closes #2007 from marmbrus/revert1891 and squashes the following commits: 68706c0 [Michael Armbrust] Revert "[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled" (cherry picked from commit 5ecb08e) Signed-off-by: Michael Armbrust <michael@databricks.com>

…n EventLogging is enabled" Revert #1891 due to issues with hadoop 1 compatibility. Author: Michael Armbrust <michael@databricks.com> Closes #2007 from marmbrus/revert1891 and squashes the following commits: 68706c0 [Michael Armbrust] Revert "[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled"

…n when EventLogging is enabled #1891 was to avoid IOException when EventLogging is enabled. The solution used ShutdownHookManager but it was defined only Hadoop 2.x. Hadoop 1.x don't have ShutdownHookManager so #1891 doesn't compile on Hadoop 1.x Now, I had a compromised solution for both Hadoop 1.x and 2.x. Only for FileLogger, an unique FileSystem object is created. Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes #1970 from sarutak/SPARK-2970 and squashes the following commits: 240c91e [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-2970 0e7b45d [Kousuke Saruta] Revert "[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled" e1262ec [Kousuke Saruta] Modified Filelogger to use unique FileSystem instance

…ogging is enabled Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes apache#1891 from sarutak/SPARK-2970 and squashes the following commits: 4a2d2fe [Kousuke Saruta] Modified comment style 8bd833c [Kousuke Saruta] Modified style 6c0997c [Kousuke Saruta] Modified the timing of shutdown hook execution. It should be executed before shutdown hook of o.a.h.f.FileSystem

…n EventLogging is enabled" Revert apache#1891 due to issues with hadoop 1 compatibility. Author: Michael Armbrust <michael@databricks.com> Closes apache#2007 from marmbrus/revert1891 and squashes the following commits: 68706c0 [Michael Armbrust] Revert "[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled"

…n when EventLogging is enabled apache#1891 was to avoid IOException when EventLogging is enabled. The solution used ShutdownHookManager but it was defined only Hadoop 2.x. Hadoop 1.x don't have ShutdownHookManager so apache#1891 doesn't compile on Hadoop 1.x Now, I had a compromised solution for both Hadoop 1.x and 2.x. Only for FileLogger, an unique FileSystem object is created. Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes apache#1970 from sarutak/SPARK-2970 and squashes the following commits: 240c91e [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-2970 0e7b45d [Kousuke Saruta] Revert "[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled" e1262ec [Kousuke Saruta] Modified Filelogger to use unique FileSystem instance

https://github.pie.apple.com/IPR/apache-incubator-iceberg/compare/IPR:9a2d360...IPR:48834b0 Internal: Change Default Optimize Threshold Internal (Boson): Bump Boson version to 0.3.23 and remove the fallbac… Internal(Boson): Populate spark.boson.exceptionOnDatetimeRebase to Bo… Releases Apple Iceberg 1.3.0.5 (apache#1152)

Modified the timing of shutdown hook execution. It should be executed…

6c0997c

… before shutdown hook of o.a.h.f.FileSystem

Modified style

8bd833c

andrewor14 reviewed Aug 11, 2014
View reviewed changes

Modified comment style

4a2d2fe

asfgit closed this in 905dc4b Aug 14, 2014

sarutak mentioned this pull request Aug 15, 2014

[SPARK-3062] [SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled #1970

Closed

marmbrus mentioned this pull request Aug 18, 2014

Revert "[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled" #2007

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled #1891

[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled #1891

sarutak commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

liancheng commented Aug 11, 2014

andrewor14 Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

liancheng commented Aug 12, 2014

liancheng commented Aug 12, 2014

andrewor14 commented Aug 12, 2014

SparkQA commented Aug 12, 2014

SparkQA commented Aug 12, 2014

liancheng commented Aug 12, 2014

liancheng commented Aug 12, 2014

SparkQA commented Aug 12, 2014

sarutak commented Aug 12, 2014

SparkQA commented Aug 12, 2014

sarutak commented Aug 12, 2014

liancheng commented Aug 13, 2014

liancheng commented Aug 13, 2014

SparkQA commented Aug 13, 2014

liancheng commented Aug 13, 2014

SparkQA commented Aug 13, 2014

liancheng commented Aug 13, 2014

marmbrus commented Aug 13, 2014

SparkQA commented Aug 13, 2014

SparkQA commented Aug 14, 2014

marmbrus commented Aug 14, 2014

liancheng commented Aug 15, 2014

sarutak commented Aug 15, 2014

[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled #1891

[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled #1891

Conversation

sarutak commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

liancheng commented Aug 11, 2014

andrewor14 Aug 11, 2014

Choose a reason for hiding this comment

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

liancheng commented Aug 12, 2014

liancheng commented Aug 12, 2014

andrewor14 commented Aug 12, 2014

SparkQA commented Aug 12, 2014

SparkQA commented Aug 12, 2014

liancheng commented Aug 12, 2014

liancheng commented Aug 12, 2014

SparkQA commented Aug 12, 2014

sarutak commented Aug 12, 2014

SparkQA commented Aug 12, 2014

sarutak commented Aug 12, 2014

liancheng commented Aug 13, 2014

liancheng commented Aug 13, 2014

SparkQA commented Aug 13, 2014

liancheng commented Aug 13, 2014

SparkQA commented Aug 13, 2014

liancheng commented Aug 13, 2014

marmbrus commented Aug 13, 2014

SparkQA commented Aug 13, 2014

SparkQA commented Aug 14, 2014

marmbrus commented Aug 14, 2014

liancheng commented Aug 15, 2014

sarutak commented Aug 15, 2014