Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-11413][BUILD] Bump joda-time version to 2.9 for java 8 and s3 #9379

Closed
wants to merge 1 commit into from

Conversation

@yongjiaw
Copy link
Contributor

yongjiaw commented Oct 30, 2015

It's a known issue that joda-time before 2.8.1 is incompatible with java 1.8u60 or later, which causes s3 request to fail. This affects Spark when using s3 as data source.
aws/aws-sdk-java#444

It's a known issue that joda-time before 2.8.1 is incompatible with java 1.8u60 or later, which causes s3 request to fail. This affects Spark when using s3 as data source.
aws/aws-sdk-java#444
@srowen

This comment has been minimized.

Copy link
Member

srowen commented Oct 30, 2015

ok to test

@SparkQA

This comment has been minimized.

Copy link

SparkQA commented Oct 30, 2015

Test build #44689 has finished for PR 9379 at commit bfeb6f4.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * public abstract class MemoryConsumer\n * final class ShuffleExternalSorter extends MemoryConsumer\n * public final class BytesToBytesMap extends MemoryConsumer\n * public final class MapIterator implements Iterator<Location>\n * public final class UnsafeExternalSorter extends MemoryConsumer\n * class SpillableIterator extends UnsafeSorterIterator\n * public final class UnsafeSorterSpillReader extends UnsafeSorterIterator\n * public final class UnsafeSorterSpillWriter\n
@SparkQA

This comment has been minimized.

Copy link

SparkQA commented Oct 30, 2015

Test build #1963 has finished for PR 9379 at commit bfeb6f4.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@SparkQA

This comment has been minimized.

Copy link

SparkQA commented Oct 30, 2015

Test build #1964 has finished for PR 9379 at commit bfeb6f4.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@srowen

This comment has been minimized.

Copy link
Member

srowen commented Oct 30, 2015

@yongjiaw do you see this failure locally? Some of these are spurious failures but I can't quite make out if there's a real one here.

@yongjiaw

This comment has been minimized.

Copy link
Contributor Author

yongjiaw commented Oct 31, 2015

@srowen On my local machine, running ./dev/run-tests also fails but for different tests.
But it should be unrelated to the change, because even I run it with the unmodified master branch, it still fails the network-shuffle test.
This is the last few lines of the console output.

[info] Test org.apache.spark.mllib.regression.JavaLassoSuite.runLassoUsingConstructor started
[info] Test org.apache.spark.mllib.regression.JavaLassoSuite.runLassoUsingStaticMethods started
[info] Test run finished: 0 failed, 0 ignored, 2 total, 6.108s
-- org.jblas INFO Deleting /local_disk/git/yongjiaw_spark/target/tmp/jblas3269426321238629267/libjblas.so
-- org.jblas INFO Deleting /local_disk/git/yongjiaw_spark/target/tmp/jblas3269426321238629267/libjblas_arch_flavor.so
-- org.jblas INFO Deleting /local_disk/git/yongjiaw_spark/target/tmp/jblas3269426321238629267
[info] ScalaTest
[info] Run completed in 1 hour, 23 minutes.
[info] Total number of tests run: 709
[info] Suites: completed 141, aborted 0
[info] Tests: succeeded 709, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 818, Failed 0, Errors 0, Passed 818
[error] (network-shuffle/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] (network-common/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] (sql/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 5157 s, completed Oct 31, 2015 2:47:48 PM
@SparkQA

This comment has been minimized.

Copy link

SparkQA commented Oct 31, 2015

Test build #1966 has finished for PR 9379 at commit bfeb6f4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@srowen

This comment has been minimized.

Copy link
Member

srowen commented Nov 1, 2015

Seems OK to me in that the release notes do not indicate any incompatible changes, and the tests pass, and it fixes some bugs. We don't actually use Joda time in Spark; it appears to be added for Hive.

@asfgit asfgit closed this in ea4a3e7 Nov 2, 2015
@srowen

This comment has been minimized.

Copy link
Member

srowen commented Nov 2, 2015

merged to master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.