Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-11236][CORE] Update Tachyon dependency from 0.7.1 -> 0.8.0. #9204

Closed
wants to merge 3 commits into from
Closed

[SPARK-11236][CORE] Update Tachyon dependency from 0.7.1 -> 0.8.0. #9204

wants to merge 3 commits into from

Conversation

calvinjia
Copy link
Contributor

Upgrades the tachyon-client version to the latest release.

No new dependencies are added and no spark facing APIs are changed. The removal of the tachyon-underfs-s3 exclusion will enable users to use S3 out of the box and there are no longer any additional external dependencies added by the module.

@SparkQA
Copy link

SparkQA commented Oct 21, 2015

Test build #44084 has finished for PR 9204 at commit f7edb37.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * case class ClassEncoder[T](\n * case class CreateRow(children: Seq[Expression]) extends Expression\n

@calvinjia
Copy link
Contributor Author

@JoshRosen could you take a look? Thanks!

@haoyuan
Copy link
Contributor

haoyuan commented Oct 23, 2015

LGTM

@JoshRosen @rxin

@calvinjia
Copy link
Contributor Author

@pwendell @srowen could either of you take a look? Thanks!

@asfgit asfgit closed this in 4f5e60c Oct 29, 2015
@pwendell
Copy link
Contributor

@haoyuan hey HY - can you not merge build related patches without getting for feedback from one of the build maintainers (me or @srowen)? This patch makes changes to Spark's dependency graph that need to be audited carefully because they affect all users. There is discussion of the maintainer/review process here:

https://cwiki.apache.org/confluence/display/SPARK/Committers#Committers-ReviewProcessandMaintainers

We are all at Spark Summit this week, so there is some delay in getting review.

I did a post hoc review and it appears this does not change the contents of the assembly jar. So I think it is okay.

Separately, it would be good to spin tachyon support out into a package so these changes do not need to go through the upstream review process.

@yhuai
Copy link
Contributor

yhuai commented Oct 30, 2015

This one breaks hadoop 1 tests (see https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/Spark-Master-SBT/3888/AMPLAB_JENKINS_BUILD_PROFILE=hadoop1.0,label=spark-test/#showFailuresLink). I am going to revert it.

@yhuai
Copy link
Contributor

yhuai commented Oct 31, 2015

Reverted by e8ec2a7.

@calvinjia calvinjia mentioned this pull request Nov 1, 2015
Closed
asfgit pushed a commit that referenced this pull request Nov 3, 2015
@calvinjia @pwendell
[SPARK-11236] [TEST-MAVEN] [TEST-HADOOP1.0] [CORE] Update Tachyon dep…
476f434 
…endency 0.7.1 -> 0.8.1

This is a reopening of #9204 which failed hadoop1 sbt tests.

With the original PR, a classpath issue would occur due to the MIMA plugin pulling in hadoop-2.2 dependencies regardless of the hadoop version when building the `oldDeps` project. These affect the hadoop1 sbt build because they are placed in `lib_managed` and Tachyon 0.8.0's default hadoop version is 2.2.

Author: Calvin Jia <jia.calvin@gmail.com>

Closes #9395 from calvinjia/spark-11236.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@calvinjia @SparkQA @haoyuan @pwendell @yhuai