Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-23336][BUILD] Upgrade snappy-java to 1.1.7.1 #20510

Closed
wants to merge 2 commits into from
Closed

[SPARK-23336][BUILD] Upgrade snappy-java to 1.1.7.1 #20510

wants to merge 2 commits into from

Conversation

wangyum
Copy link
Member

@wangyum wangyum commented Feb 5, 2018 •
edited
Loading

What changes were proposed in this pull request?

This PR upgrade snappy-java from 1.1.2.6 to 1.1.7.1.
1.1.7.1 release notes:

  • Improved performance for big-endian architecture
  • The other performance improvement in snappy-1.1.5

1.1.4 release notes:

  • Fix a 1% performance regression when snappy is used in PIE executables.
  • Improve compression performance by 5%.
  • Improve decompression performance by 20%.

More details:
https://github.com/xerial/snappy-java/blob/master/Milestone.md

How was this patch tested?

manual tests

smurakozi
smurakozi reviewed Feb 5, 2018
View reviewed changes
Copy link
Contributor

@smurakozi smurakozi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this change is ok.

@SparkQA
Copy link

SparkQA commented Feb 5, 2018

Test build #87067 has finished for PR 20510 at commit 1055afc.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Feb 5, 2018

Retest this please.

dongjoon-hyun
dongjoon-hyun reviewed Feb 5, 2018
View reviewed changes
dev/deps/spark-deps-hadoop-2.6 Outdated
@@ -182,7 +182,7 @@ slf4j-api-1.7.16.jar
slf4j-log4j12-1.7.16.jar
snakeyaml-1.15.jar
snappy-0.2.jar
snappy-java-1.1.2.6.jar
snappy-java-1.1.4.jar
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason to use 1.1.4 instead of 1.1.7.1?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@SparkQA
Copy link

SparkQA commented Feb 5, 2018

Test build #87072 has finished for PR 20510 at commit 1055afc.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member

srowen commented Feb 5, 2018

This looks like a safe upgrade, though it's always a little risky because components like Hadoop use this too, and not the same version. Tests will help show whether there's an immediate issue.

@SparkQA
Copy link

SparkQA commented Feb 5, 2018

Test build #87077 has finished for PR 20510 at commit 7565e29.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Feb 5, 2018

Retest this please.

@SparkQA
Copy link

SparkQA commented Feb 6, 2018

Test build #87080 has finished for PR 20510 at commit 7565e29.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Feb 6, 2018

The failure is due to flaky test suite.

org.apache.spark.sql.hive.client.HiveClientSuites.(It is not a test it is a sbt.testing.NestedSuiteSelector)

jenkins, retest this please.

@wangyum
Copy link
Member Author

wangyum commented Feb 6, 2018

jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Feb 6, 2018

Test build #87090 has finished for PR 20510 at commit 7565e29.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Feb 6, 2018

jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Feb 6, 2018

Test build #87098 has finished for PR 20510 at commit 7565e29.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum wangyum changed the title [SPARK-23336][BUILD] Upgrade snappy-java to 1.1.4 [SPARK-23336][BUILD] Upgrade snappy-java to 1.1.7.1 Feb 6, 2018
felixcheung
felixcheung approved these changes Feb 7, 2018
View reviewed changes
Copy link
Member

@felixcheung felixcheung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG, same concerns as Sean

@srowen
Copy link
Member

srowen commented Feb 8, 2018

Merged to master

@asfgit asfgit closed this in 4df84c3 Feb 8, 2018
robert3005 pushed a commit to palantir/spark that referenced this pull request Feb 12, 2018
@wangyum
[SPARK-23336][BUILD] Upgrade snappy-java to 1.1.7.1
202a8dc 
## What changes were proposed in this pull request?

This PR upgrade snappy-java from 1.1.2.6 to 1.1.7.1.
1.1.7.1 release notes:
- Improved performance for big-endian architecture
- The other performance improvement in [snappy-1.1.5](https://github.com/google/snappy/releases/tag/1.1.5)

1.1.4 release notes:
- Fix a 1% performance regression when snappy is used in PIE executables.
- Improve compression performance by 5%.
- Improve decompression performance by 20%.

More details:
https://github.com/xerial/snappy-java/blob/master/Milestone.md

## How was this patch tested?

manual tests

Author: Yuming Wang <wgyumg@gmail.com>

Closes apache#20510 from wangyum/SPARK-23336.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kiszk kiszk kiszk left review comments

@smurakozi smurakozi smurakozi left review comments

@dongjoon-hyun dongjoon-hyun dongjoon-hyun left review comments

@felixcheung felixcheung felixcheung approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
7 participants
@wangyum @SparkQA @srowen @kiszk @smurakozi @felixcheung @dongjoon-hyun