Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. #2003

Merged
merged 20 commits into from
May 11, 2020

Conversation

aajisaka
Copy link
Member

@aajisaka aajisaka commented May 8, 2020

JIRA: https://issues.apache.org/jira/browse/HADOOP-16768

Note that perhaps there is the same issue in other compressors.

I ran the tests in both CentOS 7 and Ubuntu 18.04.

@iwasakims
Copy link
Member

@aajisaka I observed the same issue on CentOS 8 as reported in HADOOP-17034. While the patch fixed the error of TestSnappyCompressorDecompressor, TestCompressorDecompressor still fails due to NullPointerException. There was no issue on CentOS 7 with and without the patch.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 39s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 18m 55s trunk passed
+1 💚 compile 17m 2s trunk passed
+1 💚 checkstyle 0m 52s trunk passed
+1 💚 mvnsite 1m 27s trunk passed
+1 💚 shadedclient 16m 41s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 4s trunk passed
+0 🆗 spotbugs 2m 12s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 2m 8s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 51s the patch passed
+1 💚 compile 16m 16s the patch passed
+1 💚 javac 16m 16s the patch passed
-0 ⚠️ checkstyle 0m 52s hadoop-common-project/hadoop-common: The patch generated 1 new + 25 unchanged - 0 fixed = 26 total (was 25)
+1 💚 mvnsite 1m 28s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 4s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 4s the patch passed
+1 💚 findbugs 2m 15s the patch passed
_ Other Tests _
-1 ❌ unit 9m 11s hadoop-common in the patch passed.
+1 💚 asflicense 0m 54s The patch does not generate ASF License warnings.
106m 59s
Reason Tests
Failed junit tests hadoop.io.compress.TestCompressorDecompressor
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/1/artifact/out/Dockerfile
GITHUB PR #2003
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux f17e691c35b6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / c784ba3
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/1/testReport/
Max. process+thread count 1718 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/1/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@aajisaka
Copy link
Member Author

aajisaka commented May 8, 2020

Thanks @iwasakims for the testing. Sorry I missed TestCompressorDecompressor failure on Ubuntu 18.04. I fixed the test.

@aajisaka
Copy link
Member Author

aajisaka commented May 8, 2020

In addition, I refactored the test to print the following error before fixing this failure:

[ERROR] Failures: 
[ERROR]   TestCompressorDecompressor.testCompressorDecompressor:69  Expected to find 'testCompressorDecompressor error !!!' but got unexpected exception: java.lang.IndexOutOfBoundsException
	at java.nio.Buffer.checkBounds(Buffer.java:567)
	at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:264)
	at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:238)
	at org.apache.hadoop.io.compress.CompressDecompressTester$CompressionTestStrategy$2.assertCompression(CompressDecompressTester.java:305)

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 40s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 20m 20s trunk passed
+1 💚 compile 17m 27s trunk passed
+1 💚 checkstyle 0m 49s trunk passed
+1 💚 mvnsite 1m 23s trunk passed
+1 💚 shadedclient 16m 12s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 2s trunk passed
+0 🆗 spotbugs 2m 8s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 2m 5s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 50s the patch passed
+1 💚 compile 17m 0s the patch passed
+1 💚 javac 17m 0s the patch passed
-0 ⚠️ checkstyle 0m 48s hadoop-common-project/hadoop-common: The patch generated 1 new + 53 unchanged - 0 fixed = 54 total (was 53)
+1 💚 mvnsite 1m 25s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 13m 45s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 57s the patch passed
+1 💚 findbugs 2m 16s the patch passed
_ Other Tests _
+1 💚 unit 9m 29s hadoop-common in the patch passed.
+1 💚 asflicense 0m 46s The patch does not generate ASF License warnings.
108m 26s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/2/artifact/out/Dockerfile
GITHUB PR #2003
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 211ba2aa21a4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / c784ba3
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/2/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/2/testReport/
Max. process+thread count 2445 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/2/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 26s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 22m 8s trunk passed
+1 💚 compile 18m 26s trunk passed
+1 💚 checkstyle 0m 47s trunk passed
+1 💚 mvnsite 1m 30s trunk passed
+1 💚 shadedclient 18m 1s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 55s trunk passed
+0 🆗 spotbugs 2m 10s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 2m 8s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 51s the patch passed
+1 💚 compile 17m 29s the patch passed
+1 💚 javac 17m 29s the patch passed
+1 💚 checkstyle 0m 45s the patch passed
+1 💚 mvnsite 1m 23s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 28s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 54s the patch passed
+1 💚 findbugs 2m 16s the patch passed
_ Other Tests _
-1 ❌ unit 9m 33s hadoop-common in the patch passed.
+1 💚 asflicense 0m 46s The patch does not generate ASF License warnings.
115m 42s
Reason Tests
Failed junit tests hadoop.fs.ftp.TestFTPFileSystem
hadoop.ha.TestZKFailoverController
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/3/artifact/out/Dockerfile
GITHUB PR #2003
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 92411b9a83c3 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / c784ba3
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/3/testReport/
Max. process+thread count 1367 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/3/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@aajisaka
Copy link
Member Author

Added testSnappyCompressDecompressWithSmallBuffer. This test case throws IndexOutOfBoundsException if n = Math.min(n, len); is removed.

@aajisaka aajisaka changed the title HADOOP-16768. SnappyCompressor wrongly truncates the compressed data HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. May 11, 2020
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 18s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 21m 35s trunk passed
+1 💚 compile 18m 5s trunk passed
+1 💚 checkstyle 0m 47s trunk passed
+1 💚 mvnsite 1m 24s trunk passed
+1 💚 shadedclient 17m 40s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 56s trunk passed
+0 🆗 spotbugs 2m 6s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 2m 5s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 52s the patch passed
+1 💚 compile 19m 35s the patch passed
+1 💚 javac 19m 35s the patch passed
+1 💚 checkstyle 0m 48s the patch passed
+1 💚 mvnsite 1m 24s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 41s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 1s the patch passed
+1 💚 findbugs 2m 42s the patch passed
_ Other Tests _
+1 💚 unit 10m 10s hadoop-common in the patch passed.
+1 💚 asflicense 0m 47s The patch does not generate ASF License warnings.
117m 51s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/4/artifact/out/Dockerfile
GITHUB PR #2003
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 9c9d4d3b0776 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / aab9e0b
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/4/testReport/
Max. process+thread count 2265 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/4/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 22m 28s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 19m 18s trunk passed
+1 💚 compile 17m 4s trunk passed
+1 💚 checkstyle 0m 54s trunk passed
+1 💚 mvnsite 1m 27s trunk passed
+1 💚 shadedclient 16m 36s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 1s trunk passed
+0 🆗 spotbugs 2m 8s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 2m 6s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 50s the patch passed
+1 💚 compile 16m 16s the patch passed
+1 💚 javac 16m 16s the patch passed
-0 ⚠️ checkstyle 0m 53s hadoop-common-project/hadoop-common: The patch generated 2 new + 53 unchanged - 0 fixed = 55 total (was 53)
+1 💚 mvnsite 1m 27s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 13m 57s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 4s the patch passed
+1 💚 findbugs 2m 14s the patch passed
_ Other Tests _
+1 💚 unit 9m 10s hadoop-common in the patch passed.
+1 💚 asflicense 0m 55s The patch does not generate ASF License warnings.
128m 51s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/5/artifact/out/Dockerfile
GITHUB PR #2003
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 1e7c960ec6b3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / aab9e0b
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/5/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/5/testReport/
Max. process+thread count 3251 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/5/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Member

@iwasakims iwasakims left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. The fix worked on CentOS 8 too. Thanks, @aajisaka.

@iwasakims
Copy link
Member

Oops. You can address checkstyle warnings before merge.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 34s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 19m 0s trunk passed
+1 💚 compile 16m 59s trunk passed
+1 💚 checkstyle 0m 52s trunk passed
+1 💚 mvnsite 1m 26s trunk passed
+1 💚 shadedclient 16m 24s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 6s trunk passed
+0 🆗 spotbugs 2m 8s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 2m 5s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 51s the patch passed
+1 💚 compile 16m 12s the patch passed
+1 💚 javac 16m 12s the patch passed
+1 💚 checkstyle 0m 54s the patch passed
+1 💚 mvnsite 1m 28s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 13m 52s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 4s the patch passed
+1 💚 findbugs 2m 13s the patch passed
_ Other Tests _
+1 💚 unit 9m 13s hadoop-common in the patch passed.
+1 💚 asflicense 0m 55s The patch does not generate ASF License warnings.
106m 10s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/6/artifact/out/Dockerfile
GITHUB PR #2003
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 5b285ba109b3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / aab9e0b
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/6/testReport/
Max. process+thread count 2872 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2003/6/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@aajisaka aajisaka merged commit 328eae9 into apache:trunk May 11, 2020
@aajisaka
Copy link
Member Author

Thank you @iwasakims!

@aajisaka aajisaka deleted the HADOOP-16768 branch May 11, 2020 05:44
aajisaka added a commit that referenced this pull request May 11, 2020
…pressed data is always smaller than the input data. (#2003)

(cherry picked from commit 328eae9)
aajisaka added a commit to aajisaka/hadoop that referenced this pull request Aug 4, 2021
…pressed data is always smaller than the input data. (apache#2003)

(cherry picked from commit 328eae9)
aajisaka added a commit that referenced this pull request Aug 4, 2021
…pressed data is always smaller than the input data. (#3264)

* HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003)

(cherry picked from commit 328eae9)

* Add assertj-core dependency to hadoop-common for test
* Bump assertj-core version to 3.12.2 to use assertThat.hasSizeGreaterThan
xinglin pushed a commit to xinglin/hadoop that referenced this pull request Apr 21, 2022
…pressed data is always smaller than the input data. (apache#2003)

(cherry picked from commit 328eae9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants