Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-17181. Handle transient stream read failures in FileSystem contract tests #2286

Merged
merged 1 commit into from Sep 9, 2020
Merged

HADOOP-17181. Handle transient stream read failures in FileSystem contract tests #2286

merged 1 commit into from Sep 9, 2020

Conversation

steveloughran
Copy link
Contributor

  • Fixes Seek test to readFully
  • Doesn't do this to the unbuffer test as it changes the test too much.
    Instead just note in error that this may be transient

@steveloughran
HADOOP-17181. transient stream read failures
02e9b71 
* Fixes Seek test to readFully
* Doesn't do this to the unbuffer test as it changes the test too much.
  Instead just note in error that this may be transient

Change-Id: I28d52e57192dd33dab3d25406e1e8ad12aa68572
@steveloughran
Copy link
Contributor Author

tested: AWS london

liuml07
liuml07 approved these changes Sep 8, 2020
View reviewed changes
Copy link
Member

@liuml07 liuml07 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Let's get this fix in and hopefully the error should be gone.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 35s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 30m 59s trunk passed
+1 💚 compile 19m 24s trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 compile 16m 46s trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚 checkstyle 0m 52s trunk passed
+1 💚 mvnsite 1m 29s trunk passed
+1 💚 shadedclient 16m 38s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 39s trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 javadoc 1m 33s trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+0 🆗 spotbugs 2m 15s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 2m 13s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 51s the patch passed
+1 💚 compile 18m 41s the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 javac 18m 41s the patch passed
+1 💚 compile 16m 55s the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚 javac 16m 55s the patch passed
+1 💚 checkstyle 0m 52s the patch passed
+1 💚 mvnsite 1m 26s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 9s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 38s the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 javadoc 1m 32s the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚 findbugs 2m 20s the patch passed
_ Other Tests _
+1 💚 unit 9m 36s hadoop-common in the patch passed.
+1 💚 asflicense 0m 55s The patch does not generate ASF License warnings.
161m 44s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2286/1/artifact/out/Dockerfile
GITHUB PR #2286
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 41d49a58369e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 0d85515
Default Java Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2286/1/testReport/
Max. process+thread count 3326 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2286/1/console
versions git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@liuml07
Copy link
Member

liuml07 commented Sep 9, 2020 •
edited

The PR title (and commit subject) can be a bit more verbose like:

Fix transient stream read failures in FileSystem contract tests

@steveloughran steveloughran changed the title HADOOP-17181. transient stream read failures HADOOP-17181. Fix transient stream read failures in FileSystem contract tests Sep 9, 2020
@steveloughran steveloughran merged commit 2029556 into apache:trunk Sep 9, 2020
@steveloughran
Copy link
Contributor Author

merged. Use "handle" not "fix" as the unbuffer tests may still see it.

@steveloughran steveloughran changed the title HADOOP-17181. Fix transient stream read failures in FileSystem contract tests HADOOP-17181. Handle transient stream read failures in FileSystem contract tests Sep 9, 2020
asfgit pushed a commit that referenced this pull request Sep 9, 2020
@steveloughran
HADOOP-17181. Handle transient stream read failures in FileSystem con…
262c575 
…tract tests (#2286)

Contributed by Steve Loughran.

* Fixes AbstractContractSeekTest test to use readFully
* Doesn't do this to AbstractContractUnbufferTest test as it changes the test too much.
Instead just notes in the error that this may be transient

The issue is that read(buffer) doesn't guarantee that the buffer is filled, only that it will
read up to a point, and that may be just be the amount of data left in the TCP packet.
readFully corrects for this, but using it in the unbuffer test runs the risk that what
is tested for in terms of unbuffering doesn't actually get validated.

Change-Id: I046eadb69b80ba0aac468b354c82c4d510dc3699
@steveloughran steveloughran deleted the s3/HADOOP-17181-ITestS3AContractUnbuffer branch October 15, 2021 19:43
jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023
@steveloughran
HADOOP-17181. Handle transient stream read failures in FileSystem con…
67c4eda 
…tract tests (apache#2286)

Contributed by Steve Loughran.

* Fixes AbstractContractSeekTest test to use readFully
* Doesn't do this to AbstractContractUnbufferTest test as it changes the test too much.
Instead just notes in the error that this may be transient

The issue is that read(buffer) doesn't guarantee that the buffer is filled, only that it will
read up to a point, and that may be just be the amount of data left in the TCP packet.
readFully corrects for this, but using it in the unbuffer test runs the risk that what
is tested for in terms of unbuffering doesn't actually get validated.

Change-Id: Ia1587eaa4892961136ebb7c945f951e3b9825a49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liuml07 liuml07 liuml07 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@steveloughran @hadoop-yetus @liuml07