Skip to content

Commit

Permalink
HADOOP-17181. Handle transient stream read failures in FileSystem con…
Browse files Browse the repository at this point in the history
…tract tests (apache#2286)

Contributed by Steve Loughran.

* Fixes AbstractContractSeekTest test to use readFully
* Doesn't do this to AbstractContractUnbufferTest test as it changes the test too much.
Instead just notes in the error that this may be transient

The issue is that read(buffer) doesn't guarantee that the buffer is filled, only that it will
read up to a point, and that may be just be the amount of data left in the TCP packet.
readFully corrects for this, but using it in the unbuffer test runs the risk that what
is tested for in terms of unbuffering doesn't actually get validated.

Change-Id: Ia1587eaa4892961136ebb7c945f951e3b9825a49
  • Loading branch information
steveloughran committed Jan 9, 2021
1 parent 71a0f05 commit 67c4eda
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -317,7 +317,7 @@ public void testPositionedBulkReadDoesntChangePosition() throws Throwable {

int v = 256;
byte[] readBuffer = new byte[v];
assertEquals(v, instream.read(128, readBuffer, 0, v));
instream.readFully(128, readBuffer, 0, v);
//have gone back
assertEquals(40000, instream.getPos());
//content is the same too
Expand Down Expand Up @@ -572,8 +572,7 @@ public void testReadSmallFile() throws Throwable {

// now read the entire file in one go
byte[] fullFile = new byte[TEST_FILE_LEN];
assertEquals(TEST_FILE_LEN,
instream.read(0, fullFile, 0, fullFile.length));
instream.readFully(0, fullFile, 0, fullFile.length);
assertEquals(0, instream.getPos());

// now read past the end of the file
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,8 @@ protected void validateFileContents(FSDataInputStream stream, int length,
throws IOException {
byte[] streamData = new byte[length];
assertEquals("failed to read expected number of bytes from "
+ "stream", length, stream.read(streamData));
+ "stream. This may be transient",
length, stream.read(streamData));
byte[] validateFileBytes;
if (startIndex == 0 && length == fileBytes.length) {
validateFileBytes = fileBytes;
Expand Down

0 comments on commit 67c4eda

Please sign in to comment.