Partial read bug #405
Partial read bug #405
Conversation
…th header does not match the body length.
…rom the http proy
…doing so at that point is premature and prevents appending to the channel during network-fault-induced retries.
… set when we experience the situation where the network delivers a smaller payload than the http header says.
Creating a new GetObjectRequest per item transfer in a way that gives back the same channel for like-named objects.
…into partial_read_bug # Conflicts: # ds3-sdk/src/main/java/com/spectralogic/ds3client/utils/IOUtils.java
… of the ranges to transfer in an attempt to provoke a failure that having a 0 offset would mask.
…into partial_read_bug
| try (final InputStream responseStream = response.getResponseStream()) { | ||
| final long startTime = PerformanceUtils.getCurrentTime(); | ||
| final long totalBytes = IOUtils.copy(responseStream, destinationChannel, bufferSize, objName, false); | ||
| destinationChannel.close(); |
There was a problem hiding this comment.
This is an autogen file and should not be manually modified. If you need a change in an autogened file, let me know.
There was a problem hiding this comment.
Ah. Intellij hides the auto gen comment in the copyright notice. We can't close the channel in the response parser if we want to be able to resume an interrupted http transfer. The java sdk has this notion of an AutoCloseableCache that keeps track of channel lifetimes. Closing the channel here subverts the cache lifetime management.
There was a problem hiding this comment.
Wait. There may be a way around this. I'll need to run the java integration tests on sm25-2
There was a problem hiding this comment.
OK. Should be back to way it was.
…l repositioning logic, so that the autogen code can stay as it is when we retry due to a network failure.
…into partial_read_bug
|
out of date |
…into partial_read_bug # Conflicts: # ds3-sdk-integration/src/test/java/com/spectralogic/ds3client/integration/GetJobManagement_Test.java
… mark a directory writable or not in favor of Process.waitFor().
…into partial_read_bug
| private Long numBytesToTransfer; | ||
| private ImmutableCollection<Range> ranges; | ||
| private final AtomicLong destinationChannelOffset = new AtomicLong(0); | ||
| private final AtomicReference<ItemTransferrer> itemTransferrer = new AtomicReference<>(); |
There was a problem hiding this comment.
Are we really worried that this could be modified by multiple threads?
| } | ||
|
|
||
| private void emitContentLengthMismatchFailureEvent(final BulkObject ds3Object, final Throwable t) { | ||
| final FailureEvent failureEvent = new FailureEvent.Builder() |
There was a problem hiding this comment.
I tend to favor the syntax that guava uses for its builders. ie FailureEvent.builder() rather than new FailureEvent.Builder()
…into partial_read_bug
Updating GetObject test to pass or fail taking into account partial read failures.
…into partial_read_bug
…into partial_read_bug
…3_java_sdk into partial_read_bug
3 participants
This is a fix for https://jira.spectralogic.com/browse/JSDK-189
If a network transfer gets cut off before transferring all the data we ask for, retry the transfer by
creating a new set of blob ranges.