Plugin should not throw exceptions when it gets a 404 result from GCP Storage #19
Comments
|
The only plugin code in the stack is Can we handle these better? We should just return false. |
|
Going backwards from the stack, it seem that in So that means |
|
I think the underlying problem we might be running into is due to the way @Override
public int read(ByteBuffer byteBuffer) throws IOException {
validateOpen();
if (buffer == null) {
if (endOfStream) {
return -1;
}
final int toRead =
Math.toIntExact(Math.min(limit - position, Math.max(byteBuffer.remaining(), chunkSize)));
if (toRead <= 0) {
endOfStream = true;
return -1;
}
try {
ResultRetryAlgorithm<?> algorithm =
retryAlgorithmManager.getForObjectsGet(storageObject, requestOptions);
Tuple<String, byte[]> result =
runWithRetries(
() -> storageRpc.read(storageObject, requestOptions, position, toRead),
serviceOptions.getRetrySettings(),
algorithm,
serviceOptions.getClock());
String etag = result.x();
byte[] bytes = result.y();
if (bytes.length > 0 && lastEtag != null && !Objects.equals(etag, lastEtag)) {
throw new IOException("Blob " + blob + " was updated while reading");
}
lastEtag = etag;
buffer = bytes;
} catch (RetryHelper.RetryHelperException e) {
throw new IOException(e);
}
if (toRead > buffer.length) {
endOfStream = true;
if (buffer.length == 0) {
buffer = null;
return -1;
}
}
}
int toWrite = Math.min(buffer.length - bufferPos, byteBuffer.remaining());
byteBuffer.put(buffer, bufferPos, toWrite);
bufferPos += toWrite;
if (bufferPos >= buffer.length) {
position += buffer.length;
buffer = null;
bufferPos = 0;
}
return toWrite;
}Specifically: Tuple<String, byte[]> result =
runWithRetries(
() -> storageRpc.read(storageObject, requestOptions, position, toRead),
serviceOptions.getRetrySettings(),
algorithm,
serviceOptions.getClock());The underlying implementation is making multiple RPCs to do range-gets on a blob. |
- Remove the need for retries. - When writes fail, we fail silently and treat subsequent reads as a cache-miss. - Set the `chunkSize` for reads to equal the size of the `Blob`. This way, we only make 1 RPC per blob input stream. This is safe because the size of the objects in cache are not very large.
https://ci.android.com/builds/submitted/8521547/androidx_device_tests/latest/view/logs/build_error.log
The text was updated successfully, but these errors were encountered: