Randomly Receiving com.google.cloud.storage.StorageException: Connection reset

[anjali2892]

We randomly receiving com.google.cloud.storage.StorageException: Connection reset. in some requests when uploading and downloading file from google cloud storage.
Mainly this exception is coming:

1. while fetching bucket instance
2. while creating a new ACL entry on the specified blob.

• OS: Windows
• Java version: 8
• google-cloud-java version(s): 1.2.1,

Stacktrace

Caused by: com.google.cloud.storage.StorageException: Connection reset
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:309)
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:169)
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:166)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:165)
... 19 common frames omitted
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.security.ssl.InputRecord.readFully(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:307)
... 26 common frames omitted

1. code snippet

  Storage storage = StorageOptions.getDefaultInstance()
      .getService();
  // get bucket instance from storage
  Bucket bucket = storage.get(bucketName);

2. code snippet

// get storage instance
Storage storage = StorageOptions.getDefaultInstance()
.getService();
// get bucket instance from storage
Bucket bucket = storage.get(bucketName);
BlobId blobId = BlobId.of(bucket.getName(), subDirectory + "/" + name);
BlobInfo blobInfo = BlobInfo.newBuilder(blobId)
.build();
try (WriteChannel writer = storage.writer(blobInfo)) {
writer.write(ByteBuffer.wrap(content, 0, content.length));
}
if (markPublic) {
// mark the uploaded file as public
Acl acl = storage.createAcl(blobId, Acl.of(Acl.User.ofAllUsers(), Acl.Role.READER));
return "https://storage.googleapis.com/" + blobInfo.getBucket() + "/" + blobId.getName();
}

Additional Information

I have also white listed .googleapis.com and *.google.com domains. But some of the calls from SDK are calling IP address directly like https://216.58.220.10 because of that my network is restricting that call.

Is there a way so that I can configure something in Storage SDK so that it should call only those APIs which are covered with google domain. I have restrictions on my firewall so that data packets from random IP addresses should not get transferred.
Below is the attached image that shows some requests are going to google.com, while some requests are going to IP address(google owned) directly. Requests that are going directly to IP address are throwing this exception.

Thanks!

[dgray16]

My case:
Google cloud storage: 1.78.0
Java: 12
App was running in Google Compute Engine VM.
I made read operation over com.google.cloud.ReadChannel.

com.google.cloud.storage.StorageException: Connection reset
        at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:229)
        at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:692)
        at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127)
        at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124)
        at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
        at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
        at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
        at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:123)
        at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
        at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
        at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
        at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:243)
        at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
        at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
        at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
        at java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
        at mycomp.extractor.partner3extractor.batch.reader.BaseGoogleStorageReader.read(BaseGoogleStorageReader.java:40)
        at mycomp.extractor.partner3extractor.batch.reader.BaseGoogleStorageReader.read(BaseGoogleStorageReader.java:18)
        at org.springframework.batch.core.step.item.SimpleChunkProvider.doRead(SimpleChunkProvider.java:94)
        at org.springframework.batch.core.step.item.SimpleChunkProvider.read(SimpleChunkProvider.java:161)
        at org.springframework.batch.core.step.item.SimpleChunkProvider$1.doInIteration(SimpleChunkProvider.java:119)
        at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375)
        at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
        at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145)
        at org.springframework.batch.core.step.item.SimpleChunkProvider.provide(SimpleChunkProvider.java:113)
        at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:69)
        at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407)
        at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
        at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
        at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
        at org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable.run(TaskExecutorRepeatTemplate.java:262)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:835)
Caused by: javax.net.ssl.SSLException: Connection reset
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:137)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1183)
        at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1153)
        at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:828)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
        at java.base/sun.net.www.MeteredStream.read(MeteredStream.java:134)
        at java.base/java.io.FilterInputStream.read(FilterInputStream.java:133)
        at java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3495)
        at com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.read(NetHttpResponse.java:164)
        at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
        at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:49)
        at com.google.api.client.util.IOUtils.copy(IOUtils.java:87)
        at com.google.api.client.util.IOUtils.copy(IOUtils.java:59)
        at com.google.api.client.http.HttpResponse.download(HttpResponse.java:386)
        at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:686)
        ... 38 common frames omitted
        Suppressed: java.net.SocketException: Broken pipe (Write failed)
                at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
                at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
                at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
                at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
                at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:351)
                ... 57 common frames omitted
Caused by: java.net.SocketException: Connection reset
        at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
        at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
        at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
        at java.base/sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(SSLSocketInputRecord.java:237)
        at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:190)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
        ... 54 common frames omitted

[ghost]

I suppose the solve of this issue is to override host address by calling function setHost()
like :

StorageOptions.Builder optionsBuilder = StorageOptions.newBuilder().setHost("https://storage.googleapis.com/");
Storage storage = optionsBuilder.build().getService();

[dmitry-fa]

The origin of the problem is certainly not in this API library but on the server side.
https://issuetracker.google.com/issues/123456218 was submitted to track that.
So, I'm closing this issue.