Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GoogleCloudStorageRepositoryClientYamlTestSuiteIT failed with "Socket closed" #57026

Closed
DaveCTurner opened this issue May 21, 2020 · 2 comments · Fixed by #57073
Closed

GoogleCloudStorageRepositoryClientYamlTestSuiteIT failed with "Socket closed" #57026

DaveCTurner opened this issue May 21, 2020 · 2 comments · Fixed by #57073
Assignees
Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI

Comments

@DaveCTurner
Copy link
Contributor

Build scan:

https://gradle-enterprise.elastic.co/s/nbmefwlekigxu/console-log?task=:plugins:repository-gcs:qa:google-cloud-storage:largeBlobIntegTestRunner

Repro line:

REPRODUCE WITH: ./gradlew ':plugins:repository-gcs:qa:google-cloud-storage:largeBlobIntegTestRunner' --tests "org.elasticsearch.repositories.gcs.GoogleCloudStorageRepositoryClientYamlTestSuiteIT.test {yaml=repository_gcs/10_repository/Snapshot/Restore with repository-gcs}" -Dtests.seed=65157749A621830A -Dtests.security.manager=true -Dtests.locale=ar-SY -Dtests.timezone=America/Indiana/Winamac -Dcompiler.java=14 -Druntime.java=8

Reproduces locally?:

Fails for an apparently-unrelated reason:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':plugins:repository-gcs:qa:google-cloud-storage:largeBlobIntegTestRunner'.
> supplied keystore file /home/davidturner/elasticsearch/plugins/repository-gcs/qa/google-cloud-storage/build/generated-resources/service_account_test.json does not exist, require for node{:plugins:repository-gcs:qa:google-cloud-storage:largeBlobIntegTest-0}

Applicable branches:

7.x-and-earlier.

Failure history:

16 occurrences over the last 30 days, all on 7.x-or-earlier:

https://build-stats.elastic.co/goto/a4ba4449f6e570b5ea9d31c01ae74fb7

Failure excerpt:


org.elasticsearch.repositories.gcs.GoogleCloudStorageRepositoryClientYamlTestSuiteIT > test {yaml=repository_gcs/10_repository/Snapshot/Restore with repository-gcs} FAILED |  
-- | --
  | java.lang.AssertionError: Failure at [repository_gcs/10_repository:122]: expected [2xx] status code but api [snapshot.create] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"snapshot_exception","reason":"[repository:snapshot-two/hZQLFtlcQ6e6_jLQLhnKuA] failed to update snapshot in repository","stack_trace":"SnapshotException[[repository:snapshot-two/hZQLFtlcQ6e6_jLQLhnKuA] failed to update snapshot in repository]; nested: StorageException[Socket Closed]; nested: IOException[Socket Closed];\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$36(BlobStoreRepository.java:1003)\n\tat org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)\n\tat org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:65)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)\n\tat java.util.ArrayList.forEach(ArrayList.java:1257)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)\n\tat org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)\n\tat org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$52(BlobStoreRepository.java:1503)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: com.google.cloud.storage.StorageException: Socket Closed\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:229)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:310)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:196)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:193)\n\tat com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)\n\tat com.google.cloud.RetryHelper.run(RetryHelper.java:76)\n\tat com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)\n\tat com.google.cloud.storage.StorageImpl.internalCreate(StorageImpl.java:192)\n\tat com.google.cloud.storage.StorageImpl.create(StorageImpl.java:154)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$writeBlobMultipart$7(GoogleCloudStorageBlobStore.java:342)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.lambda$doPrivilegedVoidIOException$0(SocketAccess.java:54)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedVoidIOException(SocketAccess.java:53)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlobMultipart(GoogleCloudStorageBlobStore.java:341)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlob(GoogleCloudStorageBlobStore.java:245)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlob(GoogleCloudStorageBlobContainer.java:72)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlobAtomic(GoogleCloudStorageBlobContainer.java:77)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:1651)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$54(BlobStoreRepository.java:1526)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\t... 18 more\nCaused by: java.io.IOException: Socket Closed\n\tat java.net.AbstractPlainSocketImpl.getInputStream(AbstractPlainSocketImpl.java:421)\n\tat java.net.Socket$2.run(Socket.java:931)\n\tat java.net.Socket$2.run(Socket.java:929)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.net.Socket.getInputStream(Socket.java:928)\n\tat sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:673)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)\n\tat java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)\n\tat com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:149)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)\n\tat com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequestWithoutGZip(MediaHttpUploader.java:551)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequest(MediaHttpUploader.java:568)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.directUpload(MediaHttpUploader.java:360)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:334)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:551)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:307)\n\t... 36 more\n"}],"type":"snapshot_exception","reason":"[repository:snapshot-two/hZQLFtlcQ6e6_jLQLhnKuA] failed to update snapshot in repository","caused_by":{"type":"storage_exception","reason":"Socket Closed","caused_by":{"type":"i_o_exception","reason":"Socket Closed","stack_trace":"java.io.IOException: Socket Closed\n\tat java.net.AbstractPlainSocketImpl.getInputStream(AbstractPlainSocketImpl.java:421)\n\tat java.net.Socket$2.run(Socket.java:931)\n\tat java.net.Socket$2.run(Socket.java:929)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.net.Socket.getInputStream(Socket.java:928)\n\tat sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:673)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)\n\tat java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)\n\tat com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:149)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)\n\tat com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequestWithoutGZip(MediaHttpUploader.java:551)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequest(MediaHttpUploader.java:568)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.directUpload(MediaHttpUploader.java:360)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:334)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:551)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:307)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:196)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:193)\n\tat com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)\n\tat com.google.cloud.RetryHelper.run(RetryHelper.java:76)\n\tat com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)\n\tat com.google.cloud.storage.StorageImpl.internalCreate(StorageImpl.java:192)\n\tat com.google.cloud.storage.StorageImpl.create(StorageImpl.java:154)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$writeBlobMultipart$7(GoogleCloudStorageBlobStore.java:342)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.lambda$doPrivilegedVoidIOException$0(SocketAccess.java:54)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedVoidIOException(SocketAccess.java:53)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlobMultipart(GoogleCloudStorageBlobStore.java:341)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlob(GoogleCloudStorageBlobStore.java:245)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlob(GoogleCloudStorageBlobContainer.java:72)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlobAtomic(GoogleCloudStorageBlobContainer.java:77)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:1651)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$54(BlobStoreRepository.java:1526)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)\n\tat java.util.ArrayList.forEach(ArrayList.java:1257)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)\n\tat org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)\n\tat org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$52(BlobStoreRepository.java:1503)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n"},"stack_trace":"com.google.cloud.storage.StorageException: Socket Closed\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:229)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:310)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:196)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:193)\n\tat com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)\n\tat com.google.cloud.RetryHelper.run(RetryHelper.java:76)\n\tat com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)\n\tat com.google.cloud.storage.StorageImpl.internalCreate(StorageImpl.java:192)\n\tat com.google.cloud.storage.StorageImpl.create(StorageImpl.java:154)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$writeBlobMultipart$7(GoogleCloudStorageBlobStore.java:342)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.lambda$doPrivilegedVoidIOException$0(SocketAccess.java:54)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedVoidIOException(SocketAccess.java:53)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlobMultipart(GoogleCloudStorageBlobStore.java:341)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlob(GoogleCloudStorageBlobStore.java:245)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlob(GoogleCloudStorageBlobContainer.java:72)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlobAtomic(GoogleCloudStorageBlobContainer.java:77)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:1651)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$54(BlobStoreRepository.java:1526)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)\n\tat java.util.ArrayList.forEach(ArrayList.java:1257)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)\n\tat org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)\n\tat org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$52(BlobStoreRepository.java:1503)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.io.IOException: Socket Closed\n\tat java.net.AbstractPlainSocketImpl.getInputStream(AbstractPlainSocketImpl.java:421)\n\tat java.net.Socket$2.run(Socket.java:931)\n\tat java.net.Socket$2.run(Socket.java:929)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.net.Socket.getInputStream(Socket.java:928)\n\tat sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:673)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)\n\tat java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)\n\tat com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:149)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)\n\tat com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequestWithoutGZip(MediaHttpUploader.java:551)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequest(MediaHttpUploader.java:568)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.directUpload(MediaHttpUploader.java:360)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:334)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:551)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:307)\n\t... 36 more\n"},"stack_trace":"SnapshotException[[repository:snapshot-two/hZQLFtlcQ6e6_jLQLhnKuA] failed to update snapshot in repository]; nested: StorageException[Socket Closed]; nested: IOException[Socket Closed];\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$36(BlobStoreRepository.java:1003)\n\tat org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)\n\tat org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:65)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)\n\tat java.util.ArrayList.forEach(ArrayList.java:1257)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)\n\tat org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)\n\tat org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$52(BlobStoreRepository.java:1503)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: com.google.cloud.storage.StorageException: Socket Closed\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:229)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:310)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:196)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:193)\n\tat com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)\n\tat com.google.cloud.RetryHelper.run(RetryHelper.java:76)\n\tat com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)\n\tat com.google.cloud.storage.StorageImpl.internalCreate(StorageImpl.java:192)\n\tat com.google.cloud.storage.StorageImpl.create(StorageImpl.java:154)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$writeBlobMultipart$7(GoogleCloudStorageBlobStore.java:342)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.lambda$doPrivilegedVoidIOException$0(SocketAccess.java:54)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedVoidIOException(SocketAccess.java:53)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlobMultipart(GoogleCloudStorageBlobStore.java:341)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlob(GoogleCloudStorageBlobStore.java:245)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlob(GoogleCloudStorageBlobContainer.java:72)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlobAtomic(GoogleCloudStorageBlobContainer.java:77)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:1651)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$54(BlobStoreRepository.java:1526)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\t... 18 more\nCaused by: java.io.IOException: Socket Closed\n\tat java.net.AbstractPlainSocketImpl.getInputStream(AbstractPlainSocketImpl.java:421)\n\tat java.net.Socket$2.run(Socket.java:931)\n\tat java.net.Socket$2.run(Socket.java:929)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.net.Socket.getInputStream(Socket.java:928)\n\tat sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:673)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)\n\tat java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)\n\tat com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:149)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)\n\tat com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequestWithoutGZip(MediaHttpUploader.java:551)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequest(MediaHttpUploader.java:568)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.directUpload(MediaHttpUploader.java:360)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:334)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:551)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:307)\n\t... 36 more\n"},"status":500}] |  
  | at __randomizedtesting.SeedInfo.seed([65157749A621830A:ED41489308DDEEF2]:0) |  
  | at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:405) |  
  | at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:382) |  
  | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) |  
  | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) |  
  | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) |  
  | at java.lang.reflect.Method.invoke(Method.java:498) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) |  
  | at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) |  
  | at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) |  
  | at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) |  
  | at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) |  
  | at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) |  
  | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |  
  | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) |  
  | at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) |  
  | at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) |  
  | at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) |  
  | at java.lang.Thread.run(Thread.java:748) |  
  |   |  
  | Caused by: |  
  | java.lang.AssertionError: expected [2xx] status code but api [snapshot.create] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"snapshot_exception","reason":"[repository:snapshot-two/hZQLFtlcQ6e6_jLQLhnKuA] failed to update snapshot in repository","stack_trace":"SnapshotException[[repository:snapshot-two/hZQLFtlcQ6e6_jLQLhnKuA] failed to update snapshot in repository]; nested: StorageException[Socket Closed]; nested: IOException[Socket Closed];\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$36(BlobStoreRepository.java:1003)\n\tat org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)\n\tat org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:65)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)\n\tat java.util.ArrayList.forEach(ArrayList.java:1257)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)\n\tat org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)\n\tat org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$52(BlobStoreRepository.java:1503)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: com.google.cloud.storage.StorageException: Socket Closed\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:229)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:310)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:196)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:193)\n\tat com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)\n\tat com.google.cloud.RetryHelper.run(RetryHelper.java:76)\n\tat com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)\n\tat com.google.cloud.storage.StorageImpl.internalCreate(StorageImpl.java:192)\n\tat com.google.cloud.storage.StorageImpl.create(StorageImpl.java:154)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$writeBlobMultipart$7(GoogleCloudStorageBlobStore.java:342)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.lambda$doPrivilegedVoidIOException$0(SocketAccess.java:54)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedVoidIOException(SocketAccess.java:53)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlobMultipart(GoogleCloudStorageBlobStore.java:341)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlob(GoogleCloudStorageBlobStore.java:245)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlob(GoogleCloudStorageBlobContainer.java:72)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlobAtomic(GoogleCloudStorageBlobContainer.java:77)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:1651)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$54(BlobStoreRepository.java:1526)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\t... 18 more\nCaused by: java.io.IOException: Socket Closed\n\tat java.net.AbstractPlainSocketImpl.getInputStream(AbstractPlainSocketImpl.java:421)\n\tat java.net.Socket$2.run(Socket.java:931)\n\tat java.net.Socket$2.run(Socket.java:929)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.net.Socket.getInputStream(Socket.java:928)\n\tat sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:673)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)\n\tat java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)\n\tat com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:149)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)\n\tat com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequestWithoutGZip(MediaHttpUploader.java:551)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequest(MediaHttpUploader.java:568)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.directUpload(MediaHttpUploader.java:360)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:334)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:551)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:307)\n\t... 36 more\n"}],"type":"snapshot_exception","reason":"[repository:snapshot-two/hZQLFtlcQ6e6_jLQLhnKuA] failed to update snapshot in repository","caused_by":{"type":"storage_exception","reason":"Socket Closed","caused_by":{"type":"i_o_exception","reason":"Socket Closed","stack_trace":"java.io.IOException: Socket Closed\n\tat java.net.AbstractPlainSocketImpl.getInputStream(AbstractPlainSocketImpl.java:421)\n\tat java.net.Socket$2.run(Socket.java:931)\n\tat java.net.Socket$2.run(Socket.java:929)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.net.Socket.getInputStream(Socket.java:928)\n\tat sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:673)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)\n\tat java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)\n\tat com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:149)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)\n\tat com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequestWithoutGZip(MediaHttpUploader.java:551)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequest(MediaHttpUploader.java:568)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.directUpload(MediaHttpUploader.java:360)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:334)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:551)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:307)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:196)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:193)\n\tat com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)\n\tat com.google.cloud.RetryHelper.run(RetryHelper.java:76)\n\tat com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)\n\tat com.google.cloud.storage.StorageImpl.internalCreate(StorageImpl.java:192)\n\tat com.google.cloud.storage.StorageImpl.create(StorageImpl.java:154)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$writeBlobMultipart$7(GoogleCloudStorageBlobStore.java:342)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.lambda$doPrivilegedVoidIOException$0(SocketAccess.java:54)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedVoidIOException(SocketAccess.java:53)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlobMultipart(GoogleCloudStorageBlobStore.java:341)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlob(GoogleCloudStorageBlobStore.java:245)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlob(GoogleCloudStorageBlobContainer.java:72)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlobAtomic(GoogleCloudStorageBlobContainer.java:77)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:1651)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$54(BlobStoreRepository.java:1526)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)\n\tat java.util.ArrayList.forEach(ArrayList.java:1257)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)\n\tat org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)\n\tat org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$52(BlobStoreRepository.java:1503)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n"},"stack_trace":"com.google.cloud.storage.StorageException: Socket Closed\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:229)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:310)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:196)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:193)\n\tat com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)\n\tat com.google.cloud.RetryHelper.run(RetryHelper.java:76)\n\tat com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)\n\tat com.google.cloud.storage.StorageImpl.internalCreate(StorageImpl.java:192)\n\tat com.google.cloud.storage.StorageImpl.create(StorageImpl.java:154)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$writeBlobMultipart$7(GoogleCloudStorageBlobStore.java:342)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.lambda$doPrivilegedVoidIOException$0(SocketAccess.java:54)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedVoidIOException(SocketAccess.java:53)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlobMultipart(GoogleCloudStorageBlobStore.java:341)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlob(GoogleCloudStorageBlobStore.java:245)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlob(GoogleCloudStorageBlobContainer.java:72)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlobAtomic(GoogleCloudStorageBlobContainer.java:77)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:1651)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$54(BlobStoreRepository.java:1526)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)\n\tat java.util.ArrayList.forEach(ArrayList.java:1257)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)\n\tat org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)\n\tat org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$52(BlobStoreRepository.java:1503)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.io.IOException: Socket Closed\n\tat java.net.AbstractPlainSocketImpl.getInputStream(AbstractPlainSocketImpl.java:421)\n\tat java.net.Socket$2.run(Socket.java:931)\n\tat java.net.Socket$2.run(Socket.java:929)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.net.Socket.getInputStream(Socket.java:928)\n\tat sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:673)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)\n\tat java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)\n\tat com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:149)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)\n\tat com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequestWithoutGZip(MediaHttpUploader.java:551)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequest(MediaHttpUploader.java:568)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.directUpload(MediaHttpUploader.java:360)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:334)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:551)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:307)\n\t... 36 more\n"},"stack_trace":"SnapshotException[[repository:snapshot-two/hZQLFtlcQ6e6_jLQLhnKuA] failed to update snapshot in repository]; nested: StorageException[Socket Closed]; nested: IOException[Socket Closed];\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$36(BlobStoreRepository.java:1003)\n\tat org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)\n\tat org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:65)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)\n\tat java.util.ArrayList.forEach(ArrayList.java:1257)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)\n\tat org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)\n\tat org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$52(BlobStoreRepository.java:1503)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: com.google.cloud.storage.StorageException: Socket Closed\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:229)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:310)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:196)\n\tat com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:193)\n\tat com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)\n\tat com.google.cloud.RetryHelper.run(RetryHelper.java:76)\n\tat com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)\n\tat com.google.cloud.storage.StorageImpl.internalCreate(StorageImpl.java:192)\n\tat com.google.cloud.storage.StorageImpl.create(StorageImpl.java:154)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$writeBlobMultipart$7(GoogleCloudStorageBlobStore.java:342)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.lambda$doPrivilegedVoidIOException$0(SocketAccess.java:54)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedVoidIOException(SocketAccess.java:53)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlobMultipart(GoogleCloudStorageBlobStore.java:341)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.writeBlob(GoogleCloudStorageBlobStore.java:245)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlob(GoogleCloudStorageBlobContainer.java:72)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.writeBlobAtomic(GoogleCloudStorageBlobContainer.java:77)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:1651)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$54(BlobStoreRepository.java:1526)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\t... 18 more\nCaused by: java.io.IOException: Socket Closed\n\tat java.net.AbstractPlainSocketImpl.getInputStream(AbstractPlainSocketImpl.java:421)\n\tat java.net.Socket$2.run(Socket.java:931)\n\tat java.net.Socket$2.run(Socket.java:929)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.net.Socket.getInputStream(Socket.java:928)\n\tat sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:673)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)\n\tat java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)\n\tat com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:149)\n\tat com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)\n\tat com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequestWithoutGZip(MediaHttpUploader.java:551)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequest(MediaHttpUploader.java:568)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.directUpload(MediaHttpUploader.java:360)\n\tat com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:334)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:551)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)\n\tat com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)\n\tat com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:307)\n\t... 36 more\n"},"status":500}] |  
  | at org.junit.Assert.fail(Assert.java:88) |  
  | at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:297) |  
  | at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:398) |  
  | ... 38 more |  
  | REPRODUCE WITH: ./gradlew ':plugins:repository-gcs:qa:google-cloud-storage:largeBlobIntegTestRunner' --tests "org.elasticsearch.repositories.gcs.GoogleCloudStorageRepositoryClientYamlTestSuiteIT.test {yaml=repository_gcs/10_repository/Snapshot/Restore with repository-gcs}" -Dtests.seed=65157749A621830A -Dtests.security.manager=true -Dtests.locale=ar-SY -Dtests.timezone=America/Indiana/Winamac -Dcompiler.java=14 -Druntime.java=8

@DaveCTurner DaveCTurner added :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI Team:Distributed Meta label for distributed team labels May 21, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

@original-brownbear
Copy link
Member

This fails on master as well (even after fixing the Gradle build to be able to run this test in isolation ... will open a PR for that shortly). Looks like something is broken about overriding the size of when we start a resumable upload. What's happening here is that a snapshot starts hanging perpetually without ever uploading a thing to the fixture:

{
  "snapshots": [
    {
      "snapshot": "snapshot-one",
      "repository": "repository",
      "uuid": "RZ2e0QFETFy6E7KTwDf9pw",
      "state": "STARTED",
      "include_global_state": true,
      "shards_stats": {
        "initializing": 0,
        "started": 1,
        "finalizing": 0,
        "done": 0,
        "failed": 0,
        "total": 1
      },
      "stats": {
        "incremental": {
          "file_count": 4,
          "size_in_bytes": 3701
        },
        "processed": {
          "file_count": 0,
          "size_in_bytes": 0
        },
        "total": {
          "file_count": 4,
          "size_in_bytes": 3701
        },
        "start_time_in_millis": 1590083657898,
        "time_in_millis": 52729
      },
      "indices": {
        "docs": {
          "shards_stats": {
            "initializing": 0,
            "started": 1,
            "finalizing": 0,
            "done": 0,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 4,
              "size_in_bytes": 3701
            },
            "processed": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 4,
              "size_in_bytes": 3701
            },
            "start_time_in_millis": 1590083657898,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "STARTED",
              "stats": {
                "incremental": {
                  "file_count": 4,
                  "size_in_bytes": 3701
                },
                "processed": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 4,
                  "size_in_bytes": 3701
                },
                "start_time_in_millis": 1590083657898,
                "time_in_millis": 0
              },
              "node": "K-ycYL7tQtanN6aWo5CTAw"
            }
          }
        }
      }
    }
  ]
}

... it is however able to properly make out what the contents of the repository are so I suspect that it's an issue with trying to write a very small file via a resumable upload that's killing us here.

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue May 22, 2020
A few relatively obvious issues here:

* We cannot run the different IT runs (large blob setting one and normal integ run) concurrently
* We need to set the dependency tasks up correctly for the large blob run so that it works in isolation
* We can't use the `localAddress` for the location header of the resumable upload
(this breaks in YAML tests because GCS is using a loopback port forward for the initial request and the
local address will be chosen as the actual Docker container host)

Closes elastic#57026
original-brownbear added a commit that referenced this issue May 25, 2020
A few relatively obvious issues here:

* We cannot run the different IT runs (large blob setting one and normal integ run) concurrently
* We need to set the dependency tasks up correctly for the large blob run so that it works in isolation
* We can't use the `localAddress` for the location header of the resumable upload
(this breaks in YAML tests because GCS is using a loopback port forward for the initial request and the
local address will be chosen as the actual Docker container host)

Closes #57026
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue May 25, 2020
A few relatively obvious issues here:

* We cannot run the different IT runs (large blob setting one and normal integ run) concurrently
* We need to set the dependency tasks up correctly for the large blob run so that it works in isolation
* We can't use the `localAddress` for the location header of the resumable upload
(this breaks in YAML tests because GCS is using a loopback port forward for the initial request and the
local address will be chosen as the actual Docker container host)

Closes elastic#57026
original-brownbear added a commit that referenced this issue May 25, 2020
A few relatively obvious issues here:

* We cannot run the different IT runs (large blob setting one and normal integ run) concurrently
* We need to set the dependency tasks up correctly for the large blob run so that it works in isolation
* We can't use the `localAddress` for the location header of the resumable upload
(this breaks in YAML tests because GCS is using a loopback port forward for the initial request and the
local address will be chosen as the actual Docker container host)

Closes #57026
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue May 27, 2020
A few relatively obvious issues here:

* We cannot run the different IT runs (large blob setting one and normal integ run) concurrently
* We need to set the dependency tasks up correctly for the large blob run so that it works in isolation
* We can't use the `localAddress` for the location header of the resumable upload
(this breaks in YAML tests because GCS is using a loopback port forward for the initial request and the
local address will be chosen as the actual Docker container host)

Closes elastic#57026
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue May 27, 2020
A few relatively obvious issues here:

* We cannot run the different IT runs (large blob setting one and normal integ run) concurrently
* We need to set the dependency tasks up correctly for the large blob run so that it works in isolation
* We can't use the `localAddress` for the location header of the resumable upload
(this breaks in YAML tests because GCS is using a loopback port forward for the initial request and the
local address will be chosen as the actual Docker container host)

Closes elastic#57026
original-brownbear added a commit that referenced this issue May 27, 2020
A few relatively obvious issues here:

* We cannot run the different IT runs (large blob setting one and normal integ run) concurrently
* We need to set the dependency tasks up correctly for the large blob run so that it works in isolation
* We can't use the `localAddress` for the location header of the resumable upload
(this breaks in YAML tests because GCS is using a loopback port forward for the initial request and the
local address will be chosen as the actual Docker container host)

Closes #57026
original-brownbear added a commit that referenced this issue May 27, 2020
A few relatively obvious issues here:

* We cannot run the different IT runs (large blob setting one and normal integ run) concurrently
* We need to set the dependency tasks up correctly for the large blob run so that it works in isolation
* We can't use the `localAddress` for the location header of the resumable upload
(this breaks in YAML tests because GCS is using a loopback port forward for the initial request and the
local address will be chosen as the actual Docker container host)

Closes #57026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants