Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
GCS repository snapshot fails intermittently on some shards "Failed to check if blob exists" java.io.IOException: insufficient data written #26636
Elasticsearch version (
Plugins installed: [repository-gcs discovery-gce]
JVM version (
OS version (
Description of the problem including expected versus actual behavior:
Creating a snapshot fails on certain shards. Retrying a new snapshot works. For me it seems to fail on about 10% of shards (testing with 51 shards, 4 failed last test, 2 when I retried, finally 0 on the third try)
The exception is
This is using gcs coldstorage.
I see that there are further options i can give the plugin, mainly http.connect_timeout and http.read_timeout, but im not sure if they are relevant for the exception below:
I wouldn't mind this failing if I could detect it and retry. Could I do this by deleting the snapshot and recreating it? From what I understand the successfully backed up shards will not be deleted if I did this?
Steps to reproduce:
Provide logs (if relevant):
According to googleapis/google-http-java-client#333 the error message
Using the following jar versions
bash-4.3# ls -1 repository-gcs/
added a commit
Nov 14, 2017
A new version (1.23.0) of google-http-client has been released in October 2017. I updated the versions used in the
At that stage, I suggest to close this issue for now and to wait for more tests and feedback on plugins that use the new version of google-http-java-client. This will be released in Elasticsearch 6.0.1 (and potentially in 5.6.5 if this version is released).