Running Count on a newly created index throws an exception with "Could not parse server exception"

[simendsjo]

Quite hard to find a reliable way of reproducing the error, but the following works every time for me:

            var settings = new ConnectionSettings().ThrowOnElasticsearchServerExceptions();
            var client = new ElasticClient(settings);

            const string indexName = "countbug";

            // Clean up after we found an error
            if (client.IndexExists(indexName).Exists)
                client.DeleteIndex(indexName);

            while (true)
            {
                client.CreateIndex(indexName);
                client.Count(d => d.Index(indexName));
                client.DeleteIndex(indexName);
            }

Stacktrace:

   at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.CoordinateRequest[T](TransportRequestState`1 requestState, Int32 maxRetries, Int32 retried, Boolean& aliveResponse) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\RequestHandlers\RequestHandler.cs:line 139
   at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.DoRequest[T](TransportRequestState`1 requestState) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\RequestHandlers\RequestHandler.cs:line 168
   at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.Request[T](TransportRequestState`1 requestState, Object data) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\RequestHandlers\RequestHandler.cs:line 34
   at Elasticsearch.Net.Connection.Transport.DoRequest[T](String method, String path, Object data, IRequestParameters requestParameters) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\Transport.cs:line 343
   at Elasticsearch.Net.ElasticsearchClient.DoRequest[T](String method, String path, Object data, IRequestParameters requestParameters) in c:\code\elasticsearch-net\src\Elasticsearch.Net\ElasticsearchClient.cs:line 65
   at Elasticsearch.Net.ElasticsearchClient.CountGet[T](String index, Func`2 requestParameters) in c:\code\elasticsearch-net\src\Elasticsearch.Net\ElasticsearchClient.Generated.cs:line 6884
   at Nest.RawDispatch.CountDispatch[T](ElasticsearchPathInfo`1 pathInfo, Object body) in c:\code\elasticsearch-net\src\Nest\RawDispatch.generated.cs:line 821
   at Nest.ElasticClient.<Count>b__443_1[T](ElasticsearchPathInfo`1 p, CountDescriptor`1 d) in c:\code\elasticsearch-net\src\Nest\ElasticClient-Count.cs:line 17
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[D,Q,R](D descriptor, Func`3 dispatch) in c:\code\elasticsearch-net\src\Nest\ElasticClient.cs:line 74
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[D,Q,R](Func`2 selector, Func`3 dispatch) in c:\code\elasticsearch-net\src\Nest\ElasticClient.cs:line 82
   at Nest.ElasticClient.Count[T](Func`2 countSelector) in c:\code\elasticsearch-net\src\Nest\ElasticClient-Count.cs:line 15
   at Nest.CountExtensions.Count(IElasticClient client, Func`2 countSelector) in c:\code\elasticsearch-net\src\Nest\ConvenienceExtensions\CountExtensions.cs:line 24
   at EsCountBug.Program.Main(String[] args) in C:\Users\simen endsjo\code\tmp\fsharp\EsCountBug\EsCountBug\Program.cs:line 26
   at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
   at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
   at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

This has been reproduced in Elasticsearch 1.7.2 and 1.7.3, but I've only tested Elasticsearch.Net 1.7.1.

[simendsjo]

I should note that this also throws an error:

            while (true)
            {
                client.CreateIndex(indexName);
                client.SyncedFlush(d => d.Index(indexName));
                client.DeleteIndex(indexName);
            }

Stacktrace:

   at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.CoordinateRequest[T](TransportRequestState`1 requestState, Int32 maxRetries, Int32 retried, Boolean& aliveResponse) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\RequestHandlers\RequestHandler.cs:line 139
   at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.DoRequest[T](TransportRequestState`1 requestState) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\RequestHandlers\RequestHandler.cs:line 168
   at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.Request[T](TransportRequestState`1 requestState, Object data) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\RequestHandlers\RequestHandler.cs:line 34
   at Elasticsearch.Net.Connection.Transport.DoRequest[T](String method, String path, Object data, IRequestParameters requestParameters) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\Transport.cs:line 343
   at Elasticsearch.Net.ElasticsearchClient.DoRequest[T](String method, String path, Object data, IRequestParameters requestParameters) in c:\code\elasticsearch-net\src\Elasticsearch.Net\ElasticsearchClient.cs:line 65
   at Elasticsearch.Net.ElasticsearchClient.IndicesFlushSynced[T](String index, Func`2 requestParameters) in c:\code\elasticsearch-net\src\Elasticsearch.Net\ElasticsearchClient.Generated.cs:line 14764
   at Nest.RawDispatch.IndicesFlushSyncedDispatch[T](ElasticsearchPathInfo`1 pathInfo) in c:\code\elasticsearch-net\src\Nest\RawDispatch.generated.cs:line 1836
   at Nest.ElasticClient.<SyncedFlush>b__427_0(ElasticsearchPathInfo`1 p, SyncedFlushDescriptor d) in c:\code\elasticsearch-net\src\Nest\ElasticClient-Flush.cs:line 51
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[D,Q,R](D descriptor, Func`3 dispatch) in c:\code\elasticsearch-net\src\Nest\ElasticClient.cs:line 74
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[D,Q,R](Func`2 selector, Func`3 dispatch) in c:\code\elasticsearch-net\src\Nest\ElasticClient.cs:line 82
   at Nest.ElasticClient.SyncedFlush(Func`2 selector) in c:\code\elasticsearch-net\src\Nest\ElasticClient-Flush.cs:line 49
   at EsCountBug.Program.Main(String[] args) in C:\Users\simen endsjo\code\tmp\fsharp\EsCountBug\EsCountBug\Program.cs:line 26
   at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
   at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
   at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

[simendsjo]

Response:

{StatusCode: 503, 
    Method: GET, 
    Url: http://localhost:9200/test_indexed_count_on_newly_created_index/_count?pretty=true, 
    Request: , 
    Response: {
  "count" : 0,
  "_shards" : {
    "total" : 5,
    "successful" : 0,
    "failed" : 0
  }
}
}

The ReturnTypedResponse looks like the interesting method here.

IsValidResponse returns false because the responses Success is false.
GetErrorFromStream tries to deserialize to OneToOneServerException, but it actually has a response, thus the error becomes blank.

I don't know enough Elasticsearch to know why the server returns 503, or why the count is actually set (correctly?) or what the "_shards" means.

Hopefully this will be enough so some experts might find the correct fix for this issue.

[simendsjo]

Setting a MaxRetries count on the client settings solves this issue. The root cause might be because no shards have actually had time to finish the CreateIndex before count is run, but this is only my guessing.

[simendsjo]

Somewhat related: 503 errors doesn't get a good error message. The following patch manually sets the error fields.

diff --git a/src/Elasticsearch.Net/Connection/RequestHandlers/RequestHandler.cs b/src/Elasticsearch.Net/Connection/RequestHandlers/RequestHandler.cs
index 7ad8abc..cb993f9 100644
--- a/src/Elasticsearch.Net/Connection/RequestHandlers/RequestHandler.cs
+++ b/src/Elasticsearch.Net/Connection/RequestHandlers/RequestHandler.cs
@@ -106,7 +106,7 @@ namespace Elasticsearch.Net.Connection.RequestHandlers
                return this.StreamToTypedResponse<T>(streamResponse, requestState, bytes);

            // If error read error 
-           error = GetErrorFromStream<T>(streamResponse.Response);
+           error = GetErrorFromStream<T>(streamResponse.Response, streamResponse.HttpStatusCode ?? 0);
            var typedResponse = ElasticsearchResponse.CloneFrom<T>(streamResponse, default(T));
            this.SetStringOrByteResult(typedResponse, bytes);
            return typedResponse;
diff --git a/src/Elasticsearch.Net/Connection/RequestHandlers/RequestHandlerBase.cs b/src/Elasticsearch.Net/Connection/RequestHandlers/RequestHandlerBase.cs
index 49f471c..ce8c0b6 100644
--- a/src/Elasticsearch.Net/Connection/RequestHandlers/RequestHandlerBase.cs
+++ b/src/Elasticsearch.Net/Connection/RequestHandlers/RequestHandlerBase.cs
@@ -211,11 +211,17 @@ namespace Elasticsearch.Net.Connection.RequestHandlers
            response.Response = rawResponse;
        }

-       protected ElasticsearchServerError GetErrorFromStream<T>(Stream stream)
+       protected ElasticsearchServerError GetErrorFromStream<T>(Stream stream, int httpStatusCode=0)
        {
            try
            {
                var e = this._serializer.Deserialize<OneToOneServerException>(stream);
+               if (e != null && e.status == 0)
+               {
+                   e.status = httpStatusCode;
+                   if (e.status == 503 && e.error.IsNullOrEmpty())
+                       e.error = "ServiceUnavailableException[Service Unavaliable. Try again later.]";
+               }
                return ElasticsearchServerError.Create(e);
            }
            // ReSharper disable once EmptyGeneralCatchClause

[Mpdreamz]

Your hunch is correct. The CreateIndex() API in elasticsearch can return before shards are available. There is an open ticket to make this more user friendly: elastic/elasticsearch#9126.

The way it solved right now is to wait for at least a yellow status on the cluster after index creation.

Elasticsearch 2.0 has a lot saner exception output just as an FYI :) Thanks for the PR will review it monday morning.

[Mpdreamz]

Closing this issue because we merged the service unavailable improvements but the underlying cause is elastic/elasticsearch#9126