Issues with inner exception of MaxRetryException

[satishmallik]

I am always seeing MaxRetryException inner exception set to NULL. There is no way currently I can get inner exception for MaxRetryException with nest.

Please have a look,

[gmarz]

Hi @satishmallik,

What version of NEST are you using? Does ConnectionStatus.OriginalException have a value?

[Mpdreamz]

@satishmallik any updates on this?

[satishmallik]

Sorry was too busy with project. I am using Nest 1.1.0. When MaxRetryException is thrown there is no IResponse to figure ConnectionStatus and OriginalException.

Apart whole retry logic seems to be linear. There is no delay between retries. If system is already loaded retrying linearly is going to make matter worse. There should be some way to set wait interval between retries.

[Mpdreamz]

@satishmallik our reasoning for being linear is that you want an answer as fast as possible. When a node is marked in error state they will be marked as down exponentially (with a configurable maximum) though.

i.e you have a 5 node cluster with a maxretry of 5, if your whole cluster is really churning on something and unable to process anything the client will first hit all 5 nodes but find they are all unresponsive and return as fast as possible. If a new request comes in before while all 5 nodes are in the dog house (marked as down for a configurable timeout) the client will only pick a node and fail early if its found to still be down.

The whole system being loaded is the worst case scenario in which case retrying with exponential intervals might help but might end up making it even worse in the case where you have many requests from you application turning into many requests to your ES cluster.

The way its setup is to try and be as deterministic as possible so that request fail as fast as possible.

Its also optimised for the "happy" downtime where some of your nodes are down but a quorum is still up and running as a healthy(green/yellow) cluster. In these cases you do not want to add a delay between retries.

I assume #945 was the cause of NULL innerexceptions reported here? I'm closing this issue in favour of #945 but feel free to reopen if this isn't the case!

[satishmallik]

Thanks for response Mpdreamz. But I still have the problem of not getting actual exception when MaxRetryException is thrown. As I said inner exception is null and even IResponse is NULL. So from MaxRetryException it is hard to figure what went wrong.

[mosiddi]

I just built the latest code from the develop branch and can still see the inner exception is not being reported. I looked through the requestStates' SeenExceptions property and it is always 0 though the retries are happening.

I think the the retry is being called from here (https://github.com/elasticsearch/elasticsearch-net/blob/develop/src/Elasticsearch.Net/Connection/RequestHandlers/RequestHandler.cs#L183) as the call to CallIntoConnection might be returning a response with error but not really throwing exception.

Due to this the error in stream is not being recorded in seen exceptions and we can't see the real reason on why retries happened.

Can you please (if possible) fix (https://github.com/elasticsearch/elasticsearch-net/blob/develop/src/Elasticsearch.Net/Connection/RequestHandlers/RequestHandler.cs#L124) to pass error details which we can capture in the SeenExceptions list in the handler?