Request Timeout V Network Timeout

[ydawant]

I'm not sure if this is the best place to bring this up, but I'm having some issues being able to differentiate a network timeout from a request timeout.

Is there a way to set the connection timeout separate from the request timeout?

Basically I want to be able to set different thresholds for when a node is not responding at all (which would result in a ConnectTimeout to one in which the request is just taking too long to process ReadTimeout

ConnectTimeout: HTTPConnectionPool(host=<host>, port=9200): Max retries exceeded with url: /_bulk (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f44f2c8f510>, 'Connection to <node> timed out. (connect timeout=10)'))

v

ReadTimeout: HTTPConnectionPool(host=<host> port=9200): Read timed out. (read timeout=10)

I'd like to set connect_timeout = 1 and read_timeout = 20

[ydawant]

Looking at this code here:

elasticsearch-py/elasticsearch/connection/http_requests.py

Line 79 in a03504c

except Exception as e:

Would it make sense to treat a ConnectTimeout separate from a ReadTimeout? Those seem like different errors to me, and I would expect a ConnectTimeout to raise a ConnectionError as the node is unreachable. Does that make sense or am I thinking about this wrong?

https://github.com/requests/requests/blob/master/requests/exceptions.py#L53
v
https://github.com/requests/requests/blob/master/requests/exceptions.py#L60

[fxdgear]

@ydawant So I'm trying to understand your usecase here.

You want to have a different timeout for the different types of exceptions?

So what happens if you send a request to Elasticsearch and it just takes 15s for Elasticsearch to accept/process/respond. You'll end up with a false negative where your code will be raising a ConnectTimeout exception inappropriately.

[ydawant]

@fxdgear Thanks for responding.

Yes, that is correct. The way that I see it, there are 2 distinct errors.

1. ConnectTimeout which is defined as "The request timed out while trying to connect to the remote server."

2. ReadTimeout which is defined as "The server did not send any data in the allotted amount of time."

We are using an RRDNS system, so we pass in a list of nodes. It could happen that one of those nodes drops out of the cluster, in which case we get a ConnectTimeout. This is what we want to catch and mark as dead. The other case is a ReadTimeout, we get back ReadTimeout when the operation could not be completed in time but we were able to connect to the cluster

In the current setup, both of these are bubbled up as ConnectionTimeout here -

elasticsearch-py/elasticsearch/connection/http_requests.py

Line 83 in a03504c

if isinstance(e, requests.Timeout):

I would like to be able to differentiate between the two. I understand that setting a separate timeout would be difficult, but maybe just treat the two timeouts as different types of errors.

[ydawant]

I was thinking something along these lines: http://docs.python-requests.org/en/master/user/advanced/#timeouts

So we could set a sane connection timeout, to make sure we can make a connection to the remote machine, and then a read timeout to set boundaries on how long our ES operation has to process.

Does that make sense?

[fxdgear]

have you looked at https://elasticsearch-py.readthedocs.io/en/master/#sniffing

The client already handles nodes leaving/joining the cluster for you.

[fxdgear]

Also is there any reason you're using requests instead of urllib3?

Using urllib3 is more performant that requests. Though there are cases where requests is the only option. So I'd understand why in that case.

[ydawant]

I did take a quick look at sniffing - but could take another look. I believe my initial hesitation was that the sniff_on_connection_fail was not going to get triggered, as a bad node throws a ConnectionTimeout not ConnectionError. We could set retry_on_timeout = True to get around that, though.

Also is there any reason you're using requests instead of urllib3?

We are using urllib3, I pointed to requests because the exception being caught is requests.Timeout

[ydawant]

Also, with sniffing, I didn't see this anywhere, but that wouldn't throw any sort of error or log if the nodes changed, would it? Just something to track that the nodes changed?

[fxdgear]

@ydawant I don't feel comfortable having users potentially raising incorrect errors due to setting different timeouts and getting some unexpected behaviors. I just see this raising more issues than it solves. So I'm sorry about that. But I do understand your requirement. Though, I do feel that it's solved using sniffing.

wrt to your 2nd question:

wouldn't throw any sort of error or log if the nodes changed, would it?

Not sure I follow here. If you've configured the Elasticsearch object to to sniff on the various requirements:

1. sniff on start = Sniff when you instantiate.
2. sniff on connection fail = sniff when you get any connection failures. So if there's a connection failure sniff the cluster state again for any nodes that have been added or removed from the cluster

[ydawant]

Great. Thanks for your time Nick! I'll implement sniffing and go from there. You've been very helpful.

[ydawant]

I'm going to close the issue for now.