fix 100% cpu usage on linux while using sqs

[and800]

fix celery/celery#5299

kombu uses cURL multi_socket API, which is non-blocking callback-based API. But it doesn't work well in linux, because of epoll weird file descriptor management. When you use epoll, you must always remove FD from epoll before closing FD, but when curl decides that it doesn't need some particular socket, it will close that socket and after that it will inform you that the socket is closed - so you don't have a chance to remove it from epoll in time. See https://curl.haxx.se/libcurl/c/curl_multi_socket_action.html for more info about this API.

In our case we periodically query an SQS queue, and curl performs the queries via keep-alive https connection. But sometimes (about once per minute or two) SQS sends a response with Connection: close header, then curl closes the socket.

I suggest this workaround: before every curl invocation, let's remove from our selector every curl-related file descriptor. And after we get control back from curl, we insert those back (most of the time curl will update socket states, so we should not forget to consider it). So as a result we are going to work with epoll in old-fashioned select/poll way.

[auvipy]

thanks for handling this, would need to update test too

[and800]

yup, that's still a draft, not completely sure about it

[auvipy]

OK keep working on it

[and800]

It seems to work well @auvipy

[auvipy]

LGTM