Quamash crashes when no network is available #42

Insoleet · 2015-09-14T21:14:52Z

Here is a nice bug to fix...

When running our app in a network namespace without internet connexion ( https://unix.stackexchange.com/questions/68956/block-network-access-of-a-process ), quamash sometimes crashes with the following error :

DEBUG:main:Exception handler executing
ERROR:main:Future exception was never retrieved
future: <Future finished exception=gaierror(-2, 'Name or service unknown')>
Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/quamash/__init__.py", line 89, in run
    r = callback(*args, **kwargs)
  File "/usr/lib/python3.4/socket.py", line 533, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service unknown

The text was updated successfully, but these errors were encountered:

Insoleet · 2015-12-10T22:15:59Z

So after analyzing the problem, it seems to be something deep.

This happens when aiohttp uses loop.getaddrinfo which runs in an executor : https://github.com/python/asyncio/blob/217da1db65a1eca0ea1e14362e199f048a99ec84/asyncio/base_events.py#L536

Sometimes, the future is deleted, and the exception was not catched. The exception is then handled in the exception_handler and crashes my application. It seems like the QThreadExecutor might have a wrong behaviour somehow. But I'm really bad with threads so I have trouble analyzing it more...

harvimt · 2015-12-10T23:12:29Z

things to try:

set the default executor to a concurrent.futures.ThreadPoolExecutor

loop.set_default_executor(concurrent.futures.ThreadPoolExecutor(10))

Modify getaddrinfo on the event loop (say by modifying QEventLoop)

def getaddrinfo(self, host, port, *, family=0, type=0, proto=0,

flags=0):
future = asyncio.Future()
if self._debug:
result = self._getaddrinfo_debug(host, port, family, type,
proto, flags)
else:
result = socket.getaddrinfo(host, port, family, type, proto,
flags)
future.set_result(result)
return future

(that is to make it synchronous, but keep the same signature)

that should give you more info.

On Thu, Dec 10, 2015 at 2:16 PM, Insoleet notifications@github.com wrote:

So after analyzing the problem, it seems to be something deeper.

This happens when aiohttp uses loop.getaddrinfo which runs in an executor
:
https://github.com/python/asyncio/blob/217da1db65a1eca0ea1e14362e199f048a99ec84/asyncio/base_events.py#L536

Sometimes, the future is deleted, and the exception was not catched. The
exception is then handled in the exception_handler and crashes my
application. It seems like the QThreadExecutor might have a wrong behaviour
somehow. But I'm really bad with threads so I have trouble analyzing it
more...

—
Reply to this email directly or view it on GitHub
#42 (comment).

Insoleet · 2015-12-11T00:46:39Z

Thanks.

Here is what I could find :

With the concurrent ThreadPoolExecutor, the same crash happens
With a synchronized getaddrinfo, I can get a bigger stack, but there is still nothing linked to a call in my code :

DEBUG:main:async_exception_handler:Exception handler executing
ERROR:main:async_exception_handler:Task exception was never retrieved
future: <Task finished coro=<__iter__() done, defined at /home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py:454> exception=ClientOSError(None, 'Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]')>
Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 557, in _create_connection
    server_hostname=hinfo['hostname'] if sslcontext else None)
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/base_events.py", line 569, in create_connection
    type=socket.SOCK_STREAM, proto=proto, flags=flags)
  File "/home/inso/code/quamash/quamash/__init__.py", line 481, in getaddrinfo
    result = socket.getaddrinfo(host, port, family, type, proto, flags)
socket.gaierror: [Errno -2] Nom ou service inconnu

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 289, in connect
    transport, proto = yield from self._create_connection(req)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 580, in _create_connection
    (req.host, req.port, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Can not connect to moul.re:8999 [None]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/tasks.py", line 238, in _step
    result = next(coro)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 456, in __iter__
    resp = yield from self._coro
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 173, in _request
    conn = yield from self._connector.connect(req)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 299, in connect
    .format(key, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]

Some questions :

Why in the asynchronous case the gaierror does not raise the aiohttp specific exceptions ?
Why aren't they raised from my code, but they are from nowhere in aiohttp code ?

harvimt · 2015-12-11T00:54:06Z

the sync version of getaddrinfo I sent you doesn't actually handle
exceptions correctly:

try:

def getaddrinfo(self, host, port, *, family=0, type=0, proto=0,

flags=0):
future = asyncio.Future()
try:
if self._debug:
result = self._getaddrinfo_debug(host, port, family, type,
proto, flags)
else:
result = socket.getaddrinfo(host, port, family, type,
proto, flags)
future.set_result(result)
except Exception as e:
future.set_exception(result)
return future

On Thu, Dec 10, 2015 at 4:46 PM, Insoleet notifications@github.com wrote:

Thanks.

Here is what I could find :

With the concurrent ThreadPoolExecutor, the same crash happens

With a synchronized getaddrinfo, I can get a bigger stack, but there
is still nothing linked to a call in my code :

DEBUG:main:async_exception_handler:Exception handler executing
ERROR:main:async_exception_handler:Task exception was never retrieved
future: <Task finished coro=<iter() done, defined at /home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py:454> exception=ClientOSError(None, 'Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]')>
Traceback (most recent call last):
File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 557, in _create_connection
server_hostname=hinfo['hostname'] if sslcontext else None)
File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/base_events.py", line 569, in create_connection
type=socket.SOCK_STREAM, proto=proto, flags=flags)
File "/home/inso/code/quamash/quamash/init.py", line 481, in getaddrinfo
result = socket.getaddrinfo(host, port, family, type, proto, flags)
socket.gaierror: [Errno -2] Nom ou service inconnu

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 289, in connect
transport, proto = yield from self._create_connection(req)
File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 580, in _create_connection
(req.host, req.port, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Can not connect to moul.re:8999 [None]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/tasks.py", line 238, in _step
result = next(coro)
File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 456, in iter
resp = yield from self._coro
File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 173, in _request
conn = yield from self._connector.connect(req)
File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 299, in connect
.format(key, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]

Some questions :

Why in the asynchronous case the gaierror does not raise the aiohttp
specific exceptions ?

Why aren't they raised from my code, but they are from nowhere in
aiohttp code ?

—
Reply to this email directly or view it on GitHub
#42 (comment).

Insoleet · 2015-12-11T07:36:42Z

When catching the exception and using set_exception, here is what I get :

DEBUG:main:async_exception_handler:Exception handler executing
ERROR:main:async_exception_handler:Task exception was never retrieved
future: <Task finished coro=<__iter__() done, defined at /home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py:454> exception=ClientOSError(None, 'Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]')>
Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 557, in _create_connection
    server_hostname=hinfo['hostname'] if sslcontext else None)
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/base_events.py", line 581, in create_connection
    infos = f1.result()
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/futures.py", line 275, in result
    raise self._exception
  File "/home/inso/code/quamash/quamash/__init__.py", line 482, in getaddrinfo
    result = socket.getaddrinfo(host, port, family, type, proto, flags)
socket.gaierror: [Errno -2] Nom ou service inconnu

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 289, in connect
    transport, proto = yield from self._create_connection(req)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 580, in _create_connection
    (req.host, req.port, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Can not connect to moul.re:8999 [None]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/tasks.py", line 238, in _step
    result = next(coro)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 456, in __iter__
    resp = yield from self._coro
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 173, in _request
    conn = yield from self._connector.connect(req)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 299, in connect
    .format(key, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]

As you can see, the stacktrace is still out of the code. The exception is raised after being read in f1.result().

harvimt added the bug label Sep 14, 2015

harvimt added this to the v.5 milestone Sep 14, 2015

harvimt self-assigned this Sep 14, 2015

Insoleet mentioned this issue Dec 11, 2015

Some exceptions are not raised in the correct stack aio-libs/aiohttp#674

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Quamash crashes when no network is available #42

Quamash crashes when no network is available #42

Insoleet commented Sep 14, 2015

Insoleet commented Dec 10, 2015

harvimt commented Dec 10, 2015

Insoleet commented Dec 11, 2015

harvimt commented Dec 11, 2015

Insoleet commented Dec 11, 2015

Quamash crashes when no network is available #42

Quamash crashes when no network is available #42

Comments

Insoleet commented Sep 14, 2015

Insoleet commented Dec 10, 2015

harvimt commented Dec 10, 2015

Insoleet commented Dec 11, 2015

harvimt commented Dec 11, 2015

Insoleet commented Dec 11, 2015