aiohttp request exceptions can't be caught sometimes when encountering bad http message #3031
Description
Long story short
When encountering bad http message, aiohttp request exceptions can't be caught sometimes.
For example, I use code like below to detect if a proxy is working:
import sys
import asyncio
import logging
import aiohttp
DETECT_WEBSITE = 'httpbin.org'
async def proxy_req(proxy_url):
if proxy_url.startswith('https://'):
protocol = 'https'
else:
protocol = 'http'
detect_url = f'{protocol}://{DETECT_WEBSITE}/get'
proxy_url = proxy_url.replace('https://', 'http://')
print('Detect url:', detect_url)
print('Proxy url:', proxy_url)
try:
async with aiohttp.ClientSession() as session:
async with session.get(detect_url,
proxy=proxy_url,
headers={'User-Agent': 'Mozilla/5.0'},
timeout=10) as resp:
text = await resp.text()
print('Response text:')
print(text)
except Exception as exc:
logging.error(exc)
if __name__ == '__main__':
proxy_url = sys.argv[1]
loop = asyncio.get_event_loop()
loop.run_until_complete(proxy_req(proxy_url))Expected behaviour
In the above code, I tried to catch all exceptions when doing request, so if a request exception happened, it should always be logged normally.
Actual behaviour
When I detect some broken proxies, for that proxy, sometimes the exception can be caught normally and logged, but sometimes the exception is not caught but are thrown directly.
For example, detect the broken proxy http://218.106.205.145:8080 ( When using this broken proxy to doing requests, it will return two different groups of reponse headers ), the output may look like below ( The first execution thrown exception, the second execution caught exception and logged ):
✗ python test.py http://218.106.205.145:8080
Detect url: http://httpbin.org/get
Proxy url: http://218.106.205.145:8080
Exception in callback None()
handle: <Handle cancelled>
Traceback (most recent call last):
File "/Users/xxx/Coding/zzz/venv/lib/python3.6/site-packages/aiohttp/client_proto.py", line 161, in data_received
messages, upgraded, tail = self._parser.feed_data(data)
File "aiohttp/_http_parser.pyx", line 297, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadHttpMessage: 400, message='invalid constant string'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/events.py", line 145, in _run
self._callback(*self._args)
File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/selector_events.py", line 730, in _read_ready
self._protocol.data_received(data)
File "/Users/xxx/Coding/zzz/venv/lib/python3.6/site-packages/aiohttp/client_proto.py", line 177, in data_received
self.transport.close()
AttributeError: 'NoneType' object has no attribute 'close'
Response text:
HTTP/1.1 500 OK
Date: Mon, 28 May 2018 09:43:07 GMT
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset="UTF-8"
Content-Length: 2173
Accep
✗ python test.py http://218.106.205.145:8080
Detect url: http://httpbin.org/get
Proxy url: http://218.106.205.145:8080
ERROR:root:400, message='invalid constant string'
Steps to reproduce
Run the code above to detect the broken proxy like http://218.106.205.145:8080
Your environment
aiohttp 3.2.1
Mac OS X
aiohttp client