AttributeError: 'NoneType' object has no attribute 'get_extra_info' with aiohttp==3.8.3 when using a proxy #6239
Labels
bug
client
need pull request
regression
Something that used to work stopped working "as before" after upgrade
reproducer: present
This PR or issue contains code, which reproduce the problem described or clearly understandable STR
Comments
webknjaz
added
regression
|
@bmbouter @jborean93 have you seen this problem after the proxy code update? |
|
I kind of think I did, but I tested so many permutations it's difficult to say 100%. If I am remembering correctly, if TLS trust was required but one or both of the certs was untrusted from the local trust store, I would get this traceback. Hopefully this is helpful. |
|
@bmbouter could you check if the reproducer above works for you locally? On my machine it's Traceback (most recent call last):
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 985, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 1056, in create_connection
raise exceptions[0]
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 1041, in create_connection
sock = await self._connect_sock(
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 955, in _connect_sock
await self.sock_connect(sock, address)
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/selector_events.py", line 502, in sock_connect
return await fut
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/selector_events.py", line 537, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
TimeoutError: [Errno 110] Connect call failed ('218.235.10.214', 8393)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "~/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "~/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "~/src/experiments/aiohttp-repros/issue-6239/test_case.py", line 17, in <module>
main()
File "~/src/experiments/aiohttp-repros/issue-6239/test_case.py", line 13, in main
asyncio.run(test_case())
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "~/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "~/src/experiments/aiohttp-repros/issue-6239/test_case.py", line 6, in test_case
async with session.get("https://api.ipify.org", proxy="http://218.235.10.214:8393") as response:
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/client.py", line 1140, in __aenter__
self._resp = await self._coro
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 543, in connect
proto = await self._create_connection(req, traces, timeout)
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 904, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 1230, in _create_proxy_connection
transport, proto = await self._create_direct_connection(
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 1205, in _create_direct_connection
raise last_exc
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 1174, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File "~/.pyenv/versions/aiohttp-repros-pyenv-py3.9.0/lib/python3.9/site-packages/aiohttp/connector.py", line 991, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientProxyConnectionError: Cannot connect to host 218.235.10.214:8393 ssl:default(looks like the initial report may have missed the first part of the traceback) |
|
same error here. File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/client.py", line 536, in _request
req, traces=traces, timeout=real_timeout
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/connector.py", line 543, in connect
proto = await self._create_connection(req, traces, timeout)
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/connector.py", line 904, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/connector.py", line 1329, in _create_proxy_connection
timeout=timeout,
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/connector.py", line 1126, in _start_tls_connection
tls_transport
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/base_protocol.py", line 58, in connection_made
tcp_nodelay(tr, True)
File "/Users/dev/Library/Caches/pypoetry/virtualenvs/proj/lib/python3.7/site-packages/aiohttp/tcp_helpers.py", line 26, in tcp_nodelay
sock = transport.get_extra_info("socket")
AttributeError: 'NoneType' object has no attribute 'get_extra_info' |
I don't think I did. That proxy server is no longer online. That's why you're getting a Timeout now. Let me grab another one that reproduces the original issue. |
|
Here's the revised test case with a different proxy. # test_case.py
import asyncio
import aiohttp
async def test_case():
async with aiohttp.ClientSession() as session:
async with session.get("https://api.ipify.org", proxy="http://42.194.222.36:3128") as response:
text = await response.text()
print(text)
def main():
asyncio.run(test_case())
if __name__ == "__main__":
main()Here's the copy/pasted output from my terminal. (Full, not truncated.) Here's a screenshot of my terminal:
|
|
Here's two additional proxies that exhibit the same behavior: |
|
Note: These proxies may or may not be broken. I'm not totally sure. The real bug isn't that they were necessarily working before I don't want my code to have to catch Here's a much more detailed output. Here's the script I'm running # test_case.py
import asyncio
import traceback
import aiohttp
PROXIES = [
"http://42.194.222.36:3128",
"http://178.32.107.236:3128",
"http://101.34.116.37:7890",
]
async def test_all():
async with aiohttp.ClientSession() as session:
for proxy in PROXIES:
try:
async with session.get("https://api.apify.org", proxy=proxy) as response:
text = await response.text()
except Exception as e:
print(f"{proxy} : Error")
print(traceback.format_exc())
else:
print(f"{proxy} : OK")
def main():
asyncio.run(test_all())
if __name__ == "__main__":
main()Here's the output on (Note the very helpful Here's the output with Note that all of the helpful exceptions disappeared. |
Fair enough. FWIW the best place to add regression tests is https://github.com/aio-libs/aiohttp/blob/master/tests/test_proxy_functional.py. |
|
Would you accept a pull request with a failing test, but no fix? I don't know how to fix this issue, but I could definitely add the test. |
Yes, we do this frequently by marking it with xfail, so we can come back to fix it later on. |
Yes, we're trying to follow https://pganssle-talks.github.io/xfail-lightning/. |
|
this commit breaks - cf641aa |
|
After replace https://github.com/aio-libs/aiohttp/blob/3.8/aiohttp/connector.py#L1212 |
|
So it looks like those proxies are trying to support HTTPS (TLS-in-TLS?) but there's some issue with the configuration? If that's the case, a specific exception(s) would be ideal there |
|
perhaps something similar needs to be done here |
|
@Dreamsorcerer any updates? |
|
No, I'm not familiar with this code. If someone wants to create a PR with a fix, we can take a look at it. Also doesn't look like anybody created a PR with a test yet, which was suggested before (which makes it more likely that one of the maintainers will then have time to find a fix). |
Sort of. TLS-in-TLS is tricky in CPython itself and requires monkey-patching of the stdlib to make it work, this is why it's considered a "tech preview". Before the patch, FWIW aiohttp could benefit from more regression tests. Plus a lot of those that exist could be more useful if rewritten using the new proxy.py-based pytest fixture instead of relying on mocks and monkey-patching.
I'm open to proposals. |
If it could raise It's still an issue in |
|
I am experience this issue as well, until it is fixed what is the best way to catch this error? |
5 tasks
|
Also experiencing this issue |
|
As of late I have been experiencing this issue as well using private proxies from the provider "Oxylabs", anyone come up with a fix yet? the edits in the above pr help with providing a better error message but I'm curious as to what is different with these proxies compared to the many others I've used without issue (these actually worked fine just a week ago but now seemingly only dont work in aiohttp) |
|
This #6703 does appear to fix it, but I had trouble getting |
john-parton
changed the title
|
I still see same issue 'NoneType' object has no attribute 'get_extra_info' . I used requests.get with this proxy normally. |
|
Here's an update as of today. Using Test Codeimport asyncio
import aiohttp
async def test_case():
async with aiohttp.ClientSession() as session:
async with session.get(
"https://api.ipify.org", proxy="http://185.38.111.1:8080"
) as response:
text = await response.text()
print(text)
def main():
asyncio.run(test_case())
if __name__ == "__main__":
main()Expected / Correct Behavior on
|
john-parton
changed the title
That help me, proxy is worked, thanks! |
|
Same here, full stacktrace in #3355 (comment). Is there any chance to get this fixed? Regression is two years old now. |
|
I don't know if this could help fixing the problem but it works on Windows. It fails on debian for me |
|
Any updates? |
|
I'm closing this because it seems to be a duplicate of #3355. Follow that one for updates. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Requests involving a proxy server sometimes fail with an
AttributeErrorwhich were working on versions prior to3.8.0To Reproduce
Full test case:
Expected behavior
Expected behavior (confirmed working with
aiohttp==3.7.4.post0)python3 test_case.py<prints my public ip>Logs/tracebacks
Python Version
aiohttp Version
multidict Version
yarl Version
OS
Ubuntu 21.10 x64
Related component
Client
Additional context
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: