Issue with firmwares 5.50 and older #55

Kane610 · 2020-10-29T14:26:17Z

Originally reported in HASS repo home-assistant/core#42415

There is an issue with the transition to httpx and firmwares of 5.50 and older. It works perfectly fine on newer firmwares.

Tested on a 5.75 which have Apache webserver rather than Boa and there are more differences when looking at network traces. Looking at a web request with unauthorised response setting up digest there is much more data related to the same session than on a device with boa, with boa every session ends after web server response, so looking in wire shark gives a much narrower scope per following.

Requests seemed to handle all this perfectly fine down to 5.20 or even earlier. So I think httpx is sensitive to what responses is sent from the Boa webserver.

The change from requests to httpx: https://github.com/Kane610/axis/pull/52/files
The change to async httpx: https://github.com/Kane610/axis/pull/53/files

Kane610 · 2020-10-29T14:29:33Z

Failing using httpx:

Session is created

axis/axis/vapix.py

Line 40 in 529e434

self.session = httpx.AsyncClient(

Retrieving users

axis/axis/vapix.py

Line 203 in 529e434

users = await self.request("get", PWDGRP_URL)

yield a raised transport error from

axis/axis/vapix.py

Line 213 in 529e434

response = await self.session.request(method, url, **kwargs)

Full TCP dump capturing a failing session.
axis_v41_fw5.51_httpx.pcap.zip

These two are slightly modified to not use gather in order to be easier to compare with the synchronous flow of requests. 5.51 is failing and 5.75 is succeeding.
axis_v41_fw5.51_httpx_modified_dont_use_gather.pcap.zip
axis_v41_fw5.75_httpx_modified_dont_use_gather.pcap.zip

main.py ignores the first transport error when retrieving users because it needs admin level and main is mainly for verifying function calls.

python3 -m axis 10.0.0.254 root pass --params -D
10.0.0.254, root, pass, 80, False, True
Using selector: KqueueSelector
Connecting to Axis device
http://10.0.0.254:80/axis-cgi/pwdgrp.cgi?action=get {}
HTTP Request: GET http://10.0.0.254:80/axis-cgi/pwdgrp.cgi?action=get "HTTP/1.1 401 Unauthorized"
peer unexpectedly closed connection
Error connecting to the Axis device at 10.0.0.254
{}
http://10.0.0.254:80/axis-cgi/apidiscovery.cgi {'json': {'method': 'getApiList', 'apiVersion': '1.0', 'context': 'Axis library'}}
HTTP Request: POST http://10.0.0.254:80/axis-cgi/apidiscovery.cgi "HTTP/1.1 404 Not Found"
<Response [404 Not Found]>, 404 Client Error: Not Found for url: http://10.0.0.254:80/axis-cgi/apidiscovery.cgi
For more information check: https://httpstatuses.com/404
''
http://10.0.0.254:80/axis-cgi/param.cgi?action=list&group=root.Properties {}
http://10.0.0.254:80/axis-cgi/param.cgi?action=list&group=root.Brand {}
http://10.0.0.254:80/axis-cgi/param.cgi?action=list&group=root.StreamProfile {}
http://10.0.0.254:80/axis-cgi/param.cgi?action=list&group=root.Input {}
http://10.0.0.254:80/axis-cgi/param.cgi?action=list&group=root.IOPort {}
http://10.0.0.254:80/axis-cgi/param.cgi?action=list&group=root.Output {}
HTTP Request: GET http://10.0.0.254:80/axis-cgi/param.cgi?action=list&group=root.Properties "HTTP/1.1 401 Unauthorized"
HTTP Request: GET http://10.0.0.254:80/axis-cgi/param.cgi?action=list&group=root.Brand "HTTP/1.1 401 Unauthorized"
HTTP Request: GET http://10.0.0.254:80/axis-cgi/param.cgi?action=list&group=root.StreamProfile "HTTP/1.1 401 Unauthorized"
HTTP Request: GET http://10.0.0.254:80/axis-cgi/param.cgi?action=list&group=root.IOPort "HTTP/1.1 401 Unauthorized"
HTTP Request: GET http://10.0.0.254:80/axis-cgi/param.cgi?action=list&group=root.Input "HTTP/1.1 401 Unauthorized"
HTTP Request: GET http://10.0.0.254:80/axis-cgi/param.cgi?action=list&group=root.Output "HTTP/1.1 401 Unauthorized"
peer unexpectedly closed connection
Traceback (most recent call last):
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_exceptions.py", line 326, in map_exceptions
    yield
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py", line 1502, in _send_single_request
    (status_code, headers, stream, ext,) = await transport.arequest(
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 200, in arequest
    response = await connection.arequest(
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpcore/_async/connection.py", line 100, in arequest
    return await self.connection.arequest(method, url, headers, stream, ext)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpcore/_async/http11.py", line 72, in arequest
    ) = await self._receive_response(timeout)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpcore/_async/http11.py", line 133, in _receive_response
    event = await self._receive_event(timeout)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpcore/_async/http11.py", line 169, in _receive_event
    event = self.h11_state.next_event()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.RemoteProtocolError: peer unexpectedly closed connection

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/robertsv/Development/axis/axis/vapix.py", line 213, in request
    response = await self.session.request(method, url, **kwargs)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py", line 1371, in request
    response = await self.send(
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py", line 1406, in send
    response = await self._send_handling_auth(
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py", line 1444, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py", line 1476, in _send_handling_redirects
    response = await self._send_single_request(request, timeout)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py", line 1502, in _send_single_request
    (status_code, headers, stream, ext,) = await transport.arequest(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_exceptions.py", line 343, in map_exceptions
    raise mapped_exc(message, **kwargs) from exc  # type: ignore
httpx.RemoteProtocolError: peer unexpectedly closed connection

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/robertsv/Development/axis/axis/__main__.py", line 99, in <module>
    asyncio.run(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 612, in run_until_complete
    return future.result()
  File "/Users/robertsv/Development/axis/axis/__main__.py", line 57, in main
    await device.vapix.initialize()
  File "/Users/robertsv/Development/axis/axis/vapix.py", line 103, in initialize
    await self.initialize_param_cgi(preload_data=False)
  File "/Users/robertsv/Development/axis/axis/vapix.py", line 161, in initialize_param_cgi
    await asyncio.gather(*tasks)
  File "/Users/robertsv/Development/axis/axis/param_cgi.py", line 80, in update_ports
    await asyncio.gather(
  File "/Users/robertsv/Development/axis/axis/api.py", line 38, in update
    raw = await self._request("get", path)
  File "/Users/robertsv/Development/axis/axis/vapix.py", line 238, in request
    raise RequestError("Connection error: {}".format(errc))
axis.errors.RequestError: Connection error: peer unexpectedly closed connection
/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py:1781: UserWarning: Unclosed <httpx.AsyncClient object at 0x7fb878fffd30>. See https://www.python-httpx.org/async/#opening-and-closing-clients for details.

Kane610 · 2020-10-29T14:34:19Z

Succeeding using requests

Session is created

axis/axis/configuration.py

Line 23 in e3bfe72

session = requests.Session()

Retrieving users

axis/axis/vapix.py

Line 184 in e3bfe72

users = self.request("get", PWDGRP_URL)

does not yield a transport error here

axis/axis/utils.py

Line 15 in e3bfe72

response = session(url, **kwargs)

TCP dump of a successful session using requests
axis_v37_fw5.51_requests.pcap.zip

Kane610 · 2020-10-29T18:14:21Z

I was looking at moving from using get to post when retrieving users and noticed that this also failed, in this case both on 5.51 and 5.75. Going to even newer seems to work though.

florimondmanca · 2020-10-29T22:56:05Z

@Kane610 I'd recommend starting from a minimal test script and then ramping up to the actual code in the codebase. For example, what does a simple GET request to the device look like, does it succeed? If so, build a client like in the actual codebase, retry, and slowly make your way from there... Until you find at what point exactly things start looking off. Then share what this "minimal reproducible example" looks like, stripping away anything that doesn't make the bug go away.

If you haven't already you can review our Requests Compatibility guide in case that could give you som hints: https://www.python-httpx.org/compatibility/

Kane610 · 2020-10-30T06:02:38Z

@florimondmanca absolutely i will try to reproduce it using a simple script. Indeed I followed the requests pages you'd written. And as stated above it works on newer devices so there is probably som quirk the older firmwares do that requests handle but httpx doesn't.

Kane610 · 2020-10-30T06:46:31Z

@florimondmanca I've created the smallest thing to reproduce this issue, it fails as expected on some older devices and works fine on newer.

import httpx

client = httpx.Client(auth=httpx.DigestAuth("root", "pass"))
res = client.request("get", "http://10.0.0.254:80/axis-cgi/pwdgrp.cgi?action=get")

❯ python3 axis/debug.py
Traceback (most recent call last):
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_exceptions.py", line 326, in map_exceptions
    yield
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py", line 861, in _send_single_request
    (status_code, headers, stream, ext) = transport.request(
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 200, in request
    response = connection.request(
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 100, in request
    return self.connection.request(method, url, headers, stream, ext)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpcore/_sync/http11.py", line 72, in request
    ) = self._receive_response(timeout)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpcore/_sync/http11.py", line 133, in _receive_response
    event = self._receive_event(timeout)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpcore/_sync/http11.py", line 169, in _receive_event
    event = self.h11_state.next_event()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.RemoteProtocolError: peer unexpectedly closed connection

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "axis/debug.py", line 5, in <module>
    res = client.request("get", "http://10.0.0.254:80/axis-cgi/pwdgrp.cgi?action=get")
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py", line 733, in request
    return self.send(
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py", line 767, in send
    response = self._send_handling_auth(
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py", line 805, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py", line 837, in _send_handling_redirects
    response = self._send_single_request(request, timeout)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_client.py", line 861, in _send_single_request
    (status_code, headers, stream, ext) = transport.request(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/robertsv/Development/Virtualenvs/python38/lib/python3.8/site-packages/httpx/_exceptions.py", line 343, in map_exceptions
    raise mapped_exc(message, **kwargs) from exc  # type: ignore
httpx.RemoteProtocolError: peer unexpectedly closed connection

yuriy1337 · 2020-11-30T01:58:15Z

I'm getting the same error with:

AXIS M3004 Network Camera
Firmware version: 5.51.7.2

This the the newest firmware for this device.

cdeler · 2020-12-01T17:27:06Z

Hello @Kane610

I checked the PR to h11 with the real device. I mentioned the results there:

requests lib:

In [1]: import requests
In [2]: from requests.auth import HTTPDigestAuth
In [3]: url = "http://[REDACTED]/axis-cgi/pwdgrp.cgi?action=get"
In [4]: requests.get(url, auth=HTTPDigestAuth("[REDACTED]", "[REDACTED]"))
Out[4]: <Response [200]>

without fix:

In [1]: import httpx
In [2]: url = "http://[REDACTED]/axis-cgi/pwdgrp.cgi?action=get"
In [3]: auth = httpx.DigestAuth("[REDACTED]", "[REDACTED]")
In [4]: client = httpx.Client(auth=auth)
In [5]: client.get(url)
RemoteProtocolError: peer unexpectedly closed connection

with fix:

In [1]: import httpx
In [2]: url = "http://[REDACTED]/axis-cgi/pwdgrp.cgi?action=get"
In [3]: auth = httpx.DigestAuth("[REDACTED]", "[REDACTED]")
In [4]: client = httpx.Client(auth=auth)
In [5]: client.get(url)
Out[5]: <Response [200 OK]>

Kane610 · 2020-12-01T18:09:37Z

Awesome! Thanks for the update @cdeler

Kane610 · 2020-12-01T20:32:15Z

I've verified the fix to work on both a 5.51 device and a 5.75 and additionally that newer firmwares keep on working 🎉

Kane610 · 2021-01-01T22:08:30Z

This has been solved upstream so there is no need to keep this issue open anymore.

This was referenced Nov 3, 2020

httpx cannot process http/1.1 responses with headers and body delimited by \n\n or \n\n\n\n (but requests can) encode/httpx#1378

Closed

Support for servers with broken line endings python-hyper/h11#7

Closed

Kane610 mentioned this issue Jan 1, 2021

Bump H11 library to support non RFC line endings home-assistant/core#44735

Merged

21 tasks

Kane610 closed this as completed Jan 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue with firmwares 5.50 and older #55

Issue with firmwares 5.50 and older #55

Kane610 commented Oct 29, 2020 •

edited

Loading

Kane610 commented Oct 29, 2020 •

edited

Loading

Kane610 commented Oct 29, 2020 •

edited

Loading

Kane610 commented Oct 29, 2020

florimondmanca commented Oct 29, 2020

Kane610 commented Oct 30, 2020

Kane610 commented Oct 30, 2020 •

edited

Loading

yuriy1337 commented Nov 30, 2020 •

edited

Loading

cdeler commented Dec 1, 2020 •

edited

Loading

Kane610 commented Dec 1, 2020

Kane610 commented Dec 1, 2020

Kane610 commented Jan 1, 2021

Issue with firmwares 5.50 and older #55

Issue with firmwares 5.50 and older #55

Comments

Kane610 commented Oct 29, 2020 • edited Loading

Kane610 commented Oct 29, 2020 • edited Loading

Kane610 commented Oct 29, 2020 • edited Loading

Kane610 commented Oct 29, 2020

florimondmanca commented Oct 29, 2020

Kane610 commented Oct 30, 2020

Kane610 commented Oct 30, 2020 • edited Loading

yuriy1337 commented Nov 30, 2020 • edited Loading

cdeler commented Dec 1, 2020 • edited Loading

requests lib:

without fix:

with fix:

Kane610 commented Dec 1, 2020

Kane610 commented Dec 1, 2020

Kane610 commented Jan 1, 2021

Kane610 commented Oct 29, 2020 •

edited

Loading

Kane610 commented Oct 29, 2020 •

edited

Loading

Kane610 commented Oct 29, 2020 •

edited

Loading

Kane610 commented Oct 30, 2020 •

edited

Loading

yuriy1337 commented Nov 30, 2020 •

edited

Loading

cdeler commented Dec 1, 2020 •

edited

Loading