ESP32S2 Content-Length: 0 responses timeout, sometimes exception

[anecdata]

This URL returns a valid HTTP 200 status code, headers including a 'Content-Length': '0' header, and no message-body.

This URL / code:

for _ in range(0, 5):
    response = requests.get("https://httpbin.org/status/200")
    print(response.status_code)
    print(response.reason)
    print(response.headers)
    print(response.text)

returns right away (less than a second) consistently in a browser or in python 3 (macOS).

In CircuitPython, each iteration takes about a minute to return, and sometimes gets an exception (usually the exception can happen in a loop of a few iterations):

RuntimeError: Unable to read HTTP response.

Interestingly, a shorter requests timeout in CircuitPython: requests.get("https://httpbin.org/status/200", timeout=5), comes back in 5 seconds rather than a minute, and seems far less prone to the "Unable to read HTTP response" exception so far.

[tannewt]

I suspect this is an adafruit_requests bug rather than a CircuitPython bug. In python3 are you using real requests or adafruit_requests?

[anecdata]

Real requests. Not sure how to do it with adafruit_requests (tried a few things), that's dependent on circuitpython core modules, no? Or would it be done with Blinka? If someone can point me in the right direction, I can probably figure it out.

Happy to transfer this back to the library.

[tannewt]

Here is an example: https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/master/examples/requests_simpletest_cpython.py

[anecdata]

http://httpbin.org/status/200 returns with 200 right away on numerous iterations (only one occurrence of "RuntimeError: Unable to read HTTP response"; haven't been able to replicate)

For "https://httpbin.org/status/200", not sure how to get past RuntimeError: ssl_context must be set before using adafruit_requests for https for an apples-to-apples comparison

[anecdata]

Actually, if I do everything in the right sequence (including import ssl) from scratch, https results in ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

[tannewt]

Here is an ssl example: https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/master/examples/requests_https_cpython.py

Is that what you did?

[anecdata]

Yes. Same error with that exact code. I think there's just no default certificate in my python3 environment? I poked around the CPython ssl module docs, it gets thick pretty fast. Still reading.

[anecdata]

OK, tried a bunch of things, even what seemed to be the recommended approach with SSLContext.wrap_socket() didn't get a connection, but this worked for testing purposes:

https = requests.Session(socket, ssl._create_unverified_context())

With this change to https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/master/examples/requests_https_cpython.py that code runs fine.

Then, appending my code snippet from above (adjusting variable name as needed)... runs fine, returns immediately in every case so far with 200 status code.

So it doesn't appear to be a library issue? (or if it is, it's a complex interaction between cp & library)

[tannewt]

Ok, I suspect this is how recv_into times out. Requests will fetch 32 byte chunks for parsing headers because it doesn't know the long they are. If recv_into waits for all 32 bytes which never come then the time out dictates how fast it happens.

[anecdata]

Somewhere along the way, this got fixed.

Adafruit CircuitPython 6.2.0-beta.0 on 2021-01-22; FeatherS2 with ESP32S2
adafruit-circuitpython-bundle-6.x-mpy-20210122

r = requests.get("https://httpbin.org/status/200")
print(r.status_code)
print(r.reason)
print(r.headers)
print(r.text)

now gives:

200
bytearray(b'OK')
{'Server': 'gunicorn/19.9.0', 'Content-Type': 'text/html; charset=utf-8', 'Date': 'Sat, 23 Jan 2021 18:38:20 GMT', 'Connection': 'keep-alive', 'Access-Control-Allow-Credentials': 'true', 'Content-Length': '0', 'Access-Control-Allow-Origin': '*'}

(r.content is b'')

Closing.