Failed SSL handshake

[wzaremba]

CircuitPython version

Adafruit CircuitPython 7.3.3 on 2022-08-29; Adafruit MagTag with ESP32S2
Board ID:adafruit_magtag_2.9_grayscale

Code/REPL

import time
import wifi
import socketpool
import ssl
import adafruit_requests
from secrets import secrets
wifi.radio.connect(ssid=secrets['ssid'],password=secrets['password'])
print("my IP addr:", wifi.radio.ipv4_address)
pool = socketpool.SocketPool(wifi.radio)
session = adafruit_requests.Session(pool, ssl.create_default_context())
while True:
    response = session.get("https://api.um.warszawa.pl/api/action/dbtimetable_get?id=88cd555f-6f31-43ca-9de4-66c479ad5942&busstopId=4125&busstopNr=02&apikey=apikeygoeshere")
    data = response.json()
    print("data:",data)
    time.sleep(5)

Behavior

code.py wyjście:
my IP addr: 192.168.2.156
Ślad wyjątku (najnowsze wywołanie na końcu):
Plik "code.py", linia 12, w
Plik "adafruit_requests.py", linia 725, w get
Plik "adafruit_requests.py", linia 666, w request
Plik "adafruit_requests.py", linia 518, w _get_socket
RuntimeError: Sending request failed
blinking RED

Description

Hi,
error while calling Warsaw IOT data service.
https://api.um.warszawa.pl/api/action/dbtimetable_get?id=88cd555f-6f31-43ca-9de4-66c479ad5942&busstopId=4125&busstopNr=02&apikey=apikeygoeshere"
It works on other REST calls, but not on Circuit Python using or this program or build-in MagTag API refresh.

No idea how to proceed and what to check more, as my Python is limited.
I can give full URL in private message, here I have removed API key.

Additional information

No response

[anecdata]

What version of Requests are you using (helps to match line numbers)?

There are a few possibilities, one may be that the website uses an unsupported cipher. curl -iLv https://api.um.warszawa.pl/api/... may give some clues.

Another thing to do is add some exception printing statements within the Requests library, particularly in the except blocks in ._get_socket(). If you have trouble with this, we can coach you (Discord is easier if you can get onto the Adafruit server), or provide an edited version of the library to print the additional info.

[wzaremba]

Thanks @anecdata; here is the output of curl:

*   Trying 91.237.139.103...
* TCP_NODELAY set
* Connected to api.um.warszawa.pl (91.237.139.103) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=PL; ST=mazowieckie; L=Warszawa; OU=IT; O=Miasto Stoleczne Warszawa; CN=*.um.warszawa.pl
*  start date: Feb 14 08:27:06 2022 GMT
*  expire date: Feb 14 08:27:05 2023 GMT
*  subjectAltName: host "api.um.warszawa.pl" matched cert's "*.um.warszawa.pl"
*  issuer: C=PL; O=Unizeto Technologies S.A.; OU=Certum Certification Authority; CN=Certum Organization Validation CA SHA2
*  SSL certificate verify ok.
> GET /api/action/dbtimetable_get?id=88cd555f-6f31-43ca-9de4-66c479ad5942&busstopId=4125&busstopNr=02&apikey=XXX HTTP/1.1
> Host: api.um.warszawa.pl
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Date: Sun, 08 Jan 2023 08:10:38 GMT
Date: Sun, 08 Jan 2023 08:10:38 GMT
< Location: https://api.um.warszawa.pl/api/action/dbtimetable_get/?id=88cd555f-6f31-43ca-9de4-66c479ad5942&busstopId=4125&busstopNr=02&apikey=XXX
Location: https://api.um.warszawa.pl/api/action/dbtimetable_get/?id=88cd555f-6f31-43ca-9de4-66c479ad5942&busstopId=4125&busstopNr=02&apikey=XXX
< Content-Length: 386
Content-Length: 386
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1
< Set-Cookie: TS01f95c85=0122371ac80e7f90574109e36d9539b07c00d84caf0ceaea1fd78e1084afddc9b9804c4fc9e1fbe7cbab2ae9a4a8d2e2327d410d55; Path=/; Domain=.api.um.warszawa.pl
Set-Cookie: TS01f95c85=0122371ac80e7f90574109e36d9539b07c00d84caf0ceaea1fd78e1084afddc9b9804c4fc9e1fbe7cbab2ae9a4a8d2e2327d410d55; Path=/; Domain=.api.um.warszawa.pl

< 
* Ignoring the response-body
* Connection #0 to host api.um.warszawa.pl left intact
* Issue another request to this URL: 'https://api.um.warszawa.pl/api/action/dbtimetable_get/?id=88cd555f-6f31-43ca-9de4-66c479ad5942&busstopId=4125&busstopNr=02&apikey=XXX'
* Found bundle for host api.um.warszawa.pl: 0x7fd79b818400 [can pipeline]
* Could pipeline, but not asked to!
* Re-using existing connection! (#0) with host api.um.warszawa.pl
* Connected to api.um.warszawa.pl (91.237.139.103) port 443 (#0)
> GET /api/action/dbtimetable_get/?id=88cd555f-6f31-43ca-9de4-66c479ad5942&busstopId=4125&busstopNr=02&apikey=XXX HTTP/1.1
> Host: api.um.warszawa.pl
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Sun, 08 Jan 2023 08:10:38 GMT
Date: Sun, 08 Jan 2023 08:10:38 GMT
< Access-Control-Allow-Origin: *
Access-Control-Allow-Origin: *
< Strict-Transport-Security: max-age=15768000; includeSubDomains; preload
Strict-Transport-Security: max-age=15768000; includeSubDomains; preload
< X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-Robots-Tag: none
X-Robots-Tag: none
< X-Frame-Options: SAMEORIGIN
X-Frame-Options: SAMEORIGIN
< Content-Length: 319
Content-Length: 319
< Content-Type: application/json; charset=utf-8
Content-Type: application/json; charset=utf-8
< Set-Cookie: TS01f95c85=0122371ac89c1fa2fe0d29a24339e07ba797def37855a34d956cea4dcb728416b927b4f6f0f24eacfa70af5d0413acddf712af9c66; Path=/; Domain=.api.um.warszawa.pl
Set-Cookie: TS01f95c85=0122371ac89c1fa2fe0d29a24339e07ba797def37855a34d956cea4dcb728416b927b4f6f0f24eacfa70af5d0413acddf712af9c66; Path=/; Domain=.api.um.warszawa.pl

< 
    {"result":[{"values":[{"value":"129","key":"linia"}]},{"values":[{"value":"191","key":"linia"}]},{"values":[{"value":"207","key":"linia"}]},{"values":[{"value":"220","key":"linia"}]},{"values":[{"value":"401","key":"linia"}]},{"values":[{"value":"517","key":"linia"}]},{"values":[{"value":"N35","key":"linia"}]}]} 
* Connection #0 to host api.um.warszawa.pl left intact
* Closing connection 0

No idea how to interpret the results, the actions curl takes looks similar to what the lib does inside.
I will try to get to discord server tomorrow, all new to me (need to ask my son :) )

[wzaremba]

I have found, that adafruit_requests is frozen, so I have replaced it with my version.

I have added prints at certain moments of the file (as I don't know how to debug life on the MagTag), I wanted to check, what is actually going on.
I found that function _get_socket is having problems with connecting. To be exact, this is where is the problem:
sock.connect((connect_host, port))
As my Python capabilities are limited, I am asking for help. Comparing 2 different addresses (Adafruit and my API) I found, that my site at first sends 301 code, but then all is good.

Anyone?
Thanks!

[anecdata]

The 301 redirect shouldn't be a problem, but you could try putting the final location URL in as your initial URL in Requests (looks like only a difference of a trailing slash).

Here's what I have in that section of Requests for debugging. It's likely to be an OSError, which one could give clues.

            try:
                sock.connect((connect_host, port))
                print(f"#### DEBUG sock.connect success")
            except MemoryError as e:
                traceback.print_exception(e, e, e.__traceback__)
                sock.close()
                sock = None
            except OSError as e:
                traceback.print_exception(e, e, e.__traceback__)
                sock.close()
                sock = None

[wzaremba]

@anecdata thanks for help here, get to the following error now :):
OSError: Failed SSL handshake

I think that the certificate is not trusted and thou such reaction.
Is there a way to lower the threshold of checking or make it trust?

W.

[anecdata]

Seems like a cipher or certificate issue, or something at that level, but I'm not sure at this point. There's a similar issue here.

There are ways to supply a certificate, but I don't think that's helpful here. I can't connect to api.um.warszawa.pl at all. There is a certificate for *.um.warszawa.pl, but you could try:

context = ssl.create_default_context()
context.check_hostname = False
session = adafruit_requests.Session(pool, context)

If you want to try supplying a certificate, here's some code:
https://gist.github.com/anecdata/85b761d23c14e03f4bc51525c8e7b8f2

If it's an unsupported cipher, espressif (or the website) has to make a change.

[wzaremba]

@anecdata I have taken the certificate from the page, and added as in your example - same results.
I thought that adding this cert will not turn off others, but it did and with such settings, sites that were working like adafruit quotes, stopped.
Waiting for @dhalbert or @support to come and check.
Thanks!

W.

[dhalbert]

I thought that adding this cert will not turn off others, but it did and with such settings, sites that were working like adafruit quotes, stopped.

Using load_verify_locations() replaces the default certificates with the one(s) you supplied. You could switch back to the default set by. set_default_verify_paths() switches back to the default certificates. I guess you could do that, but I think you could create two SSLContext objects, and load your certificate(s) into one. Use that one for the special connects, and use the other for regular connects.

[wzaremba]

OK, found the problem.
I have met this with other applications as well and I was able to implement this here.
As the whole site is unknown (the CA is not as known), you have to download not only site certificate, but whole certification chain (It looks like many certificates one after another). And adding this (in one string) solves the issue.

So steps to solve:

• download Cert chain
• put them as 1 string
• add to context using: context.load_verify_locations(cadata=CA_STRING)

Thank you all for help.