Add support for SSL client certificate (load_cert_chain) and self-signed certificate (load_verify_locations)

[jepler]

Tested with badssl.com on raspberry pi pico w & espressif esp32s3-eye

Testing SSL client certificates

The code and the required .pem files are in key-test.zip but you can convert them using the steps below (I'm not sure how frequently these keys are regenerated, better to do it yourself):

1. Download the client certificate in pem format from https://badssl.com/download/: https://badssl.com/certs/badssl.com-client.pem
2. Convert public portion with openssl x509 -in badssl.com-client.pem -out CIRCUITPY/cert.pem
3. Convert private portion with openssl rsa -in badssl.com-client.pem -out CIRCUITPY/privkey.pem and the password badssl.com
4. Put wifi settings in CIRCUITPY/.env
5. Run the below Python script:

import os
import wifi
import socketpool
import ssl
import adafruit_requests

TEXT_URL = "https://client.badssl.com/"
if not wifi.radio.ipv4_address:
    wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))

pool = socketpool.SocketPool(wifi.radio)
context = ssl.create_default_context()
requests = adafruit_requests.Session(pool, context)

print(f"Fetching from {TEXT_URL} without certificate (should fail)")
response = requests.get(TEXT_URL)
print(f"{response.status_code=}, should be 400 Bad Request")
input("hit enter to continue\r")

print("Loading client certificate")
context.load_cert_chain("/cert.pem", "privkey.pem")
requests = adafruit_requests.Session(pool, context)

print(f"Fetching from {TEXT_URL} with certificate (should succeed)")
response = requests.get(TEXT_URL)
print(f"{response.status_code=}, should be 200 OK")

Closes: #7002

Testing self-signed certificates and load_verify_locations

Obtain the badssl "self-signed" certificate in the correct form:

openssl s_client -servername self-signed.badssl.com -connect untrusted-root.badssl.com:443 < /dev/null | openssl x509 > self-signed.pem

Copy it and the script to CIRCUITPY:

import os
import wifi
import socketpool
import ssl
import adafruit_requests

TEXT_URL = "https://self-signed.badssl.com/"
if not wifi.radio.ipv4_address:
    wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))

pool = socketpool.SocketPool(wifi.radio)
context = ssl.create_default_context()
requests = adafruit_requests.Session(pool, context)

print(f"Fetching from {TEXT_URL} without certificate (should fail)")
try:
    response = requests.get(TEXT_URL)
except Exception as e:
    print(f"Failed: {e}")
else:
    print(f"{response.status_code=}, should have failed with exception")

print("Loading server certificate")
with open("/self-signed.pem", "rb") as certfile:
    context.load_verify_locations(cadata=certfile.read())
requests = adafruit_requests.Session(pool, context)

print(f"Fetching from {TEXT_URL} with certificate (should succeed)")
try:
    response = requests.get(TEXT_URL)
except Exception as e:
    print(f"Unexpected exception: {e}")
else:
    print(f"{response.status_code=}, should be 200 OK")

Closes: #7025

[anecdata]

Self-signed cert didn't work at first, then realized the cert was not only self-signed but expired. Re-generated a new cert and boom:

Loading server certificate
Fetching from https://192.168.4.32 with certificate (should succeed)
response.status_code=200, should be 200 OK

w00t!

[anecdata]

The same server is also accessible via mDNS:

Loading server certificate
Fetching from https://redacted.local with certificate (should succeed)
response.status_code=200, should be 200 OK

For future reference... to make this work, I added to the code:
context.check_hostname = False
and also removed the trailing period from the mDNS name (we don't seem to handle that):
redacted.local. --> redacted.local

[dhalbert]

Code structure looks good. Leaving to anecdata to test functionality, which looks good from here also. I have not used certs in this way for 8 or 9 years.

[anecdata]

I don't really know much about certs, just found the right GUI bits on the device to plow through it. Hopefully someone doing AWS IoT or something similar can test the first part (after merge even).

[dotpointer]

I tried the Testing self-signed certificates and load_verify_locations instructions on an ESP32 S2 mini clone (identified as an ESP32 S2 feather chip) running adafruit-circuitpython-adafruit_feather_esp32s2-sv-8.2.9.uf2 and it works.

To use it with your own hostname, replace the host name with your own when you retrieve the certificate file:
openssl s_client -servername your-hostname-here -connect your-hostname-here-again:443 < /dev/null | openssl x509 > self-signed.pem

Just a plan IP address did not work for me, but a local network host name did.

However, to make HTTPS requests using self-signed certificates I need to download and put the certificate .pem file on the device?

There is no way I can just bypass SSL checks like cURL does with the -k or --insecure options or wget does with the --no-check-certificate?

I thought context.check_hostname = False bypassed the checks, but apparently not, I need to supply the .pem file, otherwise I get OSError: Failed SSL handshake.