ESP32C3 GTS SSL Root certs not trusted (ie google.com not connecting)

[bruno-dasilva]

CircuitPython version

Adafruit CircuitPython 8.0.0-alpha.1-131-gf9b9f5568-dirty on 2022-08-13; Wemos Lolin C3 Mini with ESP32-C3FH4

Code/REPL

import wifi
import socketpool
import ssl

wifi.radio.connect('ssid', 'password')
pool = socketpool.SocketPool(wifi.radio)

import adafruit_ntp
import rtc
ntp = adafruit_ntp.NTP(pool, tz_offset=0)
rtc.RTC().datetime = ntp.datetime


def connect_https_socket(host, port, pool):
    addr_info = pool.getaddrinfo(host, port, 0, pool.SOCK_STREAM)[0]
    connect_host = addr_info[-1][0]
    sock = pool.socket(addr_info[0], addr_info[1])
    ssl_context = ssl.create_default_context()
    sock = ssl_context.wrap_socket(sock, server_hostname=host)
    connect_host = host
    try:
        sock.connect((connect_host, port))
    finally:   
        sock.close()


connect_https_socket('good.gsr4demo.pki.goog', 443, pool) # success
connect_https_socket('good.r1demo.pki.goog', 443, pool) # fails
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File "<stdin>", line 9, in connect_https_socket
# OSError: Failed SSL handshake
connect_https_socket('good.r2demo.pki.goog', 443, pool) # fails
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File "<stdin>", line 9, in connect_https_socket
# OSError: Failed SSL handshake
connect_https_socket('good.r3demo.pki.goog', 443, pool) # success
connect_https_socket('good.r4demo.pki.goog', 443, pool) # success

# Other examples
connect_https_socket('google.com', 443, pool) # failure
connect_https_socket('play.google.com', 443, pool) # failure
connect_https_socket('io.adafruit.com', 443, pool) # success
connect_https_socket('www.adafruit.com', 443, pool) # success
connect_https_socket('github.com', 443, pool) # success

Behavior

Running this code function connect_https_socket will either produce

1. no error if successful, or
2. a failed handshake error

Depending on the destination. These are obviously very valid certificates (literally for *.google.com).

Description

Looks like its likely some root certificates are not in the firmware generated by the build process for the esp32. I checked and looks like GTS Root R1 was specifically added on this commit to the mozille certdata.txt on 5 Dec 2021 https://hg.mozilla.org/releases/mozilla-beta/annotate/d7c8bc02bda4c5cbeacf2b165e24db7e0ba345c2/security/nss/lib/ckfw/builtins/certdata.txt#l22997

Certs are here: https://pki.goog/repository/

I encountered this on a personal domain proxied through cloudflare that had an ssl cert with the GTS Root R1 root cert.

Additional information

Not sure if related to this fix implemented here (I suspect not): adafruit/circuitpython#3424

[bruno-dasilva]

I originally encountered this using adafruit_requests library but debug further to raw sockets here. Hoping this provides a more straightforward example than using the requests library - though most of the connect_https_socket is stolen right from here: https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/a5d56f3e4866c8dbb343e03500355a42c46e557a/adafruit_requests.py#L494

[bruno-dasilva]

Some questions I have that might be easy to answer and start on the right trackL

1. how is the cert bundle generated (I assume related to this doc https://docs.espressif.com/projects/esp-idf/en/latest/esp32c3/api-reference/protocols/esp_crt_bundle.html)
2. how is the cert bundle embedded in the generated firmware for these ESP devices?

[bruno-dasilva]

I checked the fingerprint of the GTS Root R1 cert in cacrt_all.pem within the esp-idf repository and it matches the fingerprint found on the latest GTS repository.

SHA256 Fingerprint=D9:47:43:2A:BD:E7:B7:FA:90:FC:2E:6B:59:10:1B:12:80:E0:E1:C7:E4:E4:0F:A3:C6:88:7F:FF:57:A7:F4:CF

https://github.com/adafruit/esp-idf/blob/d51f7d882187afa4b39c2613fd0fe2ac2fea1145/components/mbedtls/esp_crt_bundle/cacrt_all.pem#L3193

Ran using

echo $CERT_TEXT | openssl x509 -inform pem -noout -text -fingerprint -sha256

[bruno-dasilva]

Ah I see, I guess we use the roots.pem in the nina-fw repo? https://github.com/adafruit/nina-fw/blob/d73fe315cc7f9148a0918490d3b75430c8444bf7/data/roots.pem

I searched through the roots.pem for the latest GTS Root R1 and couldn't find it. I did find the old one. So safe to say it's not up to date. This roots.pem was updated 6 months ago and is not updated with these new root certs yet.

Can we update roots.pem in nina-fw and propagate that change up through to this repo?

[jepler]

I've transferred this issue to nina-fw because that's where the update needs to take place. After that, we'll need to update the nina-fw submodule in circuitpython as well.

[bruno-dasilva]

Great thanks @jepler!

[dhalbert]

Certificates are now in a separate repo. Need to check if this is now moot or not.

[dhalbert]

GTS Root R1, etc. are now in the list.