ssl: work on anything implementing the socket protocol #8954

jepler · 2024-02-20T17:33:18Z

@anecdata pointed out that moving ssl to shared-module might pave the way to using ssl with wiznet.

I mentioned that for this to be possible, ssl would have to make Python calls for various methods (connect, send, recv, etc) instead of calling the common_hal socket routines.

I went ahead and implemented this. I very lightly tested it: it can do the standard socket test and fetch from https pages like https://www.adafruit.com/api/quotes.php on an esp32s3 device. I didn't test serving.

I also didn't benchmark, test with pico w, let alone test with wiznet.

The following socket methods need to exist and behave like a standard socket:

connect
accept
bind
listen
send
setsockopt (setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) should be implemented, as a no-op if necessary; these constants should match the ones in the core and be provided on the socketpool-compatible object)
recvfrom

Additionally, the following property needs to exist on socket objects, and for stream sockets it has to use the same value as socketpool.SocketPool.SOCK_STREAM (1):

type (this property is also added in this PR, because it didn't exist before)

TODO:

~~benchmark & short circuit for built in sockets if it is needed to avoid a performance regression~~
test with wiznet
test with wifi
test https serving

Additionally, if we choose to go this route we might want to refactor the code a bit so that the Python object representations are sent into the ssl shared_module code, because with this change e.g., a connection address is parsed in shared-bindings/ssl, then turned back into an object in shared-module/ssl to call socket connect, and then parsed a second time in shared-bindings/socket connect. This could be avoided by simply re-organizing so that the shared-module/ssl API took the object.

michalpokusa · 2024-02-20T17:57:31Z

Is this connected to this PR?
#8947

Will it resolve this issue?

jepler · 2024-03-05T03:48:09Z

I'm excited about this PR but realistically we should probably not rush it in before 9.0.0.

justmobilize · 2024-03-05T04:19:33Z

@jepler please let me know if I can do any testing or benchmarks

michalpokusa · 2024-03-05T04:36:14Z

@jepler When ready, I can also help testing HTTPS serving with adafruit_httpserver

anecdata · 2024-03-05T17:35:13Z

Tested TCP socket-level HTTPS request with these artifacts and the core-compatible-socket-type-numbers branch of [adafruit_wiznet5k](https://github.com/adafruit/Adafruit_CircuitPython_Wiznet5k/tree/core-compatible-socket-type-numbers), on an Adafruit ESP32-S3 TFT + Adafruit Ethernet FeatherWing, just confirming jepler's result. Also fine with Adafruit ESP32-S2 TFT.

PicoW + WIZnet W5100S hat gets MemoryError: in the context.wrap_socket() line, not sure how to debug or proceed from there. gc.mem_free() = 76064 after the exception.

addendum: non-SSL without context.wrap_socket() over ethernet still works fine, as does SSL over wifi (no regressions found)

michalpokusa · 2024-03-05T21:36:21Z

Tested HTTPS server on ESP32-S2 TFT.
Board crashes and restarts upon receiving the first request.

HTTP version still works the same.

anecdata · 2024-03-05T21:40:14Z

@michalpokusa I tried HTTPS Server on ESP32-S3 with sockets from WIZnet, but client got connection refused. I'll see if I can replicate espressif-only behavior...

addendum: ESP32-S3 (Adafruit TFT) HTTPS Server seems to be working OK, though I get:

Traceback (most recent call last):
  File "adafruit_httpserver/server.py", line 472, in poll
  File "adafruit_httpserver/server.py", line 604, in _debug_response_sent
TypeError: 'Socket' object isn't subscriptable

library issue, I suspect, that doesn't affect the response being received successfully.

I'm using this PR artifact + your PR88 branch of HTTPServer

michalpokusa · 2024-03-05T22:04:37Z

@anecdata Thank for testing. HTTPS server works on ESP32-S3 for me too, although I successfully get the response, maybe the problem is due to self-signed cert? When I added -k to curl if works, but this is not really the issue with CircuitPython or library.

The TypeError is also present, which is a result of (I believe) incorrect types being returned by SSLSocket.accept():

@jepler I belive it got changed in this PR, as previously this did not happen on ESP32-S3

anecdata · 2024-03-05T22:08:36Z

@michalpokusa Yes, ESP32-S3 works and I get the response. The config that didn't work was an S3 with Ethernet FeatherWing, using WIZnet sockets for the HTTPS server. I am using the -k / --insecure option.

cpython socket objects have a `type` property which gives their type as an integer (e.g., SOCK_STREAM). Add this for compatibility with standard Python. It's needed for ssl, which currently just grabs the value directly from an internal structure (naughty!)

This header can be used by ssl even if there's no core socketpool

In principle this allows core SSL code to be used with e.g., wiznet or airlift sockets. It might actually be useful with wiznet ethernet devices (it's probably not with airlift)

the ssl test program also works here with ```py cs = digitalio.DigitalInOut(board.W5500_CS) spi_bus = board.SPI() ```

We're just going to pass it down to the underlying socket, so don't parse it, multiply it, etc.

jepler · 2024-03-21T19:05:28Z

TypeError: 'Socket' object isn't subscriptable

I believe I found the cause for this. I've updated (rebased) the branch on top of current main and test built espressif, but didn't re-test anything.ese

The subscripting to get the `peer` object was wrong.

jepler · 2024-03-21T19:07:46Z

PicoW + WIZnet W5100S hat gets MemoryError: in the context.wrap_socket() line, not sure how to debug or proceed from there. gc.mem_free() = 76064 after the exception.

this may simply not be a good combo, with wifi taking up extra RAM. I think SSL needs some large blocks of memory that may simply not be left after parsing in all the wiznet code. anyway, that's a total guess.

justmobilize · 2024-04-20T23:12:11Z

I have also drafted a ConnectionManager PR to use ssl when doing adafruit_connection_manager.get_radio_ssl_context(radio)

anecdata · 2024-04-24T14:05:11Z

This PR should close #6535. Related: #2202.

AndreasTheCat · 2024-04-25T09:21:10Z

Based on internal discussions, this is the only board that doesn't already have SSL where we'll enable it

There's also a separate UF2 for W5100S-EVB-Pico (The W6100-EVB-Pico doesn't have its own, but will run the W5100S).

@anecdata you mentioned a separate UF2 for the W5100S-EVB-Pico, which one is it ?

anecdata · 2024-04-25T13:21:04Z

@AndreasTheCat https://circuitpython.org/board/wiznet_w5100s_evb_pico/

anecdata · 2024-04-25T14:48:35Z

Is everyone comfortable with this on the client side? It would be nice to get it merged. Any server tweaks could come after.

edit: looks like there are other open To Dos, not sure the best strategy to avoid risk of performance or other regressions

dhalbert

This looks OK to me. I did not test: thanks everyone else who did test. Let's get it in the next beta!

dhalbert · 2024-04-25T15:01:45Z

edit: looks like there are other open To Dos, not sure the best strategy to avoid risk of performance or other regressions

I saw this after I reviewed. @anecdata Which things are you referring to?

anecdata · 2024-04-25T15:03:07Z

@dhalbert the open checkboxes in the original post. Serving we're aware of. The question I guess is benchmarking / possible refactoring.

dhalbert · 2024-04-25T15:04:51Z

@jepler how about those unchecked tasks? I will withdraw my approval for the moment.

awaiting feedback on remaining tasks in OP

dhalbert · 2024-04-25T15:06:20Z

I am not too worried about performance, but testing on pico w and testing serving I think are necessary.

jepler · 2024-04-25T15:16:15Z

Bypassing the HTTPServer library and running a TCP server directly with this code, modified for WIZnet, results in the same OSError: [Errno 22] Invalid argument upon attempt to connect, followed by a safe mode.

I investigated this.

With an older version of wiznet, I also got an "OSError: [Errno 22] Invalid Argument".

This appears to be due to incorrect handling of timeout=None in recv_into. (the timeout is always None during an SSL handshake) However, this was corrected as a part of adafruit/Adafruit_CircuitPython_Wiznet5k#156

When using the current latest version of wiznet, curl --insecure https://<ip addr>:443/ was able to get the content, but it also reports "curl: (56) Recv failure: Connection reset by peer". Adding a content-length header fixed this.

My code which is for the wiznet 5500 pico-style board: https://gist.github.com/jepler/f479119ce7526d95106b1db924df92ac

Generated a self-signed certificate on linux with openssl req -newkey rsa:4096 -x509 -sha512 -days 365 -nodes -out cert.pem -keyout key.pem

anecdata · 2024-04-25T15:52:32Z

ports/raspberrypi/boards/wiznet_w5500_evb_pico/mpconfigboard.mk

@@ -9,3 +9,4 @@ CHIP_FAMILY = rp2
 EXTERNAL_FLASH_DEVICES = "W25Q16JVxQ"

 CIRCUITPY__EVE = 1
+CIRCUITPY_SSL = 1


Enable SSL also on wiznet_w5100s_evb_pico? I could do this in a separate PR if desired.

jepler · 2024-04-25T16:51:09Z

I also re-tested https serving over wifi on matrixportal s3 with the script at #9003 (comment)

jepler · 2024-04-25T16:51:54Z

Enable SSL also on wiznet_w5100s_evb_pico? I could do this in a separate PR if desired.

@anecdata yes if you feel like putting in a PR for this I'd be grateful!

anecdata · 2024-04-25T17:35:11Z

For completeness, tested Pico W with SSL URL, no apparent regressions. Not sure if there was some other aspect of Pico W to test.

AndreasTheCat · 2024-05-16T11:53:39Z

For completeness, tested Pico W with SSL URL, no apparent regressions. Not sure if there was some other aspect of Pico W to test.

@anecdata pico w and w5100s (using this firmware, this latest adafruit_wiznet5k and this latest adafruit_connection_manager) still runs into MemoryError at 235 in connection_manager- am I missing a part of the discussion here before I try to free any memory (via freezing/ommiting)

anecdata · 2024-05-16T13:06:26Z

@AndreasTheCat there are comments above about memory issues with Pico W + WIZnet. ESP32-S3 + WIZnet seems OK. Not sure about ESP32-S2. I don't know if Pico + WIZnet was explicitly tested... Pico should have more memory than Pico W. I'll try to set one up to replicate later.

anecdata · 2024-05-16T14:29:57Z

Looks like jepler did test a WIZnet dev board (Pico + WIZnet):
#8954 (comment)
So I suspect the issue is the extra memory on a Pico W taken up by wifi-related code. But I'll test some variations and add to this comment later. At the moment I'd say ESP32-S3 + WIZnet is the best combination for SSL on Ethernet... it can run native wifi + multiple WIZnet interfaces with SSL + multiple ESP32SPI (Airlift) interfaces with SSL simultaneously.

Update 1: Pico + W5500:

CircuitPython: Adafruit CircuitPython 9.1.0-beta.2 on 2024-05-15; W5500-EVB-Pico with rp2040
Libraries: ...9.x-mpy-20240516

HTTPS Requests: OK
~50-70KB free after each request

HTTPS Server: OK¹
(using server library from PR #88)
¹still some side effects, but it serves with ~40-50KB free

Update 2: Pico W:

CircuitPython: Adafruit CircuitPython 9.1.0-beta.2 on 2024-05-15; Raspberry Pi Pico W with rp2040
Libraries: ...9.x-mpy-20240516

HTTPS Requests: OK
~50-70KB free after each request

HTTPS Server: MemoryError in poll()
(using server library from PR #88)

Update 3: Pico W + W5100S (using WIZnet interface):

CircuitPython: Adafruit CircuitPython 9.1.0-beta.2 on 2024-05-15; Raspberry Pi Pico W with rp2040
Libraries: ...9.x-mpy-20240516

HTTPS Requests: OK
~10-30KB free after each request

HTTPS Server: MemoryError in WIZnet init
(using server library from PR #88)

AndreasTheCat · 2024-05-17T05:19:47Z

@anecdata

simply amazing to see

`{
"args": {},
"headers": {
"Host": "httpbin.org",
"User-Agent": "Adafruit CircuitPython",
"X-Amzn-Trace-Id": "Root=1-6646e7c5-53d3c3926e38d67d4b0313a8"
},
"origin": "84.59.157.248",
"url": "https://httpbin.org/get"
}

41332 //<-gc.memfree()`

for the first time on pico w with w5100s.
Thank you so much 💯

jepler marked this pull request as draft February 20, 2024 17:42

This was referenced Feb 20, 2024

Error: Failure to bind on ESP when using HTTPS with adafruit_httpserver #8947

Closed

cpython difference in bind(), connect() arguments #8957

Closed

jepler force-pushed the ssl-anything branch 5 times, most recently from 00020eb to 2bb7125 Compare March 5, 2024 03:31

jepler added 7 commits March 21, 2024 13:54

socketpool: alphabetize dict names

7e7918e

socketpool: factor out constants

5973c4a

This header can be used by ssl even if there's no core socketpool

ssl: work on anything implementing the socket protocol

c793a02

In principle this allows core SSL code to be used with e.g., wiznet or airlift sockets. It might actually be useful with wiznet ethernet devices (it's probably not with airlift)

wiznet w5500: improve pin naming & enable ssl

40cc500

the ssl test program also works here with ```py cs = digitalio.DigitalInOut(board.W5500_CS) spi_bus = board.SPI() ```

sslsocket: Simplify handling the timeout value

b330989

We're just going to pass it down to the underlying socket, so don't parse it, multiply it, etc.

SSLSocket: propagate any exception from socket.settimeout directly

f960c5b

jepler force-pushed the ssl-anything branch from 2bb7125 to e3dc82f Compare March 21, 2024 19:05

jepler added 2 commits March 21, 2024 14:06

ssl: make bind & listen into void functions (they throw exceptions)

8c5d9d2

Fix return value of accept()

14a1972

The subscripting to get the `peer` object was wrong.

jepler force-pushed the ssl-anything branch from e3dc82f to 14a1972 Compare March 21, 2024 19:06

jepler added 2 commits April 20, 2024 18:59

Disable some things for the ssl enhancements to fit

879f241

Merge remote-tracking branch 'origin/main' into ssl-anything

a93cbb2

anecdata mentioned this pull request Apr 24, 2024

Wiznet5k SSL support adafruit/Adafruit_CircuitPython_Wiznet5k#59

Closed

dhalbert previously approved these changes Apr 25, 2024

View reviewed changes

jepler requested a review from dhalbert April 25, 2024 15:17

anecdata reviewed Apr 25, 2024

View reviewed changes

dhalbert approved these changes Apr 25, 2024

View reviewed changes

jepler merged commit c92bb9b into adafruit:main Apr 25, 2024
12 checks passed

This was referenced May 17, 2024

Implement TLS for Ethernet #6535

Closed

mbedtls: refactor so it can be used more easily by multiple ports #7007

Closed

Add Native SSL Library #2202

Open

Make sockets compatible with native ssl / CPython adafruit/Adafruit_CircuitPython_ESP32SPI#203

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ssl: work on anything implementing the socket protocol #8954

ssl: work on anything implementing the socket protocol #8954

jepler commented Feb 20, 2024 •

edited

michalpokusa commented Feb 20, 2024 •

edited

jepler commented Mar 5, 2024

justmobilize commented Mar 5, 2024

michalpokusa commented Mar 5, 2024

anecdata commented Mar 5, 2024 •

edited

michalpokusa commented Mar 5, 2024

anecdata commented Mar 5, 2024 •

edited

michalpokusa commented Mar 5, 2024 •

edited

anecdata commented Mar 5, 2024 •

edited

jepler commented Mar 21, 2024

jepler commented Mar 21, 2024

justmobilize commented Apr 20, 2024

anecdata commented Apr 24, 2024

AndreasTheCat commented Apr 25, 2024

anecdata commented Apr 25, 2024 •

edited

anecdata commented Apr 25, 2024 •

edited

dhalbert left a comment

dhalbert commented Apr 25, 2024

anecdata commented Apr 25, 2024

dhalbert commented Apr 25, 2024

dhalbert commented Apr 25, 2024

jepler commented Apr 25, 2024

anecdata Apr 25, 2024 •

edited

jepler commented Apr 25, 2024

jepler commented Apr 25, 2024

anecdata commented Apr 25, 2024

AndreasTheCat commented May 16, 2024

anecdata commented May 16, 2024 •

edited

anecdata commented May 16, 2024 •

edited

AndreasTheCat commented May 17, 2024

ssl: work on anything implementing the socket protocol #8954

ssl: work on anything implementing the socket protocol #8954

Conversation

jepler commented Feb 20, 2024 • edited

michalpokusa commented Feb 20, 2024 • edited

jepler commented Mar 5, 2024

justmobilize commented Mar 5, 2024

michalpokusa commented Mar 5, 2024

anecdata commented Mar 5, 2024 • edited

michalpokusa commented Mar 5, 2024

anecdata commented Mar 5, 2024 • edited

michalpokusa commented Mar 5, 2024 • edited

anecdata commented Mar 5, 2024 • edited

jepler commented Mar 21, 2024

jepler commented Mar 21, 2024

justmobilize commented Apr 20, 2024

anecdata commented Apr 24, 2024

AndreasTheCat commented Apr 25, 2024

anecdata commented Apr 25, 2024 • edited

anecdata commented Apr 25, 2024 • edited

dhalbert left a comment

Choose a reason for hiding this comment

dhalbert commented Apr 25, 2024

anecdata commented Apr 25, 2024

dhalbert commented Apr 25, 2024

dhalbert commented Apr 25, 2024

jepler commented Apr 25, 2024

anecdata Apr 25, 2024 • edited

Choose a reason for hiding this comment

jepler commented Apr 25, 2024

jepler commented Apr 25, 2024

anecdata commented Apr 25, 2024

AndreasTheCat commented May 16, 2024

anecdata commented May 16, 2024 • edited

anecdata commented May 16, 2024 • edited

Update 1: Pico + W5500:

Update 2: Pico W:

Update 3: Pico W + W5100S (using WIZnet interface):

AndreasTheCat commented May 17, 2024

jepler commented Feb 20, 2024 •

edited

michalpokusa commented Feb 20, 2024 •

edited

anecdata commented Mar 5, 2024 •

edited

anecdata commented Mar 5, 2024 •

edited

michalpokusa commented Mar 5, 2024 •

edited

anecdata commented Mar 5, 2024 •

edited

anecdata commented Apr 25, 2024 •

edited

anecdata commented Apr 25, 2024 •

edited

anecdata Apr 25, 2024 •

edited

anecdata commented May 16, 2024 •

edited

anecdata commented May 16, 2024 •

edited