Feature request: secure WebSocket (wss) #34
Comments
beyonlo
changed the title
|
SSL and WebSocket are completely independent features. I have the intention to investigate adding SSL support. I wasn't considering websocket, but it's not out of the question either. My concern is that with each addition the size of the package will continue to grow and the lower-end microprocessors will not be able to run anymore, so I'll have to think about doing this in a way that doesn't affect the smaller platforms. |
Well, the microdot webserver/framework do not support HTTPS (SSL over HTTP) right? I think, even lower-end microcontrollers (like as ESP32-C3) would be great a secure connection over browser to configure/operate. So, if you add SSL on HTTP, the websocket can be the same way - I think that SSL is what will use more memory, right? Well, already exists a official module Well, my intention (my application) is to put in ESP32 works a secure HTTPS Server (to serve pages) + secure WebSocket Server, both running on uasyncio. I posted in MicroPython page, a time ago, a issue looking for a Simple example of uasyncio WebSocket Server (secure - with SSL) micropython/micropython#8177 I think that most work to support ssl + asyncio / websockets was done, in many PRs, like this PR shows micropython/micropython#5611 Thank you in advance! |
|
I believe uwebsocket does not work with uasyncio. But in general I agree, I'm open to add support for both SSL and WS. |
Hey @miguelgrinberg That's will be great :) I see that you have two microdot on Well, just as suggestion, maybe a option is have more options, so user can to choose what want: Particularly I liked so much uasyncio/asyncio :) Going forward, with this feature in microdot is possible great applications. I can have the mobile app, desktop app and web app (microdot serving) all working with ESP32 using the same protocol, the Thank you so much! |
|
Update: WebSocket support is coming in the next release of Microdot. Will investigate SSL support after that. |
@miguelgrinberg That is a amazing news!!! I'm very interested!! Actually I started to use this project https://github.com/marcidy/micropython-uasyncio-webexample to have a HTTP Server and a Websocket Server. My intention is to have just one HTML page with an entire large web application using javaScript (Jquery), exactly as that project works. This way is good because all processing will be on Client side (browser), not in the Microcontroller. As the my application has no access to the internet, is needed to put inside HTTP Server the JQuery lib do use on that unique page, exactly as that project works as well. Look here https://github.com/marcidy/micropython-uasyncio-webexample/tree/main/www Will be possible to works in that way (just one page.html and put JQuery lib inside HTTP Server) on the Microdot as well? Thank you very much. |
You can already serve static files with Microdot. I have added an example of how to do this just a few days ago: https://github.com/miguelgrinberg/microdot/tree/main/examples/static. And for websocket it will be just a normal route. This isn't finished code, but if you want to have a look, here is a WebSocket echo example: https://github.com/miguelgrinberg/microdot/blob/websocket/examples/websocket/echo_async.py |
Excellent! That will solve the use case to serve images,
Great, very clean with On the next As soon the next version is released I will start to use Microdot! :) |
Here is one from another project of mine: https://github.com/miguelgrinberg/flask-sock/blob/main/examples/templates/index.html. I did not test it, but I think it should directly work with the echo example I linked above. |
Very good. I will test that, thanks :)
To support SSL on HTTP Server and WebSocket Server will be amazing. I don't know if you know, but recently was created a ticket (micropython/micropython#8915) that has a relation with this feature on Microdot - maybe help. |
|
I tested the However after I finished the test, I observed an error on the terminal. That error do not stopped the send/receive data from the browser (
from microdot_asyncio import Microdot
from microdot_asyncio_websocket import websocket
app = Microdot()
htmldoc = '''<!doctype html>
<html>
<head>
<title>Flask-Sock Demo</title>
</head>
<body>
<h1>Flask-Sock Demo</h1>
<p>Type <b>close</b> to end the connection.</p>
<div id="log"></div>
<br>
<form id="form">
<label for="text">Input: </label>
<input type="text" id="text" autofocus>
</form>
<script>
const log = (text, color) => {
document.getElementById('log').innerHTML += `<span style="color: ${color}">${text}</span><br>`;
};
const socket = new WebSocket('ws://' + location.host + '/echo');
socket.addEventListener('message', ev => {
log('<<< ' + ev.data, 'blue');
});
socket.addEventListener('close', ev => {
log('<<< closed');
});
document.getElementById('form').onsubmit = ev => {
ev.preventDefault();
const textField = document.getElementById('text');
log('>>> ' + textField.value, 'red');
socket.send(textField.value);
textField.value = '';
};
</script>
</body>
</html>
'''
@app.route('/echo')
@websocket
async def echo(request, ws):
c = 0
while True:
data = await ws.receive()
print('Received data from client: {} - type is: {}'.format(data, type(data)))
data = '{}_{}'.format(data, c)
await ws.send(data)
c += 1
@app.route('/')
async def hello(request):
return htmldoc, 200, {'Content-Type': 'text/html'}
@app.route('/shutdown')
async def shutdown(request):
request.app.shutdown()
return 'The server is shutting down...'
app.run(debug=True)Output terminal: Output Browser: |
|
@miguelgrinberg I did tests with the example above ( Works: Chrome, Chromium, Microsoft Edge and Opera. I have not tested others browsers, just these above. Terminal output for Firefox test: Firefox browser output:
Thank you. |
|
Thanks, this is useful feedback. |
|
Hello @miguelgrinberg I did some more tests.
Ps1: one kind of error here is the same found in the reply above (that as running as STA mode). Others errors are a bit different. |
|
Looks like all the errors are caused by not handling the ECONNRESET error. I did not notice this error in my tests, it might be that they are used by the ES32 implementation, which I did not test. I'll look into handling this error properly. |
|
Hi @beyonlo @miguelgrinberg git diff --staged src/*
diff --git a/src/microdot.py b/src/microdot.py
index 5b9e77e..48149a1 100644
--- a/src/microdot.py
+++ b/src/microdot.py
@@ -51,6 +51,8 @@ except ImportError:
except ImportError: # pragma: no cover
socket = None
+import ssl
+
def urldecode(string):
string = string.replace('+', ' ')
@@ -847,7 +849,7 @@ class Microdot():
"""
raise HTTPException(status_code, reason)
- def run(self, host='0.0.0.0', port=5000, debug=False):
+ def run(self, host='0.0.0.0', port=5000, debug=False, key=None, cert=None):
"""Start the web server. This function does not normally return, as
the server enters an endless listening loop. The :func:`shutdown`
function provides a method for terminating the server gracefully.
@@ -882,7 +884,11 @@ class Microdot():
self.server = socket.socket()
ai = socket.getaddrinfo(host, port)
addr = ai[0][-1]
-
+ ctx = None
+ if key:
+ ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
+ ctx.load_cert_chain(cert,
+ keyfile=key)
if self.debug: # pragma: no cover
print('Starting {mode} server on {host}:{port}...'.format(
mode=concurrency_mode, host=host, port=port))
@@ -893,12 +899,17 @@ class Microdot():
while not self.shutdown_requested:
try:
sock, addr = self.server.accept()
+ if key:
+ ssl_sock = ctx.wrap_socket(sock, server_side=True)
except OSError as exc: # pragma: no cover
if exc.errno == errno.ECONNABORTED:
break
else:
raise
- create_thread(self.handle_request, sock, addr)
+ if key:
+ create_thread(self.handle_request, ssl_sock, addr, ctx)
+ else:
+ create_thread(self.handle_request, sock, addr, ctx)
def shutdown(self):
"""Request a server shutdown. The server will then exit its request
@@ -927,7 +938,7 @@ class Microdot():
f = 405
return f
- def handle_request(self, sock, addr):
+ def handle_request(self, sock, addr, ctx):
if not hasattr(sock, 'readline'): # pragma: no cover
stream = sock.makefile("rwb")
else:
@@ -955,6 +966,8 @@ class Microdot():
print('{method} {path} {status_code}'.format(
method=req.method, path=req.path,
status_code=res.status_code))
+ if hasattr(ctx, 'reset'):
+ ctx.reset()
def dispatch_request(self, req):
if req:And the from microdot import Microdot
app = Microdot()
htmldoc = '''<!DOCTYPE html>
<html>
<head>
<title>Microdot Example Page</title>
</head>
<body>
<div>
<h1>Microdot Example Page</h1>
<p>Hello from Microdot!</p>
<p><a href="/shutdown">Click to shutdown the server</a></p>
</div>
</body>
</html>
'''
@app.route('/')
def hello(request):
return htmldoc, 200, {'Content-Type': 'text/html'}
@app.route('/shutdown')
def shutdown(request):
request.app.shutdown()
return 'The server is shutting down...'
with open('ec-cakey.pem', 'rb') as keyb:
key = keyb.read()
with open('ec-cacert.pem', 'rb') as certb:
cert = certb.read()
app.run(debug=True, key=key, cert=cert)Sadly I don't know when or even if my implementation of I will try with microdot |
|
@beyonlo I have put some fixes based on your reports:
Let me know if things look better now. |
|
@Carglglz Thanks for doing this work! This is going to make my work a lot easier when I attempt to do this. Sounds like I might need to use the older (current) SSL implementation though, since it is unclear when or if yours is going to be merged, correct? I also want to see if it is possible to put the SSL support in a separate extension, as I'm doing with WebSocket and other features, so that it does not continue to make the main server larger. |
|
Hello @miguelgrinberg
That's great!
I confirm that it works!
Yes, that old errors do not show anymore, thank you. But now I did two other tests and show these errors: 1. Killing the Browser after connected. Killing the Chrome on Android: Killing the Chromium on Ubuntu: Killing the Firefox on Ubuntu do not show errors 2. Happened a non-intentional lost network, and it back a few seconds after, and show this error:
Excellent!
I never did/used
All right! Additional comments:
|
|
@beyonlo I added fixes for the new errors. Thanks! |
|
Confirm, that ECONNRESET has been fixed ! |
|
@miguelgrinberg I confirm that is working without Thank you! |
|
Could you please to provide a simple example a bit different than echo_async.py using I would like to have many Thank you in advance! |
|
@beyonlo I don't have any examples, but all you need to do is store the |
@miguelgrinberg with this explanation now I understand how to do it - I will to try, thanks! In my tests I found more errors. I don't know when that errors happened and I was no capable to reproduce. Note that the second error ( Edit: Happened one more time: |
|
@beyonlo This is useful, thanks. I think I've fixed these new errors now. |
|
@beyonlo @Carglglz TLS/SSL support is now committed to the main branch. Couple of examples here: https://github.com/miguelgrinberg/microdot/tree/main/examples/tls. Would appreciate it if you test it and provide feedback. Note that the async support cannot currently be used with MicroPython, since uasyncio does not have the necessary support. |
|
@miguelgrinberg I've just tested and it works both in CPython and current MicroPython UNIX port 👍🏼 . This is what I meant, once |
|
@miguelgrinberg I tested on the $ mpremote run hello_tls.py
Starting sync server on 0.0.0.0:4443...
Traceback (most recent call last):
File "<stdin>", line 36, in <module>
File "microdot.py", line 914, in run
File "microdot_ssl.py", line 46, in accept
OSError: (-30592, 'MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE')
$ mpremote ls
ls :
139 boot.py
1493 cert.der
2375 key.der
37277 microdot.py
16463 microdot_asyncio.py
3780 microdot_asyncio_websocket.py
2362 microdot_ssl.py
5935 microdot_websocket.py
Chromium output: Firefox output: |
|
@beyonlo This is expected as you are using self signed certificates, to make it work with your web browser you need to add the self signed certificate to your system trusted certificates. Look for |
|
@Carglglz yes, you are correct. I added that exception and works - screenshot below. But the Microdot do not should to stop in that exception, I believe. @miguelgrinberg it is already possible as well to use Secure WebSocket (wss), os just the HTTPS for now?
|
|
@miguelgrinberg based on your feedback I've modified my implementation of I have to try them in ESP32 port now... [EDIT] |
@Carglglz Is true that will be not possible to have more than 1 Secure Actually I'm using the My intention is after that I need to use the I checked that for each Sorry, but when you say that the Thank you in advance! |
|
@beyonlo Correct, this should not stop Microdot. If it does, then that is an exception that I'm not catching, I'll look into that. Regarding WebSocket, I think it should work, but I have not tested wss yet. @Carglglz I did see that you cannot wrap the main socket in MicroPython. This fake SSL context class that I built allows you to do it, but it is handled internally by delaying the actual call to wrap_socket until after a client socket is available. |
|
@beyonlo SSL use HEAP memory not RAM memory from MicroPython, there is some work to be able to support usage of RAM memory too so it can support multiple SSL sockets, check #8940 @miguelgrinberg yes, I've tried to do the same with mixed results in ESP32 port, for server socket it does work at expense of increasing handshake time, and for client sockets I cannot get more than 4 handshakes in a row... [EDIT] For server side it's still better to enforce connect first since CPython already supports this, and it saves a considerable amount of memory since it does not create additional objects. Also it's far more stable. |
@miguelgrinberg @beyonlo I did a test with WSS and it does work 👍🏼 . It was not too difficult either. Just modify diff --git a/index.html b/../../_local_tests/index.html
index 6e240d5..51d6b86 100644
--- a/index.html
+++ b/../../_local_tests/index.html
@@ -16,7 +16,7 @@
document.getElementById('log').innerHTML += `<span style="color: ${color}">${text}</span><br>`;
};
- const socket = new WebSocket('ws://' + location.host + '/echo');
+ const socket = new WebSocket('wss://' + location.host + '/echo');
socket.addEventListener('message', ev => {
log('<<< ' + ev.data, 'blue');
});And in MicroPython a fake class sslsocket_class:
def __init__(self, sock):
self.sock = sock
def write(self, buf):
return self.sock.write(buf)
def read(self, size):
return self.sock.read(size)
def readline(self, maxbuff=None):
if not maxbuff:
return self.sock.readline()
else:
return self.sock.readline()[:maxbuff]
def readinto(self, *args):
return self.sock.readinto(*args)
def setblocking(self, flag):
return self.sock.setblocking(flag)
def makefile(self, *args, **kwargs):
return self.sock.makefile(*args, **kwargs)
def send(self, buf):
return self.sock.write(buf)
def recv(self, buf):
return self.sock.read(buf)
def __del__(self):
self.sock.__del__()
def getpeercert(self, *args):
return self.sock.getpeercert(*args)
def cipher(self):
return self.sock.cipher()
def close(self):
self.sock.close()Finally from microdot import Microdot, send_file
from microdot_websocket import with_websocket
from microdot_ssl import create_ssl_context
import sys
app = Microdot()
@app.route('/')
def index(request):
return send_file('index.html')
@app.route('/echo')
@with_websocket
def echo(request, ws):
while True:
data = ws.receive()
ws.send(data)
ext = 'der' if sys.implementation.name == 'micropython' else 'pem'
sslctx = create_ssl_context('cert.' + ext, 'key.' + ext)
app.run(port=4443, debug=True, ssl=sslctx) |
|
@Carglglz That's great! Maybe @miguelgrinberg can to generate a ready to go |
|
Thanks. Pushed a fix for wss and also an example. |
|
@miguelgrinberg I tested the new echo_tls.py example on the |
|
@beyonlo these errors did not stop the application, so they are not a cause for concern. Given that it is impossible for me to know all the possible error codes, and when and if it is safe to suppress the error, the option that I prefer is to log the errors and continue, which is what I'm doing here. |
|
@miguelgrinberg I agree! To log the errors and continue is the really the best option, so if something more happen, there is a error log to analyse! |
|
@miguelgrinberg Now that I was to stopping the test I observed that there is one more error, but is a same error happened some weeks ago and was fixed by you. Complete log: EDIT: Just to know: will this error or any error (even already fixed by you in the past) follow the same idea, do not will be suppressed/fixed, just do not stop the Microdot and show that error on the log? EDIT 2: My question above is important to me know if I need to report about a new/different errors or not. Like as this one, one more different error happened: |
|
@beyonlo There are no strict rules. If the error(s) prevent something from working, then they need to be investigated. If the errors are just noise, but everything works, then it really depends. I can silence errors that are clearly not a problem (such as an EPIPE, which just means that the other side closed the socket). Non-standard errors with crazy error codes I feel less inclined to mess with, as these are often the result of specific drivers or libraries, and can mean different things under different stacks. |
|
Report: 1. Strange Behaviour: Since last week (on my first test of hello_tls.py) I would like to report you about a different behaviour in different platforms: -- Using -- Using -- This long time to reach -- Some times, errors are very more intense (using chromium), and bigger delay, and ONE time was need to manually to stop the -- Using the -- More tests was done using -- -- 2. The Bug: -- While I was doing that tests above with hello_tls.py, when I tried to open the |
|
@beyonlo So all these tests that you are doing are with a self-signed certificate that the browser does not accept? Have you tested a certificate that the browser is configured to accept/trust, which is the normal workflow? I haven't really tested self-signed certificates, they're just not very practical since all browsers reject them. |
Yes, following the README.md how to create self-signed certificates - that is exactly what I need for my project, thank you!
I don't know if I understand this question part very well. After that browser detect that is a self-signed certificate, it show a option to
This means that is not self-signed certificate, but a certificate (mostly payed) that is generated and guaranteed by a entity, right? No, I do not have this kind of certificate! Anyway, what I need (and maybe many others) is just to use the self-signed certificates on the applications, where this applications will not be online on the internet, but mostly offline, but supporting secure connection (
Sorry, I was thinking that you created the the README.md just to explain how to create a self-signed certificates to me and other people to test the TLS examples. So you do not used that for tests? Sorry, I'm a bit confuse. |
|
@beyonlo self-signed certificates have become less practical, because browsers introduce all these barriers to use them. Chrome for example does not provide an option to access the site anymore when the certificate is self-signed. I believe this was mentioned above, you can configure your browser to accept certificates from a certificate authority that is managed by yourself. This is closer to how official certificates work. The mkcert tool makes it easy to create them, and to configure browsers to accept them. My goal is to offer support for proper certificates. Self-signed certs is less interesting to me because they have no practical use outside if testing, and even that is not easy anymore with some browsers. The examples in the README are okay to use when you use a client that is not a browser that you can configure to ignore validation. Most clients have an option for this. Browsers unfortunately are not a good case for self-signed certs. I think I'm going to update those instructions to use mkcert instead, because it was not my intention to suggest or recommend self-signed certs for any use other than quick tests between two scripts running a client and a server. |
Understood. I tested that on the Chrome for Android and works. But I tested just Chrome for Android, not Chrome for PC. As my smartphone has a old version of Android, maybe this version of Chrome on Mobile still works.
So sorry, I remember that, but I was thinking that is just about
Perfect! I wasn't know if that is possible - like as a official certificates!
All right, understood!
I think that instructions for the |
@beyonlo This delay is expected (at least what I've tested so far), TLS handshake takes approx 1 second in ESP32 And for the requests ESP32 UNIX |
@Carglglz So, if I use just |
@beyonlo Yes, although there is a feature in TLS called session resumption (it is available in Python see rfc 5077 and SSLContext.wrap_socket), but unfortunately it hasn't been implemented yet in MicroPython and I'm not sure if it will be. |
Hello!
Congratulations for the great project.
I would like to know if you have intention to support secure WebSocket (use SSL over WebSocket) on the Microdot.
Thank you.
The text was updated successfully, but these errors were encountered: