Design goals and milestones

[siddacious]

This is a rough summary of the discussion about Adafruit_CircuitPython_WSGI that I had with @brentru and @mscosti. Any input from the community is welcome and appreciated.

Design Goals:
A flask or sinatra like syntax for defining routes
Pico web is an example to follow though our lib should be more hardware independent than picoweb. It's also worth looking at Werkzeug which is the WSGI engine that flask wraps.

https://palletsprojects.com/p/flask/
https://palletsprojects.com/p/werkzeug/
https://github.com/pfalcon/picoweb

Template support with jinja or similar. Simple one pass substitution to start. Possibly look at cookiecutter for ideas
https://palletsprojects.com/p/jinja/

Security:
Limit the access to the hardware that the webserver has, if possible. It would be great if a user couldn't brick the device with a well-crafted POST

Requests lib from esp32spi is a good example of porting a native module to circuitpython
It is based on python’s requests module

Designs spaces it should ideally supportWYSIWYG tool for creating and exporting UI elements, using the AIO dashboard editor as an example.

Sudo OTA related things
= changing code via drop from web

Questions:
Do we want the user to be responsible for calling the poller?

• perhaps do both; an automated poller and a manual one

This may be useful down the line:
https://palletsprojects.com/p/markupsafe/

[mscosti]

Thanks for writing this up! Below are some of my thoughts / notes / questions.

Things I like about flask and think we should include:

• I really like the idea of using annotations for route specification
• I think we should support wildcard routes
• I think we should support parameter based routes, and the route handler would be given the parsed out parameter value given to it.
• need automatic static asset file serving

General things I think would be neat:

• Standardized way of passing / making available data about the HTTP Request to the custom route handler that was invoked.
  • Some frameworks put this kind of stuff in a Request object and pass it to the route handler function call
• Standardized way of accepting response data.
  • Some frameworks pass a long a 'Response` object that can be filled out and automatically use that
  • Some frameworks expect a Response object to just be returned.
• Automatic Exception handling?
  • catch exceptions and return a 500 error, possibly with the stack trace.
• Custom exception handling
  • Basically register 'exception callbacks' with the application, and if that type of exception happens anywhere in the application, the exception callback gets invoked and can write a a custom Response to deal with that exception type.
• A way to work with reading / writing cookies

Notes:

• I'm not concerned with the security note you have made. I don't think we should limit what an application developer can do, and they can shoot themselves in the foot if they want to write an endpoint that does something bad. IMO opinion its no different than someone today writing CPy code to respond to incoming MQTT messages or bluetooth messages.
  • What we can make sure to do (as a part of the source code of this lib) is NOT execute any code with data from an incoming HTTP Message that could somehow do something malicious. (ex, in JS, there exists a method called 'eval()` that literally takes in a string and executes that string as if it were JS)
  • I'm not that familiar with python attack vectors (read -> 0% familiar), but we could possibly provide utilities to escape input so it is safe to use (like how you would prevent SQL or JS script injection)
• In general I like the idea of OTA python code updates (sounds pretty freaking cool), but I do not think that is something this lib should offer. If the implementation depends on this lib, it could be an addon or 'plugin'. Adding more notes about it, but I don't want to distract the conversation from what we want to develop in this lib
  • It's probably bigger than just an plugin. For this to be really useful, It will need to automatically create its own custom handler to support the hot code reloading functionality, and we would probably want to develop a whole javascript application that can be loaded in your browser that is basically a code text editor.
  • VSCode is open source, is written in JS, and I have seen examples of people having gotten it to run in browser.
• Same thing with a WYSIWYG web app builder. I think it could be really beneficial for a lot of people, (buttons that execute http requests, widgets make polling requests to an endpoint to read live sensor data) but probably a separate tool.
  - Maybe the tool can export some HTML/JS that can be put in your static folder?
  - Maybe the tool exports a JSON/YAML config describing the dashboard, and some additional lib running on the device knows how to understand the config to serve what it needs.

Questions:

• Can we develop this with a 'plugin' system, or 'optional' features in mind? We need to be cognizant of our source code footprint, and I think there will be some features we want to put in (like templating, custom exception handling, cookie handling) that not every project may want or need.
  • Will need to have clear boundaries between what we consider 'core' and what can be added on.

I Agree that we should support both automated and manual poll calling.

[siddacious]

Overall I think we're pretty much on the same page, except for the part where you disagree :P
I agree with your thoughts about things to borrow from flask, and I agree with your list of near things. I like the idea of receiving a Request object into the route handler, but I'm somewhat undecided about how to handle responses. I think this is something we can play with a bit, but I like Sinatra's flexibility and simplicity here where a handler can return:

• An Array with three elements: [status (Fixnum), headers (Hash), response body (responds to #each)]
• An Array with two elements: [status (Fixnum), response body (responds to #each)]
• An object that responds to #each and passes nothing but strings to the given block
• A Fixnum representing the status code

Just an idea, we can explore this further.

On the topic of limiting what an app developer can do, the fact that there are footguns aplenty elsewhere isn't convincing to me. My main thing is that IOT has a well deserved reputation for not taking security seriously, and I'd rather not participate in that. I don't want to make users jump through hoops to do simple stuff just in the name of security, but I also want to make sure we're considering security when we make decisions.

What this means for limiting what the server has access to is open to debate, but I'm leaning in the direction of not limiting the developer but making them make deliberate choices to do stupid things by having sane and reasonable defaults. One example could be "don't serve the whole filesystem by default" or defaulting to read-only access to files.

An real-world example of something I'd like to avoid is making it easy to accidentally or intentionally serve your secrets.py

Regarding a 'plugin' system, this sounds like a good idea. The features you mention are good examples of optional features, I'm sure more will come.

One more general thought is that we can always try stuff in a 0.Y.Z release before stabilizing the API and releasing a 1.0.0

[mscosti]

Yeah, there are loads of different ways we could handle response, we just need to figure out what we like best. I'm guessing we'll end up supporting multiple scenarios like how you mentioned Sinatra does, because you don't always need to send custom headers or a body, etc.

And ok, those are some good security examples I hadn't thought of :P FileSystem access is definitely something we should be careful with. We can probably have a default blacklist of files and file types, that you could override if you knew what you were doing. Very rarely would you need to return .py files for example. You might if we had an OTA coding system, but that's a ways down the line.

I'm all for quick iteration so having 0.minor.patch releases until we arrive at an API we like sounds good to me.

[cefn]

Hi all. Thought a project I put together recently might be worth exploring...

https://github.com/vgkits/corequest/

It hosts coroutine-based state management (intended for novice programmers), and uses a coroutine based request parser and templating under the hood. It runs in Micropython (only tested on Unix version so far).

One layer evolved into something a lot like WSGI, hence I thought it might prompt some thoughts or be a foil for discussions. The WSGI-ish bit is at
https://github.com/vgkits/corequest/blob/master/src/vgkits/corequest/__init__.py#L61

For easy adoption of async web UIs the generator-based state-management paradigm intended for novice developers might offer a handy pattern for simple embedded programming too.

[kattni]

@mscosti @siddacious Did this get wrapped up with the creation of this library? Or was this intended to be a long-term issue?

[mscosti]

I think this was more meant to be a long standing github 'issue', kind of like a discussion space around what this could possibly evolve into.

That being said, maybe with the initial library code merged into an available package as a v1.0 , this could be closed, and separate, more focused issues could be opened for specific features we want added ?

That would also make it easier to call out and manage 'things we definitely think should be included as a feature', and 'things that need more discussion to find out if this is right place for this thing'.

[siddacious]

I meant this as a non-specific discussion of what the requirements and scope of the library should be. That said I think it's got enough content that as @mscosti suggests we can agree on a scope of a 1.0 release, get that out the door, and close this while creating individual issues for discussion.

@mscosti tbh I haven't given this much thought as of late so maybe if you're interested we can schedule a chat for later in the week or the near future to wrap this up.

[siddacious]

@mscosti please take a look over the summary below of what I think we've agreed on. If you agree, we can discuss what remains to be done and then close this and make new issues

Proposed features

• Annotations for route specification
• Wildcard routes
• Parameter based routes
  • route handler given parsed parameter values
• Automatic static asset file serving
• Request object passed during route handler function call
  • Requested URL
  • Parsed parameters
  • Cookies
  • Headers
• Response handling options:
  • A Fixnum representing the status code
  • Array with two elements: [status (Fixnum), response body (responds to #each)]
  • Array with three elements: [status (Fixnum), headers (Hash), response body (responds to #each)]
  • Object that responds to #each and passes nothing but strings to the given block
  • Response object passed by reference and mutated, or created and returned
  • Response obj would likely contain the cookie jar
• Automatic Exception handling
  • catch exceptions and return a 500 error
  • optionally with the stack trace.
• Custom exception handling
  • register 'exception callbacks' with the application
  • exception callback gets invoked if that type of exception happens
  • can write a a custom Response to deal with that exception type
• Reading & writing cookies
  • cookie jar object in request & response
• Automated and manual poll calling.
• Security
  • Don't execute tainted code
  • provide utilities to escape input
  • don't make users jump through hoops to do simple stuff
  • consider security when we make decisions
  • require deliberate choices to do stupid things by having sane and reasonable defaults
  • don't serve the whole filesystem by default
  • default to read-only access to files.
  • make it hard to accidentally or intentionally serve your secrets.py
  • blacklist of files and file types with sane defaults
• Plugins/Optional features
  • Mind our source code footprint
  • Not every project may want features we want to put in like templating, custom exception handling, cookie handling
  • Clear boundaries between what we consider 'core' and what can be added on.

Out of scope but cool:

OTA python code updates (sounds pretty freaking cool), but I do not think that is something this lib should offer. If the implementation depends on this lib, it could be an addon or 'plugin'. Adding more notes about it, but I don't want to distract the conversation from what we want to develop in this lib
It's probably bigger than just an plugin. For this to be really useful, It will need to automatically create its own custom handler to support the hot code reloading functionality, and we would probably want to develop a whole javascript application that can be loaded in your browser that is basically a code text editor.
VSCode is open source, is written in JS, and I have seen examples of people having gotten it to run in browser.
Same thing with a WYSIWYG web app builder. I think it could be really beneficial for a lot of people, (buttons that execute http requests, widgets make polling requests to an endpoint to read live sensor data) but probably a separate tool.

• Maybe the tool can export some HTML/JS that can be put in your static folder?
• Maybe the tool exports a JSON/YAML config describing the dashboard, and some additional lib running on the device knows how to understand the config to serve what it needs.

[dhalbert]

We now recommend using https://github.com/adafruit/Adafruit_CircuitPython_HTTPServer instead, and would like to discontinue supporting this library. I'm closing this issue, but any features suggestions not already in HTTPServer could be added as issues there.